AWK one-liner collection

AWK is a pattern matching and string processing language named after the surnames of the original authors: Alfred Aho, Peter Weinberger and Brian Kernighan. Print selected fields Split up the lines of the file file.txt with ":" (colon) separated fields and print the second field ($2) of each line:  awk  -F":" '{print $2}' file.txt Same as above but print only output if the second field ($2) exists and is not empty:  awk  -F":" '{if ($2)print $2}' file.txt Print selected fields from each line separated by a dash:  awk...

Grep your bash history commands and execute one

While working in the Bash shell it is common to want to repeat a command that you have recently executed. Bash keeps a history of executed commands in a history file  .bash_history that you can access by simply typing history. > history 1  ls 2  cd ~ 3  ls .* 4  cat .bash_history 5  history This will output a list of commands prefixed with an identification number. If you only want to see the last N entries in the history, type history N. > history 4 3  ls .* 4  cat .bash_history 5  history 6 ...

Grep, Awk, and Sed in bash

I have used grep, awk, and sed to manipulate and rewrites these log files. Here are some commands that I found useful to document for the future. If you have never used these bash tools, this might be useful, especially if you are trying to mess around with files that are really big. I’m going to consider sample log file and explain a couple different things you can do with these tools. I am using these tools on OSX and these commands or one-line scripts will work with any Linux flavor. So, let’s consider the following log file: www.three.com...

USING GREP TO SEARCH FOR TEXT IN LINUX

Grep is a Unix tool used for finding text within files. This tool is very simple to use, much misunderstood and very powerful. It is an essential command to master when using Unix and Linux. Grep is the Linux equivalent of Windows Find in Files. Grep can use regular expressions to search files or output for text, it can also use plain text searches. To search for files containing a particular string, it is as easy as typing in grep "findme" This will list out the files and the text surrounding the match. You can add the -r flag to search recursively. grep...

One line programs in awk

Awk can do very useful things with as little as one line of code, only few other programming languages can do so much with so little. In this article, I show some examples of these one liners.  Unix/Linux word count utility awk '{ C += length($0) +1; W += NF } END {print NR, W, C}' To print origional data values and their logarithms for one column datafiles awk '{print $1, log($1) }' file(s) To print a random sample of about 5 percent of the lines from text file awk 'rand() < 0.05' file(s) Reporting the sum of the nth column in tables...

Rearranging Fields with awk

Awk is a useful programming language. Although you can do a lot with awk, it was purposely designed to be useful in text manipulation, such as field extraction and rearrangement. In this article, I just show the basics of awk so that you can understand One line programs in awk Awk patterns and actions awk's basic paradiam is different from many programming languages. It is similar in many ways to sed: awk 'program' [ file ...] The basic structure of an awk program is  : pattern {action} pattern {action} ... The pattern can...

Checkout file original timestamp for files from Git

For people like me used to use cvs for version control, when switching to Git, one of the thing bothers me a lot is file's origional timestamp. In case you want to checkout the files origional create/commit timestamp, here is my trick. After checkout, run the script below, it will get the original create/commit timestamp from Git db. #!/bin/bash -e PATH=/usr/bin:/bin unalias -a get_file_rev() {     git rev-list -n 1 HEAD "$1" } update_file_timestamp() {   file_time=$(git show --pretty=format:%ai --abbrev-commit "$(get_file_rev...

convert upper case to lower case on Linux, in different ways

convert upper case to lower case on linux, in different waysThere are quite a lot ways to do the task on linux, most of them are easy to use. Here are different useful examples for different cases. #1 text file editor, vi, vim, ex etc.. They are in the same editor familyOpen the file you want to conver all upper case to lower case, then use the following commands :1,$ s/[a-z]/\u &/g Above command can be explained as follows: Command         Explanation 1,$     Line Address...

Linux - awk useful examples

Awk  is a pattern scanning and processing language, full-featured text processing language with a syntax reminiscent of C. While it possesses an extensive set of operators and capabilities, we will cover only a few of these here - the ones most useful in shell scripts.Awk breaks each line of input passed to it into fields. By default, a field is a string of consecutive characters delimited by whitespace, though there are options for changing this. Awk parses and operates on each separate field. This makes it ideal for handling structured...

Linux - cut command useful examples

cut is a command from coreutility which allows you to print selected parts of lines from each FILE to standard output. Here is an output from lscpu -p option, it will be used in examples below as testfile # Socket,CPU,Core,Address,Online,Configured,Polarization 0,0,0,,Y,, 0,1,1,,Y,, 0,2,2,,Y,, 0,3,3,,Y,, 0,4,4,,Y,, 0,5,5,,Y,, 1,6,6,,Y,, 1,7,7,,Y,, 1,8,8,,Y,, 1,9,9,,Y,, 1,10,10,,Y,, 1,11,11,,Y,, Example 1: print one specified field(second field in below example) # cut -d',' -f2 testfile CPU 0 1 2 3 ... Option -d and -f      ...