Showing posts with label AWK. Show all posts
Showing posts with label AWK. Show all posts

Wednesday, 31 July 2019

AWK one-liner collection

AWK is a pattern matching and string processing language named after the surnames of the original authors: Alfred Aho, Peter Weinberger and Brian Kernighan.
Print selected fields
Split up the lines of the file file.txt with ":" (colon) separated fields and print the second field ($2) of each line:
 awk  -F":" '{print $2}' file.txt

Same as above but print only output if the second field ($2) exists and is not empty:
 awk  -F":" '{if ($2)print $2}' file.txt

Print selected fields from each line separated by a dash:
 awk -F: '{ print $1 "-" $4 "-" $6 }' file.txt

Print the last field in each line:
 awk -F: '{ print $NF }' file.txt

Print every line and delete the second field:
 awk '{ $2 = ""; print }' file.txt

Good to know:
The command line option -F sets the field separator. The default is space.
$0 the entire line without the newline at the end
$1 to $9, $10 to ..., the fields
NF number of fields
NR currant line number (counting across all files for multiple files)
FNR line number (just for that file)
Print matching lines
Print field number two ($2) only on lines matching "some regexp" (fiel separator is ":"):
 awk  -F":" '/some regexp/{print $2}' file.txt

Print lines matching "regexp a" and lines matching "regexp b" but the later ones are printed without newline (note the printf):
 awk  '/regexp a/{print};/regexp b/{printf $0}' file.txt

Print field number two ($2) only on lines not matching "some regexp" (fiel separator is ":"):
 awk  -F":" '!/some regexp/{print $2}' file.txt
or
 awk  -F":" '/some regexp/{next;}{print $2}' file.txt

Print field number two ($2) only on lines matching "some regexp" otherwise print field number three ($3) (fiel separator is ":"):
 awk  -F":" '/some regexp/{print $2;next}{print $3}' file.txt
The "next" command causes awk to continue with the next line and execute "{print $3}" only for non matching lines. This is like
/regexp/{...if..regexp..matches...;next}{...else...}

Print lines where field number two matches regexp (apply regexp only to field 2, not the whole line):
 awk '$2 ~ /regexp/{print;}' file.txt
Here is an example parsing the linux "ps aux" command. It has in the eighth column the process state. To print all processes that are in running or runnable state you would look for the letter "R" in that 8-th column. You want as well to print line 1 of the ps command printout since it contains the column header:
 ps aux | awk '$8 ~ /R/{print;}NR==1{print}'

Print the next two (i=2) lines after the line matching regexp:
 awk '/regexp/{i=2;next;}{if(i){i--; print;}}' file.txt

Print the line and the next two (i=2) lines after the line matching regexp:
 awk '/regexp/{i=2+1;}{if(i){i--; print;}}' file.txt

Print the lines from a file starting at the line matching "start" until the line matching "stop":
 awk '/start/,/stop/' file.txt

Print fields 1 and 2 from all lines not matching regexp:
 awk '!/regexp/{print $1 " " $2 }' file.txt

Print fields 1 and 2 from lines matching regexp1 and not matching regexp2:
 awk '/regexp1/&&!/regexp2/{print $1 " " $2 }' file.txt


Regexp syntax:
c matches the non-metacharacter c.
\c matches the literal character c.
. matches any character including newline.
^ matches the beginning of a string (example: ^1 , only lines starting with a one)
$ matches the end of a string (example: end$ , only lines ending in "end")
[abc...] character list, matches any of the characters abc....
[0-9a-zA-Z] range of characters 0-9 and a-z,A-Z
[^abc...] negated character list, matches any character except abc....
r1|r2 alternation: matches either r1 or r2.
r1r2 concatenation: matches r1, and then r2.
r+ matches one or more r's.
r* matches zero or more r's.
r? matches zero or one r's.
(r) grouping: matches r.

In languages like Perl you can use the grouping feature to extract a substring from the matching string. Normal AWK can not use a grouping to chapture a string. However gawk has the match function which can be used for that. The string matched by the first bracket will be in arr[1].
Print the content of the part of the matching regexp that is enclosed by the round brackets:
  gawk 'match($0, /length:([0-9]+) cm/,arr){ print arr[1]}' file.txt

If file.txt looks as shown below then the above command would print 12:
width:3 cm
length:12 cm
height:14 cm

Insert a string after the matching line
This inserts a new line after the matching line:
awk '/regexp/{print $0; print "text inserted after matching line";next}{print}' file.txt
$0 is the line where the search pattern "regexp" matches without the newline at the end. The awk print command prints the string and appends a new line.

This appends a string to the matching line:
awk '/regexp/{print $0 "text appended at end of the matching line";next}{print}' file.txt
If matching "do A" else "do B" (if .. then .. else in awk)
awk '/regexp/{A-here;next}{B-here}' file.txt
Example:
awk '/regexp/{gsub(/string/,"replacement");print $1;next}{print;}' file.txt
The example would print lines that do not match unchanged (action B is just "print;") while on lines that match /regexp/ it would replace /string/ by replacement and print the first element ($1).
If matching "A do..." OR if matching "B do.." (if .. then, if .. then, ...., in awk)
awk '/regexpA/{A-do-here;}/regexpB/{B-do-here}' file.txt
Example:
awk '/house/{print $1;}/cat/{print;}' file.txt

Replacement for some common unix commands (useful in a non unix environment)
Count lines (wc -l):
 awk 'END{print NR}'

Search for matching lines (egrep regexp):
 awk '/regexp/'

Print non matching lines (egrep -v regexp):
 awk '!/regexp/'

Print matching lines with numbers (egrep -n regexp):
 awk '/regexp/{print FNR,$0}'

Print matching lines and ignore case (egrep -i regexp):
 awk 'BEGIN {IGNORECASE=1};/regexp/'

Number lines (cat -n):
 awk '{print FNR "\t" $0}'

Remove duplicate consecutive lines (uniq):
 awk 'a !~ $0{print}; {a=$0}'

Print first 5 lines of file (head -5):
 awk 'NR < 6'
Number non empty lines
This prints all lines and adds a line number to non empty lines:
 awk '/^..*$/{ print FNR ":" $0 ;next}{print}' file.txt
Remove empty lines
This prints all lines except empty ones and lines with only space and tab:
 awk '/^[ \t]*$/{next}{print}' file.txt
Number lines longer than 80 char and show them
This is useful to find all the lines longer than 80 characters (or any other length):
 awk 'length($0)>80{print FNR,$0}'  file.txt
Substitute foo for bar on lines matching regexp
 awk '/regexp/{gsub(/foo/, "bar")};{print}' file.txt
Delete trailing white space (spaces, tabs)
 awk '{sub(/[ \t]*$/, "");print}' file.txt
Delete leading white space
 awk '{sub(/^[ \t]+/, ""); print}' file.txt
Add some characters at the beginning of matching lines
Add ++++ at lines matching regexp.
 awk '/regexp/{sub(/^/, "++++"); print;next;}{print}' file.txt
Color gcc warnings in red
 gcc -Wall main.c |& awk '/: warning:/{print "\x1B[01;31m" $0 "\x1B[m";next;}{print}'
The "\x1B" means the ascii character with hex number 1B (ESC).
Print only lines of less than 80 characters
 awk 'length < 80' file.txt
Renaming files with AWK
You can use awk to generate shell commands such as e.g mv commands to rename files according to a given recipe. I suggest to always print the commands before piping them to sh in order to execute them. A small typo can have very significant side effect so double check what would happen by printing the commands first.

Rename all .MP3 file to be lower case:
ls *.MP3 | awk '{ printf("mv \"%s\" \"%s\"\n", $0, tolower($0)) }'
The above will just print what would happen. To actually execute it you run:
ls *.MP3 | awk '{ printf("mv \"%s\" \"%s\"\n", $0, tolower($0)) }' | sh

Substitute a regexp pattern with a given replacement string. We can e.g replace " " (spaces in the file names) by "-":
ls | awk '{ printf("mv \"%s\" \"%s\"\n", $0, gensub(/ +/,"-","g")) }'
The above will just print what would happen. To actually execute it you run:
ls | awk '{ printf("mv \"%s\" \"%s\"\n", $0, gensub(/ +/,"-","g")) }' | sh
The gensub function reads the strings from $0 (=current line) and returns the modified string. The third argument, the "g", means to find and replace everywhere (globally) on the current line.
AWK as a command-line calculator
This prints 5.1:
awk 'BEGIN{print 3.1+4/2}'

This prints 1.41421:
awk 'BEGIN{print sqrt(2)}'

This prints 1.41421:
awk 'BEGIN{print 2^(1/2)}'

This prints 3.141592653589793 (PI with a 15 digits behind the decimal point):
awk 'BEGIN{printf "%.15f\n",4*atan2(1,1)}'

Print decimal number as hex (this prints 0x20):
awk 'BEGIN{printf "0x%x\n", 32}'

Convert hex string to decimal (this prints 32):
awk 'BEGIN{print strtonum(0x20)}'

Math operators in gnu awk:
+ - * /
^ or ** Exponentiation
% Modulo
exp(), log() Exponential function and natural logarithm
atan2(y, x), sin(), cos() work all in radians (fraction of PI)
sqrt() same as **(1/2) Square root
strtonum() Convert hex (start with 0x) and octal (start with 0) to decimal
If you want to use this frequently then you could put this into your .bashrc file:
# add the awc function to .basrc
# use awc like this: awc "3.4+2+8+99.2" (do not forget the quotes)
awc(){ awk "BEGIN{ print $* }" ;}
On the shell you can then type awc "3.4+2+8+99.2" and it will print 112.6.
AWK minimal web server
You can't write a web server as a reasonable one-liner in AWK, you can do that with netcat but there are some cases where you don't have a real web server and you don't have netcat but you have a very basic shell environment and that does usally include gawk (note: you need gawk, gnu version of awk). Here is a web server that allows you to serve files at port 8080 (or any port, just change the number):
#!/usr/bin/gawk -f
BEGIN {
if (ARGC < 2) { print "Usage: wwwawk  file.html"; exit 0 }
Concnt = 1;
        while (1) {
        RS = ORS = "\r\n";
        HttpService = "/inet/tcp/8080/0/0";
        getline Dat < ARGV[1];
        Datlen = length(Dat) + length(ORS);
        while (HttpService |& getline ){
if (ERRNO) { print "Connection error: " ERRNO; exit 1}
                print "client: " $0;
                if ( length($0) < 1 ) break;
        }
        print "HTTP/1.1 200 OK"             |& HttpService;
        print "Content-Type: text/html"     |& HttpService;
        print "Server: wwwawk/1.0"          |& HttpService;
        print "Connection: close"           |& HttpService;
        print "Content-Length: " Datlen ORS |& HttpService;
        print Dat                           |& HttpService;
        close(HttpService);
        print "OK: served file " ARGV[1] ", count " Concnt;
        Concnt++;
      }
}
Copy this code and save it into a file called wwwawk and then make it executable with "chmod 755 wwwawk". Now take some file (e.g somefile.html) and you can serve it via that little web server:
chmod 755 wwwawk
./wwwawk somefile.html

from another shell:
curl http://localhost:8080
 or
lynx http://localhost:8080
 or
firefox -new-tab http://localhost:8080
This is e.g. a great way to serve kickstart files for automated Linux installations. Note that this awk web server requires gawk. Most linux distributions use gawk by default except for raspberry pi which uses mawk and mawk does not support network connections. 

One line programs in awk

Awk can do very useful things with as little as one line of code, only few other programming languages can do so much with so little. In this article, I show some examples of these one liners. 

Unix/Linux word count utility

awk '{ C += length($0) +1; W += NF } END {print NR, W, C}'

To print origional data values and their logarithms for one column datafiles

awk '{print $1, log($1) }' file(s)

To print a random sample of about 5 percent of the lines from text file

awk 'rand() < 0.05' file(s)

Reporting the sum of the nth column in tables with whitespace separated columns

awk -v COLUMN=n '{ sum += $COLUMN } END { print sum }' file(s)

Report the average of column n

awk -v COLUMN=n '{ sum += $COLUMN } END { print sum / NR }' file(s)

To print the sum of an amount in the last field( number of columns are vary)

awk '{ sum += $NF ; print $0, sum}' file(s)

Some simple ways to search for text in files

egrep 'pattern|pattern' file(s)
awk '/pattern|pattern/' file(s)
awk '/pattern|pattern/ {print FILENAME ":" FNR ":" $0 }' file(s)

Search range of lines

Search lines between 100-150 for the text
awk '{100 <= FNR ) && ( FNR <= 150 ) && /pattern/ {print FILENAME ":" FNR ":" $0 }' file(s)
An alternative way in shell
sed -n -e 100,150p -s file(s) | egrep 'pattern'

To swap the second and third columns in a four column table, assuming tab separators, use any of them below

awk -F'\t' -v OFS='\t' '{print $1,$3,$2,$4}' old >new
awk 'BEGIN { FS = OFS ="\t" } {print $1,$3,$2,$4}' old >new

To convert column separators from tab to ampersand

sed -e 's/tab/\&/g' file(s)
awk 'BEGIN { FS = "\T"; OFS = "&" } { $1 = $1; print }' file(s)

To eliminate duplicate lines from a sorted stream

sort file(s) | uniq
sort file(s) | awk 'Last != $0 { print } { Last = $0 }'

To convert carriage return/newline line terminators to newline terminators, use one of them below

sed -e 's/\r$//' file(s)
sed -e 's/^M$//' file(s)
mawk 'BEGIN { RS = "\r\n" } { print }' file(s)
Note:
The first sed example needs a modern version that recognizes escape sequences.
In the second example, ^M represents a literal Ctrl-M(Carriage return) character.
For the third example, we need either gawk or mawk because nawk and POSIX awk do not support more than a single character in RS.

To convert single spaced text lines to double spaced lines, use any of these

sed -e 's/$/\n' file(s)
awk 'BEGIN { ORS ="\n\n" } { print }' file(s)
awk 'BEGIN { ORS = "\n\n" } 1' file(s)
awk '{print $0 "\n" }' file(s)
awk '{print; print ""}' file(s)

Conversion of double spaced lines to single spacing is equally easy

gwak 'BEGIN { RS="\n *\n" } { print }' file(s)

To strip angle bracketed markup tags from HTML documents, treat the tags as record separators, like this:

mawk 'BEGIN { ORS = " "; RS = "<[^<>]*>" } { print }' *.html
By setting ORS to a space, HTML markup gets converted to a space, and all input line breaks are preserved.

To extract all of the titles from a collection of XML documents

mawk -v ORS=' ' -v RS='[ \n]' '/<title *>/, /<\title *>/' *.xml | sed -e 's@<title *> *@&\n@g
In the example above, it extracts the titles from XML documents, print them one title per line, with surrounding markup. it works correctly even when the titles span multiple lines, and handles the uncommon, but legal, case of spaces between the tag word and the closing angle bracket

Rearranging Fields with awk

Awk is a useful programming language. Although you can do a lot with awk, it was purposely designed to be useful in text manipulation, such as field extraction and rearrangement. In this article, I just show the basics of awk so that you can understand One line programs in awk

Awk patterns and actions

awk's basic paradiam is different from many programming languages. It is similar in many ways to sed:
awk 'program' [ file ...]
The basic structure of an awk program is  :
pattern {action}
pattern {action}
...
The pattern can be almost any expression, but most is print in text manipulation.
awk '{print something}' ...

Fields

awk has fields and records as a central part of its design. awk reads input records and automatically splits each record into fields. It sets the built in valuable NF to the number of fields in each record.
awk '{print NF}'
In above example, awk prints the total number of fields for each record.
Field values are designated as such with the $ character. Usually $ is followed by a numeric constant. However it can be followed by an expression. Here are some examples:
awk '{print $1}'        Print first field 
awk '{print $2,$5}'     print second field and fifth fields 
awk '{print $1,$NF}'    print first and last fields
awk 'NF > 0 {print $0}' print nonempty lines 
awk 'NF > 0'

Setting the field separators

For simple programs, you can change the input field separator with the -F option
awk -F: '{print $1, $5}' /etc/passwd
As for output tet field, the separator can be specified by variable OFS
$awk -F: -v 'OFS=,' '{print $1,$5}' /etc/passwd
root,root
bin,bin
daemon,daemon
adm,adm
...

Printing lines

awk printing is not just limited to fields, but also variables, or strings. For example:
$awk -F: '{print "user:"$1,"description:"$5}' /etc/passwd
...
user:dbus description:System message bus
user:rpc description:Rpcbind Daemon
user:usbmuxd description:usbmuxd user
user:avahi-autoipd description:Avahi IPv4LL Stack

Startup and cleanup actions

There are two special patterns, BEGIN and END, awk uses them to do startup and cleanup actions. It is common to use them in larger awk programs.
The basic structure is like this
BEGIN { start up code }
pattern1 {action1}
pattern2 {action2}
END { cleanup code }
For example:
awk 'BEGIN { FS=":"; COLUMN=2 }' '{ sum += $COLUMN } END { print sum, sum / NR }' file(s)
The example above, input file separator is ":", and column 2 is to be processed, print sum and average of column from input files.

Linux - awk useful examples

Awk  is a pattern scanning and processing language, full-featured text processing language with a syntax reminiscent of C. While it possesses an extensive set of operators and capabilities, we will cover only a few of these here - the ones most useful in shell scripts.

Awk breaks each line of input passed to it into fields. By default, a field is a string of consecutive characters delimited by whitespace, though there are options for changing this. Awk parses and operates on each separate field. This makes it ideal for handling structured text files -- especially tables -- data organized into consistent chunks, such as rows and columns.

Let's see how it works. At the command line, enter the following command:

Print out the whole file

$ awk '{ print }' /etc/fstab or awk '{ print $0 }' /etc/fstab
You should see the contents of your /etc/fstab file as output, same as cat /etc/fstab.
When we executed awk, it evaluated the print command for each line in /etc/passwd in order.
For an explanation of the { print } code block.In awk, curly braces are used to group blocks of code together, similar to C.
Inside our block of code, we have a single print command.In awk, when a print command appears by itself, the full contents of the current line are printed, the $0 variable represents the entire current line, so print and print $0 do exactly the same thing.

Deal with multiple fields

It works like cut, but more powerful than cut, which can only use single character as seperator. By default, it uses whitespace as separator.
As we mentioned above, $0 represents the entire current line of the input, $1 represents the first colomn of the input, while $2 is for the second column, etc..
$awk '{print $1,$2}' /etc/fstab
It will print out the first and the second column of the file /etc/fstab

Print out your own string

$ awk '{ print "#" $0 }' /etc/fstab
It prints every line in /etc/fstab, and adds "#" the begining of every line.

Specify separator for the input file

The following script will print out a list of all user accounts on your system:
$ awk -F":" '{ print $1 }' /etc/passwd

In above case, we use the -F option to specify ":" as the field separator. When awk processes the print $1 command, it will print out the first field that appears on each line in the input file.
Here's another example:
$ awk -F":" '{print $1,$3}' /etc/passwd
root 0
bin 1
daemon 2
adm 3
In above example, awk prints out username and uid of each user in your system. Also you may have noticed that there is a ',' between $1 and $2 field, this is to tell awk to separate the two fields in output. Default output seperator is a single space.

Specify separator for the output

Awk default seperator is 'OFS', a single space.

If you want to assign a different seperator, for example, a tab
$ awk -F":" --assign OFS="\t" '{print "user:"$1,"uid:"$3}' /etc/passwd
user:root    uid:0
user:bin    uid:1
user:daemon    uid:2
In above example, awk prints out the the first and third column of /etc/passwd file, separated by ":" and output fields are separated by tab. Note: there is no OFS between "user:" and $1, "uid:" and $3.
why ?
"," is needed between fields to ask awk to use output separator, otherwise, fields will be concatenated.

Search pattern

awk search pattern is a regular expression, for example,

Search and print lines with ext string

# awk '/ext/  {print }' /etc/fstab
LABEL=/1                /                       ext3    defaults        1 1
LABEL=/tmp              /tmp                    ext3    defaults        1 2
LABEL=/home             /home                   ext3    defaults        1 2
LABEL=/usr              /usr                    ext3    defaults        1 2

Print uncommented out lines in the file /etc/fstab

# awk '$0 !~ "^#" {print}' /etc/fstab
LABEL=/1                /                       ext3    defaults        1 1
LABEL=/tmp              /tmp                    ext3    defaults        1 2
LABEL=/home             /home                   ext3    defaults        1 2
LABEL=/usr              /usr                    ext3    defaults        1 2
LABEL=/opt              /opt                    ext3    defaults        1 2
...

Print file systems that kernel will mount by default.

# awk '$4 == "defaults" && $1 !~ "^#"  {print}' /etc/fstab
LABEL=/1                /                       ext3    defaults        1 1
LABEL=/tmp              /tmp                    ext3    defaults        1 2
LABEL=/home             /home                   ext3    defaults        1 2
LABEL=/usr              /usr                    ext3    defaults        1 2

The BEGIN and END blocks

Normally, awk executes each block of your script's code once for each input line. However, there are many programming situations where you may need to execute initialization code before awk begins processing the text from the input file. For such situations, awk allows you to define a BEGIN block. The BEGIN block is evaluated before awk starts processing the input file, it's an excellent place to initialize the FS (field separator) variable, print a heading, or initialize other global variables that you'll reference later in the program.

Awk also provides another special block, called the END block. Awk executes this block after all lines in the input file have been processed. Typically, the END block is used to perform final calculations or print summaries that should appear at the end of the output stream.
# awk 'BEGIN{FS=":";OFS="\t\t"; print "username\tuid"}  {print $1,$3}' /etc/passwd
username    uid
root        0
bin        1
daemon        2
adm        3
Another fine print control example: using printf
awk 'BEGIN{FS=":";OFS="\t\t"; print "username\tuid"} {printf "%8s\t%d\n", $1,$3} END{print "Total " NR " fields have seen so far"}' /etc/passwd
username    uid
    root    0
     bin    1
...
      nx    990
  Salina    1003
Total 36 fields have seen so far
Note: in the example above, OFS is ignored
Below is the common variable awk uses
       NF          The number of fields in the current input record.
       NR          The total number of input records seen so far.
       FS          The output field separator, a space by default.
       OFS        The output field separator, a space by default.

Conditional statements

Awk also offers very nice C-like if statements.
{ if ( $5 ~ /root/ ) { print $3 } }
In the example, the block is executed for every input line

Here's a more complicated example of an awk if statement. As you can see, even with complex, nested conditionals, if statements look identical to their C counterparts:
{
  if ( $1 == "foo" ) 
    { if ( $2 == "foo" ) 
      { print "uno" } 
    else
      { print "one" }
    }
  else if ($1 == "bar" ) 
    { print "two" } 
  else 
    { print "three" } 
}

Numeric variables

So far, we've either printed strings, the entire line, or specific fields. However, awk also allows us to perform both integer and floating point math. Using mathematical expressions, it's very easy to write a script that counts the number of blank lines in a file. Here's one that does just that:
BEGIN { x=0 }
/^$/  { x=x+1 }
END   { print "I found " x " blank lines. :)" }

In the BEGIN block, we initialize our integer variable x to zero. Then, each time awk encounters a blank line, awk will execute the x=x+1 statement, incrementing x. After all the lines have been processed, the END block will execute, and awk will print out a final summary, specifying the number of blank lines it found.

HOW TO LEFT PAD NUMBERS IN AWK

Occasionally when working in a Linux/Unix environment comes the need to left pad numbers with zeros (add leading zeros to numbers). This is often done to make a series of numbers the same length. An example of doing this with awk can be seen below:

HOW TO LEFT PAD NUMBERS IN LINUX

There are many ways to left pad numbers in Linux. Usually this seems to be done by adding leading zeros in front of the main number. A few examples of doing this can be seen below:

Using printf

Using AWK

Looping in Bash

Subscribe to: Posts (Atom)