Grep, Awk, and Sed in bash

I have used grep, awk, and sed to manipulate and rewrites these log files. Here are some commands that I found useful to document for the future. If you have never used these bash tools, this might be useful, especially if you are trying to mess around with files that are really big. I’m going to consider sample log file and explain a couple different things you can do with these tools. I am using these tools on OSX and these commands or one-line scripts will work with any Linux flavor.

So, let’s consider the following log file:

www.three.com 10.15.101.11 1353280801 TEST 345
www.one.com 10.14.101.11 1353280801 TEST 343
www.three.com 1.10.11.71 1353280801 TEST 323
www.one.com 10.15.11.61 1353280801 TEST 365
www.two.com 10.10.11.51 1353290801 TEST 55
www.two.com 10.20.13.11 1353290801 REST 435
www.one.com 10.20.14.41 1353290801 REST 65
www.two.com 10.10.11.14 1353290801 REST 345
www.three.com 10.10.11.31 1354280801 REST 34
www.one.com 10.10.13.144 1354280801 JSON 65
www.two.com 10.50.11.141 1354280801 JSON 665
www.three.com 120.10.11.11 1354280801 JSON 555
www.two.com 10.144.11.11 1383280801 RAW 33
www.one.com 10.103.141.141 1383280801 RAW 315
view rawsample.log hosted

Now, here are some things you can do to this log file:

How many files are in a directory: ls | wc -l

Print the file: cat sample.log

Print lines that match a particular word: grep “RAW” sample.log

Print those lines to a file called test.log: grep “RAW” sample.log > test.log

Print particular columns and sort: cat sample.log | awk ‘{ print $1,$2}’ | sort -k 1

Find and Replace using SED and Regex: cat sample.log | sed ‘s/TEST/JSON/g’

Split a log file into multiple files using a column as name with AWK: awk ‘{ print >>($4″.log”); close($4″.log”) }’ sample.log

Use substr (removes last character) in AWK to manipulate a string per line: cat sample.log | awk ‘{ print $1,$2,$3,substr($4,1,length($4)-1),$5}’

Print first line of file with SED: sed q test.log

Print last line of file with SED: sed ‘$!d’ sample.log

Perform a regular expression on last character of entire file using SED: cat sample.log | sed ‘$ s/5$//’

Add some text to beginning and end of a file with AWK: cat sample.log | awk ‘BEGIN{print “START” } { print } END{print “END”}’

Count and print how many unique fields are in all rows using AWK: cat sample.log | awk ‘{ if (a[$1]++ == 0) print $1 }’ | wc -l

Make everything lowercase with AWK: cat sample.log | awk ‘{print tolower($0)}’

Multiple SED regular expressions: sed ’1s/^/START/;$ s/5$/END/’ sample.log

Regex with SED on multiple files: for file in *; do sed ’1s/^/START/’ $file > $file’.json’; done