Linux awk example

Use awk to filter log files on values 'greater than' or 'less than' a threshold in Linux
Assume the following scenario: you want to tail the log files of a webserver, but you only want to see the requests whose files size exceeds 10MB.

Normally, you'd tail your logs like this.

$ tail -f access.log
...
127.0.0.1 "GET /path/to/files.css HTTP/1.1" 200 145848 "" "Mozilla"
127.0.0.1 "GET /path/to/files.js HTTP/1.1" 200 195848 "" "Mozilla"
127.0.0.1 "GET /path/to/big-movie-file.webm HTTP/1.1" 200 11146409 "" "Mozilla"
You can use awk to filter the log files where a particular value exceeds a threshold. For instance, assuming the log files above (which have been shortened for readability), the filesize of the request is in the 6th field in the logs. In the above example, it's right after the HTTP 200 status code.

To clarify, since awk by default splits on spaces (or "whitespace") in the output, it detects the following blocks. Each are logically numbered $1, $2, $3, ...

awk_variables_in_output

So in order to filter your logs, you can use awk like this, which uses field $6 to compare a value against..

$ tail -f access.log | awk '$6 > 10000000'
To break it down:

$6: the 6th field (by default, space separated) in the output of tail
> 10000000: the value should exceed 10.000.000. Since logfiles express this value in bytes, we do 1024 * 1024 * 10 to get 10MB
awk is a very powerful tool, even understanding just the basics like these comparison parameters can get you a long way!