Showing posts with label Linux grep. Show all posts
Showing posts with label Linux grep. Show all posts

Friday, 9 August 2019

Grep for Windows – findstr example

I love grep command on Linux, it helped to search and filter strings easily, always wonder what is the equivalent tool on Windows, and found this findstr recently.
In this article, I will share some of my favorite “grep” examples on Linux, and how to “port” it to Windows with “findstr” command.

1. Filter a result

1.1 Classic example to filter a listing result.
#Linux
$ ls -ls | grep mkyong

#Windows
c:\> dir | findstr mkyong

1.2 Add ignore case, and filter the listing result with multiple strings.
#Linux - Need '-E' option and Uses "|" to separate multiple search strings.
$ ls -ls | grep -iE "mkyong|music"

#Windows - Use spaces to separate multiple search strings
c:\> dir | findstr -i "mkyong music"

2. Search a File

2.1 Search matched string in a file.
#Linux 
$ grep mkyong test.txt

#Windows
c:\> findstr mkyong test.txt

2.2 Counting the number of matches.
#Linux
$ grep -c mkyong test.txt

#Windows - Piped with find /c command.
c:\> findstr -N "mkyong" test.txt | find /c ":"

3. Search a list of files

3.1 Search matched string in a list of files.
#Linux
$ grep mkyong -lr /path/folder

#Windows
c:\> findstr /M mkyong c:\folder\*

* (grep) -l , (findstr) /M = print only name of files containing matches.

4. Help

4.1 The most powerful command ~
#Linux 
$ grep --help 
$ man grep 

#Windows
c:\> findstr -?

Thursday, 8 August 2019

Look for a string, word, or sentence in a file with Linux grep command, recursively

Introduction

Sometimes you remember a phrase or a given word or words, you put in a document, but you do not remember the name of the document. How to find it?, well do a search of all your documents, looking for that word or words or sentence, in other words, look for a string or strings.
Now, thanks to computers this is easier now, than it was in our parents' days.
If you are using Linux, you have grep to help on this job.

Look for a document, containing a given string

First the easy case, you know the exact sentence, you are looking for, and you at least remember the folder where the file is.
grep "sentence to look for" /home/user/docs/
Now, let's suppose you do not know if the sentence was in uppercase or in lowercase, so ask grep to ignore case.
grep -i "sentence to look for" /home/user/docs/
Well, let's suppose you have a lot of sub-folders, and you do not remember where your file is.
grep -r "sentence to look for" /home/user/docs/
Let's see an example of the output in my PC
grep -ri "Introduction" /home/user/post/
The output is:
/home/user/post/monthy-newsletter.txt:###Introduction###
/home/user/post/monthy-newsletter.txt:I'm no expert in MySQL, but anyway I have written three introduction-type MySQL posts, something we all need to know to start [how to create databases][1], [How to list those databases][2] and [How to create tables in MySQL][3]
/home/user/post/how-to-debug-bash-shell-scripts.txt:##Introduction##
/home/user/post/how-to-setup-dns-bind-master-slave-linux.txt:##Introduction
/home/user/post/interview-raphael-hertzog.txt:I also have plans for bigger changes concerning Debian, and among them is the introduction of Debian Rolling, a distribution similar to testing but with some design choices to make it more usable at any point in time.
/home/user/post/four-years-with-debian-testing.txt:##Introduction##
Binary file /home/user/post/.find-a-documents-with-given-string.txt.swp matches
/home/user/post/how-to-change-the-priority-of-Linux-processes.txt:##Introduction##
As you can see there are some binary files, also scanned, if we want to avoid that:
grep -riI "introduction" /home/user/post/
The output now is:
/home/user/post/monthy-newsletter.txt:###Introduction###
/home/user/post/monthy-newsletter.txt:I'm no expert in MySQL, but anyway I have written three introduction-type MySQL posts, something we all need to know to start [how to create databases][1], [How to list those databases][2] and [How to create tables in MySQL][3]
/home/user/post/how-to-debug-bash-shell-scripts.txt:##Introduction##
/home/user/post/how-to-setup-dns-bind-master-slave-linux.txt:##Introduction
/home/user/post/interview-raphael-hertzog.txt:I also have plans for bigger changes concerning Debian, and among them is the introduction of Debian Rolling, a distribution similar to testing but with some design choices to make it more usable at any point in time.
/home/user/post/four-years-with-debian-testing.txt:##Introduction##
/home/user/post/how-to-change-the-priority-of-Linux-processes.txt:##Introduction##
Finally, I just want the file names, and not the sentences where the sentence or word appears.
grep -riIl "introduction" /home/user/post/
The output will be:
/home/user/post/monthy-newsletter.txt
/home/user/post/how-to-debug-bash-shell-scripts.txt
/home/user/post/how-to-setup-dns-bind-master-slave-linux.txt
/home/user/post/interview-raphael-hertzog.txt
/home/user/post/four-years-with-debian-testing.txt
/home/user/post/how-to-change-the-priority-of-Linux-processes.txt

Search patterns in files using Linux grep command

Does you job requires you to frequently search for patterns in files through Linux command line? Or, do you feel frustrated when you have to open files in UI editors to search for strings or patterns on Linux? Well, if yes then the Linux grep command is for you. This command can be used to search a pattern in one or more files directly from the command line. 

In this article, we will understand the usage of Linux grep command through practical examples.
 

SYNTAX

Before jumping on to the examples, lets first take a look on how to use the grep command. Here is the basic syntax information of grep command from the man page :
grep [OPTIONS] PATTERN [FILE...]
So we see that the grep command does require PATTERN as a mandatory argument. The OPTION and FILE arguments are non-mandatory. While the OPTION argument tells the grep command to act in a way as specified by the definition of that OPTION, the FILE argument tells the grep command about the files in which the pattern needs to be searched. The ellipsis '...' in the argument FILE indicates that more than one files can be presented in the argument list. 

NOTE For those who are new to this type of syntax information, any argument specified in square brackets [] are non-mandatory.
 

EXAMPLES

 

1. A basic example

Here is how the grep command can be used in its most basic form.
# grep "Linux" input.txt 
Welcome to Linux.
In the output above, the line in the file input.txt containing the pattern or string "Linux" was displayed as output.
 

2. Pattern matching is case sensitive

The pattern matching done by grep command is case sensitive. For example, if the argument to grep command is "LINUX" (instead of "Linux") then grep will not match the lines containing string "Linux". 

Here is an example :
# grep "LINUX" input.txt 
#
So we see that no output was displayed. If it is desired that grep command should ignore the case sensitiveness then the option -i can be used. Here is the example :
# grep -i "LINUX" input.txt 
Welcome to Linux.
So we see that this time the string "LINUX" matched with the line containing the string "Linux".
 

3. Search in more that one file

If more than one file is supplied in argument list then grep searches for the pattern or string in all the files. 

For example :
# grep "Linux" input.txt output.txt 
input.txt:Welcome to Linux.
output.txt:I hope you enjoyed working on Linux.
As we can see in the output above, the lines containing the string "Linux" along with their respective file names were displayed in the output.

Also, to search in a complete directory, the argument '*' can passed as input.

Here is an example :
# grep "Linux" *
input.txt:Welcome to Linux.
output.txt:I hope you enjoyed working on Linux.
Binary file test_strace matches
test_strace.c:    if(NULL == fopen("Linux","rw"))
So we see that lines in all the files (in current directory) containing the string "Linux" were displayed as output.
 

4. Search recursively using -r option

There exists an option -r through which the grep command can search for pattern (or string) recursively in the sub-directories. 

Here is an example :
# grep -r "Linux" *
input.txt:Welcome to Linux.
new_dir/new.txt:Linux vs Windows
output.txt:I hope you enjoyed working on Linux.
Binary file test_strace matches
test_strace.c:    if(NULL == fopen("Linux","rw"))
So we see that the output contains matching results from files contained in sub-directories.
 

5. Match patterns using regular expressions

The grep command also allows the usage of regular expressions in pattern matching. This provides tremendous power to user using the grep command to search for any possible pattern that can be represented through regular expression. 

Here is an example :
# grep -r ".*Linux" output.txt output1.txt 
output.txt:I hope you enjoyed working on Linux.
output1.txt:Welcome to Linux.
output1.txt:I hope you will have fun with Linux.
As we can see that the grep command above used a regular expression ".*Linux" for pattern matching in files output.txt and ouput1.txt. 

Here is a table of regular expression operators and their effect :
Operator Effect
. Matches any single character.
? The preceding item is optional and will be matched, at most, once.
* The preceding item will be matched zero or more times.
+ The preceding item will be matched one or more times.
{N} The preceding item is matched exactly N times.
{N,} The preceding item is matched N or more times.
{N,M} The preceding item is matched at least N times, but not more than M times.
- represents the range if it's not first or last in a list or the ending point of a range in a list.
^ Matches the empty string at the beginning of a line; also represents the characters not in the range of a list.
$ Matches the empty string at the end of a line.
\b Matches the empty string at the edge of a word.
\B Matches the empty string provided it's not at the edge of a word.
\< Match the empty string at the beginning of word.
\> Match the empty string at the end of word.

Any one or a combination of these operators can be used to form a regular expression that represents the pattern of user's choice.

GREP COMMAND EXAMPLES WITH EXTENDED REGEX IN LINUX/UNIX

In this post, we will see how to use extended regular expressions to increase the power of grep command even better than Basic regular expression.
Extended regular expressions:
+ --Match one or more occurrences of the previous character.

| -- Match Either character

? – Match 0 or 1 occurrence of the previous character.

() –match a group of characters

{number} –Match number of occurrence of a character

{1, 3} –Match a character which is 1 to 3 times repetition

{5, } –Match a repeated character which is repeated 5 or more times.
Note1: To use this extended regular expressions we have to use –E option give grep the capability to understand Extended regular expressions.
Note2: egrep command is nothing but grep with –E option, so please don’t learn it. If the grep command it self can do the work for you, why to learn new Linux command?
Examples:
Example1: Search for a word which contains one or more occurrence of ‘b’ between a and c. The below command will search for words like abc, abbc, abbbc etc.
grep –E ‘ab+c’ filename
Example2: Search for a word which contains zero or one occurrence of b between a and c. The below command will search for ac, abc. This is a bit tricky as the first search ac do not contain b. As ? states, it searches for zero or one occurrence of a character.
grep –E ‘ab?c’ filename
Example3: Search for multiple words with single grep command.
Suppose you want to search for john and you want to search for marry as well. Do not use two grep commands as shown.
grep john filename

grep marry filename
Instead of using two grep commands, use -E options to search multiple words with single grep command.
grep –E ‘john|marry’ filename 
How about searching three words with single grep command?
grep -E 'john|marry|tony' filename
Example4: Search for a word which contains either a or b, but not both a and b between d, e characters
grep –E ‘d(a|b)e’ filename 

INTERVAL REGULAR EXPRESSIONS WITH GREP

Example5: Search for a word which contains only 2 ‘b’ between a and c character
grep –E ‘ab{2}c’ filename
Example6: Search for a word which contains 3 to 4 ‘b’ between a and c character. so this will search for abbc, abbbc, and abbbbc as well.
grep –E ‘ab{2,4}c’ filename
Example7: Search for a word which contains 3 or more ‘b’ between a and c character. This will search for abbbc, abbbbc, and so on.
grep –E ‘ab{3, }c’ filename 
Note: When we are using {} we have to give a range, but in this example, we did not provide range we just started the range but did not end the range which indicates infinity.

GREP COMMAND BASIC REGEX EXAMPLES IN LINUX/UNIX

In this post, we will see how to use Basic regular expressions to increase the power of grep command.
Basic regular expressions:
^ -- Caret symbol, Match beginning of the line.

$ -- Match End of the line

* -- Match 0 or more occurrence of the previous character

. – Match Any single character

[] – Match Range of characters, just single occurrence.

            [a-z] –Match small letters

            [A-Z] –Match cap letters

            [0-9] – Match numerical.

[^] – Match Negate a sequence

 -- Match Escape character.
If we learn any new thing with the example, it will be there for a long time in our mind. Now we will see one example each regular expression what we have given above.
Example1: Find all the lines which start with “Mr”
grep ‘^Mr’ filename
Example2: Find all the lines which end with ‘sh’
grep ‘sh$’ filename
Example3: Display all the lines in a file expect empty lines.
grep –v ‘^$’ filename
Note:-v option is used to negate the search term, here ^$ indicates empty line, so our grep –v is filtering blank lines in the output.
Example4: Search for words which are bash, baash, bsh, baaash, baaaaash,
grep ‘ba*s’ filename
The above grep command will search for words which have a between b and s zero or more times.
Example5: Search for all words which start with b and h
grep ‘b.*h’ filename
Example6: Search for a word which is having three letters in it and starts with x and ends with m.
grep ‘x[a-z]m’ filename
This search will return all the three letter words which start with x and ends with m.
Example7: Search words do not contain ‘ac’ in a file.
grep ‘[^ac]’ filename
Example8: Search for a ‘[‘ in a file
Note: The “[“ is a special character, you cannot search with regular grep we have to use escape character () to negate it. So use ‘[‘ to search for [. This is applicable for all the special characters mentioned above.
grep ‘[’ filename
Please feel free to comment on this, if you have more thoughts. Stay tuned to our next grep post on how to use extended regular expressions.

GREP COMMAND EXAMPLES IN LINUX/UNIX

GREP is a command line search utility or tool to filter the input given to it. Grep got its name from ed editor as g/re/p (global / regular expression / print). Grep command can improve a command output by filtering out the required information. Grep will become a killer command when we combined it with regular expressions. In this post, we will see how to use grep command in a basic way and then move on some advanced and rarely used options. In our next couple of posts, we will see what grep can do with the help of regular expressions.
GREP command syntax
grep [options] [searchterm] filename
or
command | grep [options] [searchterm]
Before starting to grep command examples, I used below file which contains following content.
cat file1.txt
Output:
surendra 31 IBM Chennai
Steve 45 BOA London
Barbi 25 EasyCMDB Christchurch
Max 25 Easy CMDB Christchurch
Nathan 20 Wipro Newyark
David 20 ai Newyark

SEARCH SINGLE FILE USING GREP

Example 1: Search for a word “nathan” in file1.txt
grep nathan file1.txt
You don’t get any output, as the word nathan is not there. By this type you should know grep is a case sensitive command. If you want specifically Nathan, use caps N in nathan and try once again.
Example 2: Search for exact word “Nathan”
root@linuxnix:~# grep Nathan file1.txt
 Nathan 20 Wipro Newyark
Example 3: Search for a word which has either capital or small letters in it, no confusion between nathan or Nathan. The -ifor ignoring case.
root@linuxnix:~# grep -i Nathan file1.txt
 Nathan 20 Wipro Newyark
Example 4: I suggest you always use single quotes for your search term. This will avoid confusions to gerp. Suppose if you want to search for “Easy CMDB” in a file, it will be difficult to search without single quotes. Try below examples.
Without quotes:
root@linuxnix:~# grep Easy CMDB file1.txt
 grep: CMDB: No such file or directory
 file1.txt:Barbi 25 EasyCMDB Christchurch
 file1.txt:Max 25 Easy CMDB Christchurch

WHAT GREP DID?

If you observe, you got an error stating that there is no file called CMDB. That is true; there is no such file. This output has two issues
1) Grep is considering the second word in the command as file
2) Grep is thinking just “Easy” as a search term.
Example 5: Search for exact search term using single quotes.
root@linuxnix:~# grep 'Easy CMDB' file1.txt
 Max 25 Easy CMDB Christchurch
You may get doubt why single quotes and not double quotes. You can use double quotes as well when you want to send bash variable into search term.
Example 6: Search for a shell variable in a file. My shell is NAME1 which is assigned with Nathan. See below examples of single and double quotes.
root@linuxnix:~# NAME1=Nathan
 root@linuxnix:~# grep '$NAME1' file1.txt
No output, now try with double quotes
root@linuxnix:~# grep "$NAME1" file1.txt
 Nathan 20 Wipro Newyark
 root@linuxnix:~#
See the difference? So it depends on when you want to use single quotes and double quotes. If you want to pass shell variables to grep use double quotes and remaining time always use single quotes.
Example 7: If you want to inverse your search criteria, ie, to display all the lines which do not contain your search term then use -v option. The -v option will show all the lines which do not contain your search term. Suppose list all the lines which do not contain Nathan in the file.
root@linuxnix:~# grep -v 'Nathan' file1.txt
 surendra 31 IBM Chennai
 Steve 45 BOA London
 Barbi 25 EasyCMDB Christchurch
 Max 25 Easy CMDB Christchurch
 David 20 ai Newyark
Example 8: How about getting the count of a word in a file? Search for word ‘Easy’ and tell me how many times it is there in the file.
root@linuxnix:~# grep -c 'Easy' file1.txt
 2
Example 9: Ok, finding some occurrences is okay, how about displaying line number as well for our search term? Search for Easy and display line number where that word is present.
root@linuxnix:~# grep -n 'Easy' file1.txt
 3:Barbi 25 EasyCMDB Christchurch
 4:Max 25 Easy CMDB Christchurch
Example 10: Some times you want to find exact word instead of finding an occurrence of a word sequence. Suppose you want to find is a word in a file, this will grep “this” as well as “this” contains “is” in it. Grep has a Technic to do this with the help of -w which filter out this type of stuff. Suppose I want to search for ai in our file, with standard grep it will print first and last lines as both contain ai in them. Use -w for avoiding this stuff.
With out -w option:
root@linuxnix:~# grep 'ai' file1.txt
 surendra 31 IBM Chennai
 David 20 ai Newyark
With -w option:
root@linuxnix:~# grep -w 'ai' file1.txt
 David 20 ai Newyark
Example 11: Grep has a unique feature which can print lines before, after and between for a search term. Suppose if you want to print two lines before to the search, use below example. Search for Max and below example prints two lines above search term as well. This will be handy when searching for log files.
root@linuxnix:~# grep -B 2 'Max' file1.txt
 Steve 45 BOA London
 Barbi 25 EasyCMDB Christchurch
 Max 25 Easy CMDB Christchurch
Example 12: Search for a word and print two lines after to this search
root@linuxnix:~# grep -A 2 'Barbi' file1.txt
 Barbi 25 EasyCMDB Christchurch
 Max 25 Easy CMDB Christchurch
 Nathan 20 Wipro Newyark
Example 13: Search for a word and print one line Before and after center to this search term
root@linuxnix:~# grep -C 1 'Max' file1.txt
 Barbi 25 EasyCMDB Christchurch
 Max 25 Easy CMDB Christchurch
 Nathan 20 Wipro Newyark

SEARCH MULTIPLE FILES USING GREP

Example 14: Grep has one more feature which allows searching multiple files at a time. We no need to use grep for each file. Suppose you want to search for Nathan in /root folder recursively use below example.
root@linuxnix:~# grep -r 'Nathan' /root/
 Nathan 20 Wipro Newyark
This will not display which file contain Nathan who may confuse you. We do not know from which file Nathan came. To avoid this confusion use below example.
Example 15: Search for a word and just list the file names which contain it with in a directory.
root@linuxnix:~# grep -l 'Nathan' /root/
 file1.txt
Example 16: Ok, again this command do not show the line which is again a confusion. To search multiple files and display their contents which match our search criteria use combination of -r and -n
root@linuxnix:~# grep -rn 'Nathan' /root/
 file1.txt:5:Nathan 20 Wipro Newyark
 test.txt:1:Nathan abc xyz who
Example 17: Restrict your search to specific files. Suppose search only txt files.
root@linuxnix:~# grep -rn 'Nathan' *.txt
 file1.txt:5:Nathan 20 Wipro Newyark
 test.txt:1:Nathan abc xyz who
Example 18: Some times you may see below error when opening log files.
Binary file <input-file> matches.
This error is because grep thinks it is a binary file. But our log files are never a binary one. This error is due to the null value found in the log file. To read such files, you have to use -a which tell grep to read this file as a binary file.
grep -a xyz abc.txt
In our next two posts, we will see how to use grep with regular expressions.