Linux :: RSYNC command – sync the data between two directory

I have to sync up the two directory many times on Unix boxes.The simple steps would be to delete the directory from destination and take a backup on source destination  and copy it to the destination directory.

It was good for small directories but This process will consume lot of time  as directory size grow up.

I searched around and find that The fastest way to do this process would be to use RSYNC utilty

It uses ‘rsync algorithm’ which provides a very fast method for syncing the directories or filesystems. An important feature of rsync is that the mirroring takes place with only one transmission in each direction and which is not available in other similar programs.

Rsync’s default port is 873 and its an opensource software. Rsync is available for Unix,Linux and windows operating systems. You can freely download rsync source code from rsync.samba web portal.

Key features of RSYNC command;

Support for copying links, devices, owners, groups and permissions
Exclude and exclude-from options similar to GNU tar
A CVS exclude mode for ignoring the same files that CVS would ignore
Does not require root privileges
Pipelining of file transfers to minimize latency costs
Support for anonymous or authenticated rsync servers (ideal for mirroring)

I would be giving the basics on RSYNC and how to use it effectively to sync the directories/code

How to use RSYNC – sync the data between two directory

SOURCE_PATH=’/u900/oracle/’

SOURCE_SERVER=’myserver1′
DESTINATION_PATH=’/u900/oracle-bck/’
DESTINATION_HOST=’myserver2′
DESTINATION_USER=’oracle’

rsync -av –rsh=ssh $SOURCE_PATH $DESTINATION_USER@$DESTINATION_HOST:$DESTINATION_PATH

Here

-a option is called the archive option,It means following things

Recursive mode
Preserves symbolic links
Preserves permissions
Preserves timestamp
Preserves owner and group
-v  Verbose option

This command once run will ask for the password of oracle user on Destination machine.Once password is given it will sync the change files in the directory to the destination server

If u need to enable this sync through a automated job, then this script should be working without prompting for password.

We  can password less ssh setup between the machine to achieve it

How to Enable Password less Authentication with SSH between two servers

Automated script would be  like
$ cat rsync_oracle.sh
#!/bin/ksh

SOURCE_PATH=’/u900/oracle/’

SOURCE_SERVER=’myserver1′
DESTINATION_PATH=’/u900/oracle-bck/’
DESTINATION_HOST=’myserver2′
DESTINATION_USER=’oracle’

LOGFILE=’rsync_oracle.log’
echo $’\n\n’ >> $LOGFILE
rsync -av –rsh=ssh $SOURCE_PATH $DESTINATION_USER@$DESTINATION_HOST:$DESTINATION_PATH 2>&1 >> $LOGFILE
echo “Sync Completed at:`/bin/date`” >> $LOGFILE

$chmod 700 rsync_oracle.sh

The  same  command of RSYNC can be used in the same server also

SOURCE_PATH=’/u900/oracle/’

SOURCE_SERVER=’myserver1′
DESTINATION_PATH=’/u900/oracle-bck/’
DESTINATION_HOST=’myserver1′

rsync -av  $SOURCE_PATH $DESTINATION_PATH

Some more points to remember

1)If the directories contain mulitple owner,then I would suggest doing it using root user

2) We can delete the files at destination site which are deleted on source site by adding –delete in RSYNC

rsync -av –rsh=ssh –delete $SOURCE_PATH $DESTINATION_PATH

3) it can also be acheived using RSYNC deamon which is configured through /etc/rsyncd.conf on Solaris.But it is not necessary to use that.

Benefits of RSYNC command

1) First time, rsync replicates the whole content between the source and destination directories. Next time, rsync transfers only the changed blocks or bytes to the destination location, which makes the transfer really fast.
2) We can use ssh protocol for rsync which allows encryption of data during transfer.So it is quite secure
3) Less Bandwidth: rsync uses compression and decompression of data block by block at the sending and receiving end respectively. So the bandwidth used by rsync will be always less compared to other file transfer protocols.

Some Practical RSYNC Command

1.   Synchronizing the directories on the local server

rsync -av  /app/src/ /app1/dest/

2.  Synchronizing the directories on the remote  server

rsync -av  /app/src/ tech@remotehost:/app1/dest/

3. Synchronizing the  directory structure on the remote  server

rsync -v  -d  /app/src/ tech@remotehost:/app1/dest/

4. How to Include and Exclude Pattern during File Transfer

rsync -av  –exclude ‘U*’  /app/src/ tech@remotehost:/app1/dest/

The above command exclude any file starting with U

5. How to delete files at the destination

rsync -av –rsh=ssh –delete /app/src/ tech@remotehost:/app1/dest/