Rsync

 

RSYNC

 

rsync is used to sync files on remote and local machines. You can use rsync to copy files to remote systems and back again, or to make backups to locally mounted hard drives.

As an effective tool, rsync is great at moving data from one machine to another. If used incorrectly, though, it can also be an effective tool at overwriting data or deleting file.

The first time you get ready to run an rsync command, particularly the first few times out, you might want to make use of the -n option (also --dry-run if you prefer the more verbose option name). This is to run through a trial run with no actual changes made.

rsync is a replacement for rcp (and scp) that has many more features. It uses the "rsync algorithm" which provides a very fast method for remote files into sync. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand.

Features

· Support for copying links, devices, owners, groups and permissions

· Exclude and exclude-from options similar to GNU tar

· A CVS exclude mode for ignoring the same files that CVS would ignore

· Does not require root privileges

· Pipelining of file transfers to minimize latency costs

· Support for anonymous or authenticated rsync servers (ideal for mirroring)

Important features of rsync

Speed: First time, rsync replicates the whole content between the source and destination directories. Next time, rsync transfers only the changed blocks or bytes to the destination location, which makes the transfer really fast.

Security: rsync allows encryption of data using ssh protocol during transfer.

Less Bandwidth: rsync uses compression and decompression of data block by block at the sending and receiving end respectively. So the bandwidth used by rsync will be always less compared to other file transfer protocols.

Privileges: No special privileges are required to install and execute rsync

Syntax

$ rsync options source destination

When to Use rsync

Use rsync when you want to do unattended backups, or complex syncing operations. It can do a lot of things that GUI tools can't, or don't do well -- like syncing files from your home directory to a locally attached USB disk.

Using rsync

Here's a basic example to copy files from a home directory to a USB drive

rsync -avh --exclude="*.iso" /home/user/bin/ /media/diskid/user_backup/

As you can see, you can combine options after one dash. The -avhz following rsync is the same as using -a -v -h -z, but much easier to type.

Let's break that down. You have the rsync command followed by four options and one argument. The options are archive (-a), which tells rsync to copy files recursively and to preserve group and user ownership when it copies files. This is generally a good idea to include.

The verbose option (v) tells rsync to print to the terminal what it's doing in greater detail. I like to use this when I'm testing an rsync command before adding it to a script. When you use this, rsync will print the list of changed files that it's sending and the time it takes, the amount of data copied over, etc. If you omit -v rsync will just print a short message about the number of bytes sent, size, and so on. But you won't see a detailed file list. If you'd like to do away with this entirely, you can use the -q option instead to "quiet" rsync -- which might be useful in some cases when writing scripts for rsync.

The human-readable option (h) directs rsync to produce slightly more readable output. Without -h, rsync just shows bytes, which might not be terribly useful. Note that -h isn't the same as --h: rsync uses --h for its help option.

Finally, this command directs rsync to ignore files that end with .iso. I download milestone and release candidate images of openSUSE pretty often, and don't see any need to maintain an offsite or local backup of them since I can always get them again if needed.

Unattended Backups

Copying files to a locally attached disk can be useful, but I also like to do a backup to a remote machine right before I travel so that I have access to my most important files even if my laptop dies or is stolen. It hasn't happened yet, but I attribute that to doing backups beforehand. If there's anything one should be superstitious about, it's backing up files. If you believe nothing else, believe that files that are not backed up will spontaneously delete themselves if only to spite you.

But make it easy on yourself. Odds are you don't feel like sitting at your computer entering a password every time rsync opens a new connection to a remote host. You can run unattended backups by setting up an SSH key at the remote host. You can also SSH into a remote host without needing to type the password each time too. If you haven't set this up previously, it's very easy. Open a terminal and run:

ssh-keygen -t dsa

This will create a public and private SSH key for you. You'll be prompted twice for a passphrase. Just hit Enter each time. Now you need to get the public key to the remote host. For this you want to use ssh-copy-id:

ssh-copy-id -i ~/.ssh/id_dsa.pub user@host

This is a script that will copy your public key to a remote host. It may not work on systems that have the remote shell disabled. For instance, I've had no success getting it to work with rsync.net. In those cases, you need to copy your public key to ~/.ssh/authorized_keys on the remote system.

Now it's time to tell rsync to use SSH, and point it at a remote host. To do this, use the -e SSH option, like so:

rsync -avze ssh /home/user/directory/ user@remote.host.net:home/user/directory/

Here I tacked on the remote shell (-e) option to -avz and told rsync that it should use SSH. Note that rsync can also use other methods like rsh, but in practice I've never seen rsync used with anything but SSH.

When specifying the directories, be clear on whether you include the trailing slash. In the above example, /home/user/directory would not be the same as /home/user/directory/. The additional slash tells rsync to copy the contents but not the directory itself. Without it, rsync will also create the directory.

What's the z option do? It tells rsync to compress the data sent.

Note that you can do this the other way too. If you want to back up a remote system to the local system, just swap the remote and local host targets, like so:

rsync -avze ssh user@remote.host.net:home/user/directory/ /home/user/directory/

Other Important Options

If you want to maintain an exact copy of a directory, you can add the --delete option to your rsync command. This will compare the destination and delete any files that aren't present on the local system.

The advantage of using --delete is that you keep a more or less identical copy of the two filesystems. The disadvantage is that if you delete a local file accidentally and you have an rsync backup going at regular intervals, you will lose the opportunity to recover files from the remote backup.

The flip side of --delete is --backup and --backup-dir or --suffix. If a file already exists on the host, rsync will back it up before it's overwritten with the file being transferred. I don't use these much myself, but it might be useful in some cases.

When running an attended copy, I like to use --progress, so I can see what's going on as rsync copies each file. When doing unattended backups, --progress isn't terribly useful.

Using Rsync Itself

Now on to actually using, or initiating an rsync transfer with rsync itself. It's the same binary as the daemon, just without the "--daemon" flag. It's simplicity is a virtue. I'll start with a commandline that I use in a script to synchronize a Web tree below.

rsync --verbose --progress --stats --compress --rsh=/usr/local/bin/ssh \

--recursive --times --perms --links --delete \

--exclude "*bak" --exclude "*~" \

/www/* webserver:simple_path_name

Let's go through it one line at a time. The first line calls rsync itself and specifies the options "verbose," progress" and "stats" so that you can see what's going on this first time around. The "compress" and "rsh" options specify that you want your stream compressed and to send it through ssh (remember from above?) for security's sake.

The next line specifies how rsync itself operates on your files. You're telling rsync here to go through your source pathname recursively with "recursive" and to preserve the file timestamps and permissions with "times" and "perms." Copy symbolic links with "links" and delete things from the remote rsync server that are also deleted locally with "delete."

Now we have a line where there's quite a bit of power and flexibility. You can specify GNU tar-like include and exclude patterns here. In this example, I'm telling rsync to ignore some backup files that are common in this Web tree ("*.bak" and "*~" files). You can put whatever you want to match here, suited to your specific needs. You can leave this line out and rsync will copy all your files as they are locally to the remote machine. Depends on what you want.

Finally, the line that specifies the source pathname, the remote rsync machine and rsync "path." The first part "/www/*" specifies where on my local filesytem I want rsync to grab the files from for transmission to the remote rsync server. The next word, "webserver" should be the DNS name or IP address of your rsync server. It can be "w.x.y.z" or "rsync.mydomain.com" or even just "webserver" if you have a nickname defined in your /etc/hosts file, as I do here. The single colon specifies that you want the whole mess sent through your ssh tunnel, as opposed to the regular rsh tunnel. This is an important point to pay attention to! If you use two colons, then despite the specification of ssh on the commandline previously, you'll still go through rsh. Ooops. The last "www" in that line is the rsync "path" that you set up on the server as in the sample above.

How to use rsync for transferring files under Linux or UNIX

rsync is a free software computer program for Unix and Linux like systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. An important feature of rsync not found in most similar programs/protocols is that the mirroring takes place with only one transmission in each direction.

Always use rsync over ssh

Since rsync does not provide any security while transferring data it is recommended that you use rsync over ssh . This allows a secure remote connection. Now let us see some examples of rsync.

rsync command common options

--delete : delete files that don't exist on sender (system)

-v : Verbose (try -vv for more detailed information)

-e "ssh options" : specify the ssh as remote shell

-a : archive mode

-r : recurse into directories

-z : compress file data

Task : Copy file from a local computer to a remote server

Copy file from /www/backup.tar.gz to a remote server called openbsd.nixcraft.in

$ rsync -v -e ssh /www/backup.tar.gz jerry@openbsd.nixcraft.in:~

Please note that symbol ~ indicate the users home directory (/home/jerry)

Task : Copy file from a remote server to a local computer

Copy file /home/jerry/webroot.txt from a remote server openbsd.nixcraft.in to a local computer /tmp directory:

$ rsync -v -e ssh jerry@openbsd.nixcraft.in:~/webroot.txt /tmp

Task: Synchronize a local directory with a remote directory

$ rsync -r -a -v -e "ssh -l jerry" --delete openbsd.nixcraft.in:/webroot/ /local/webroot

Task: Synchronize a remote directory with a local directory

$ rsync -r -a -v -e "ssh -l jerry" --delete /local/webroot openbsd.nixcraft.in:/webroot

Task: Mirror a directory between my "old" and "new" web server/ftp

You can mirror a directory between my "old" (my.old.server.com) and "new" web server with the command (assuming that ssh keys are set for password less authentication)

$ rsync -zavrR --delete --links --rsh="ssh -l vivek" my.old.server.com:/home/lighttpd /home/lighttpd

Synchronize Two Directories in a Local Server

To sync two directories in a local computer, use the following rsync -zvr command.

$ rsync -zvr /var/opt/installation/inventory/ /root/temp

In the above rsync example:

-z is to enable compression

-v verbose

-r indicates recursive

Preserve timestamps during Sync using rsync -a

rsync option -a indicates archive mode. -a option does the following,

Recursive mode

Preserves symbolic links

Preserves permissions

Preserves timestamp

Preserves owner and group

$ rsync -azv /var/opt/installation/inventory/ /root/temp/

Synchronize Only One File

rsync -v /var/lib/rpm/Pubkeys /root/temp

Synchronize Files From Local to Remote

rsync allows you to synchronize files/directories between the local and remote system.

$ rsync -avz /root/temp/ techrock@192.168.200.10:/home/techrock/temp/

While doing synchronization with the remote server, you need to specify username and ip-address of the remote server. You should also specify the destination directory on the remote server. The format is username@machinename:path

As you see above, it asks for password while doing rsync from local to remote server.

Sometimes you don’t want to enter the password while backing up files from local to remote server. For example, If you have a backup shell script, that copies files from local to remote server using rsync, you need the ability to rsync without having to enter the password.

Synchronize Files From Remote to Local

When you want to synchronize files from remote to local, specify remote path in source and local path in target as shown below.

$ rsync -avz techrock@192.168.200.10:/var/lib/rpm /root/temp

Remote shell for Synchronization

rsync allows you to specify the remote shell which you want to use. You can use rsync ssh to enable the secured remote connection.

Use rsync -e ssh to specify which remote shell to use. In this case, rsync will use ssh.

$ rsync -avz -e ssh techrock@192.168.200.10:/var/lib/rpm /root/temp

Do Not Overwrite the Modified Files at the Destination

In a typical sync situation, if a file is modified at the destination, we might not want to overwrite the file with the old file from the source.

Use rsync -u option to do exactly that. (i.e do not overwrite a file at the destination, if it is modified). In the following example, the file called Basenames is already modified at the destination. So, it will not be overwritten with rsync -u.

$ rsync -avzu techrock@192.168.200.10:/var/lib/rpm /root/temp

Synchronize only the Directory Tree Structure (not the files)

Use rsync -d option to synchronize only directory tree from source to the destination. The below example, synchronize only directory tree in recursive manner, not the files in the directories.

$ rsync -v -d techrock@192.168.200.10:/var/lib/ .

View the rsync Progress during Transfer

When you use rsync for backup, you might want to know the progress of the backup. i.e how many files are copies, at what rate it is copying the file, etc.

rsync –progress option displays detailed progress of rsync execution as shown below.

$ rsync -avz --progress techrock@192.168.200.10:/var/lib/rpm/ /root/temp/

Delete the Files Created at the Target

If a file is not present at the source, but present at the target, you might want to delete the file at the target during rsync.

In that case, use –delete option as shown below. rsync delete option deletes files that are not there in source directory.

rsync -avz --delete techrock@192.168.200.10:/var/lib/rpm/ .

Do not Create New File at the Target

If you like, you can update (Sync) only the existing files at the target. In case source has new files, which is not there at the target, you can avoid creating these new files at the target. If you want this feature, use –existing option with rsync command.

$ rsync -avz --existing root@192.168.1.2:/var/lib/rpm/ .

Include and Exclude Pattern during File Transfer

rsync allows you to give the pattern you want to include and exclude files or directories while doing synchronization.

$ rsync -avz --include 'P*' --exclude '*' techrock@192.168.200.10:/var/lib/rpm/

In the above example, it includes only the files or directories starting with ‘P’ (using rsync include) and excludes all other files. (using rsync exclude ‘*’ )

Do Not Transfer Large Files

You can tell rsync not to transfer files that are greater than a specific size using rsync –max-size option.

$ rsync -avz --max-size='100K' techrock@192.168.200.10:/var/lib/rpm/ /root/temp/

max-size=100K makes rsync to transfer only the files that are less than or equal to 100K. You can indicate M for megabytes and G for gigabytes.

Transfer the Whole File

One of the main feature of rsync is that it transfers only the changed block to the destination, instead of sending the whole file.

If network bandwidth is not an issue for you (but CPU is), you can transfer the whole file, using rsync -W option. This will speed-up the rsync process, as it doesn’t have to perform the checksum at the source and destination.

# rsync -avzW techrock@192.168.200.10:/var/lib/rpm/ /root/temp 

Backup Scripts

du -csh /filer/legato_index/

1. rysnc.sh

echo "******************** Timestamp *************************" >> /var/log/rsync_mtree.log

date >> /var/log/rsync_mtree.log

echo "******************** Timestamp *************************" >> /var/log/rsync_mtree.log

/usr/bin/rsync  -ae "/usr/bin/ssh -i /root/.ssh/RSA_IIL" /home/rediff root@172.16.1.20:/backup

ErrorCode=`/bin/echo $?`

if [ $ErrorCode -ne 0 ]

then

if [ $ErrorCode -ne 24 ]

then

#    /bin/echo "Rsync folder /home/rediff : successful" | /bin/mail -s "Rsync Successful :rediff.com" support@netmagicsolutions.com

#else

   /bin/echo "Rsync Completed with Error Code (May not be successful, Check errorcode): $ErrorCode" | /bin/mail -s "Rsync completed :rediff.com" support@techrock.com

fi

fi

2.  vi script.sh

#!/bin/bash

MASTER="production1"

DIR="/archivelog/oracle/"

LDIR="/archivelog/oracle/"

SSH="/usr/bin/ssh"

rsync –avtpogv –-rsh:$SSH $MASTER:$DIR $LDIR > /archivelog/backup/copyarchive.log

Note:-

production1 :- Production server name

DIR = Production server archive log path

LDIR= DR server archive log Path

SSH= Service

Rsync:- Its used for copy archive logfile from prod.

Save script.sh on production2/archivelog/backup/

To avoid password prompt you need to setup ssh-keys, so on production2(DR) type following commands

Code:

ssh-keygen -t rsa

scp .ssh/id_rsa.pub Username@Hostname:.ssh/authorized_keys2

Note:-

1. ssh-keygen -t rsa :- This command will generated one Key,This key will go into /home/.ssh/id_rsa.pub(.SSH is hidden folder)
2. Run second command give production username and hostname.

Now you can login to production1 from production2 without password.

Now setup cron job using following script on production2

Crontab –e

*/30 * * * *  /archivelog/backup/script.sh

Script.sh will run in every 30 minutes

• replace production1 with FQDN of production1 server
• /archivelog/backup/copyarchive.log keeps log of the latest run script on production2; earlier logs will be replaced when the script is run next time.

3. #!/bin/bash

for folder in /etc /usr/local /home /var/gmail /var/yatra /var/shailesh /var/rediff /export/home/yahoo /var/msn

do

#/usr/bin/rsync  -aze "/usr/bin/ssh -i /root/.ssh/rsyncbkp" $folder root@1.2.3.4:/home/gmail/

/usr/bin/rsync  -aze "/usr/bin/ssh -i /root/.ssh/rsyncbackup.key" $folder --delete root@1.2.3.4:/backup/

ErrorCode=`/bin/echo $?`

if [ $ErrorCode -eq 0 -o $ErrorCode -eq 24 ]

then

    /bin/date >> /var/log/rsync.log

    /bin/echo folder: $folder rsynced successfully >> /var/log/rsync.log

else

    /bin/echo "Rsync $folder with Error Code (May not be successful): $ErrorCode" | /bin/mail -s "Rsync: Gmail " support@techrock.com

    /bin/date >> /var/log/rsync.log

    /bin/echo folder: $folder rsynced with Errorcode: $ErrorCode >> /var/log/rsync.log

fi

done