Synchronizing files from one server to another is quite awesome. You can
use it for backups, for keeping web servers in sync, and much more. It’s
fast and it doesn’t take up as much bandwidth as normal copying would. And the
best thing is, it can be done with only 1 command. Welcome to the wonderful
world of rsync.
On most modern Linux distributions you will find rsync comes
preinstalled. If that’s not the case, just install it with your package
manager. On Ubuntu this would look like:
aptitude -y install rsync
Simple - one command
Let’s copy our local /home/kevin/source to /home/kevin/destination which
resides on the server: server.example.com:
-a archive, preserves all attributes like recursive ownership, timestamps, etc
-z compress, saves bandwidth but is harder on your CPU so use it for slow/expensive connections only
--progress shows you the progress of all the files that are being synced
--size-only compare files based on their size instead of hashes (less CPU, so faster)
Note that this sync excludes hidden files since it uses the bash *. If you want to include
hidden files, write the source like this: /home/kevin/source/ and remove the trailing slash
from the destination like so: /home/kevin/destination.
Well, that’s it! But read on if you want to learn how to automate this.
Advanced - automatic syncing with SSH keys
Alright so syncing files on Linux is pretty easy. But what if we want to
automate this? How can we avoid that rsync asks for a password every time?
There are different ways to go about this, but the one I mostly use is
installing SSH keys. By installing your SSH key on the destination
server, it will recognize you in the future and permit instant access. So this
way we can automate the synchronization with rsync.
I’ve written another article explaining on setting up SSH keys. It also
includes a script that can do all the work for you.
Did it work?
Open a terminal and type:
It should not ask you for any password. Great! this means we can also run
rsync directly without logging in! If you need more in depth information on
this, I wrote an article on logging in automatically with SSH keys.
Let’s create a sync script
So now just create a script /root/bin/syncdata.bash
That’s it! New files are automatically updated @
server.example.com:/home/kevin/destination/ every hour. Files that are
deleted from /home/kevin/source/* are also deleted at the destination, thanks
to the --delete parameter.
Some extra rsync command line options
Some extra arguments that might come in handy customizing your synchronization
--delete delete files remotely that no longer exist locally
--dry-run show what would have been transferred, but do not transfer anything
--max-delete=10 don’t delete more than 10 files in one run, safety precaution
--delay-updates put all updated files into place at transfer’s end, very useful for live systems
--compress-level=9 explicitly set compression level 9. 0 disabled compression
--exclude-from=/root/sync_exclude specifies a /root/sync_exclude that contains exclude patterns (one per line). filenames matching these patterns will not be tranfered
--bwlimit=1024 This option specifies a maximum transfer rate of 1024 kilobytes per second.
Of course you should really be carefull where and when to install ssh
keys, because if one machine is comprimised, it’s very easy for a cracker to
hop to the next system without logging in. So choose wisely when to use this
Keys are user user specific. So if you’re going to run programs as root
that need to automatically login to systems, you must also install the key as
These were imported from my old blog. Please use disqus below for new comments
on 2012-09-27 15:27:46
do you have a page for windows to windows synchronization through Cygwin and Rsync?
Kiran S Murthy
on 2012-09-11 09:00:48
Thank you, Simple and Clear
on 2012-09-05 10:51:18
Thanks man! you saved my day! I’m migrating a mail server and this made moving the inboxes a lot easier.
on 2012-03-01 08:22:24
From the man page, actually:
–delete delete extraneous files from dest dirs
So if you are doing a &
quot; instead of &
quot; –delete should work at the local end.
on 2011-11-15 14:22:17
Please put some ‘restarting crontab’ for who want some immediate effect.
on 2011-11-13 21:12:57
How do you know if your rsync fails or gives some kind of output?
on 2011-09-18 23:40:56
Really useful piece. The problem of trying to sync without hashes for huge directories has flummoxed me for ages. Glad you’ve helped me find a way around it by comparing on size only.
on 2011-09-16 04:57:23
That’s so helpful because i’m looling for a solution to auto-synchronize the file system between web servers. I’ll try it. Thank so much
on 2011-06-27 13:17:40
–delete does not work because you have to use it like:
rsync [OPTIONS] $SOURCEDIR/ $DESTDIR
so without the * for the source dir and without the ‘/’ for the destination dir
on 2011-06-15 10:50:45
very good tutorial!!!!
Just 1 question,
Lets say the remote server doesn’t allow ssh on port 22 but rather on port 2255, how do you specify the correct port for rsync to use?
on 2011-06-07 04:33:27
A better automation option to crontab would be inotifywait for linux, and building the script around it.
on 2011-02-06 23:31:32
Thanks a lot, easy to use , and implement… –delete seems not to work thouh
on 2011-01-30 11:36:49
Very good article. Well organized and well written.
Thank you very much for sharing.
on 2011-01-20 11:08:06
Thanks a lot ! Very good article…
on 2010-08-15 01:25:39
Thanks… Its great
on 2010-08-12 12:36:27
@ Arie: That's the kind of advice you should pay for. You could have a look at my crontab article though.
on 2010-07-08 22:38:36
I'm new to all this rsync/linux/cron staff.
I've been asked to keep two web servers (one master and the other replica) synchronized on a daily basis using rsync.
Can you please explain to me how can I automate rsync to execute once a day, every day of the week (or month) at the same time with an example please?
on 2010-06-29 12:48:01
@ Mario: You are right. Old article this one. Fixed tho, thanks.
on 2010-06-18 11:13:42
When using the -a option the -r option is unnecessary.
So -raz is the same as -az,
on 2009-02-21 12:46:31
@ Rajendran: Well it's all in the article, really. On .55 you do something like:
But if you really don't know what you're doing, maybe it's a bit too soon to be rsyncing.
on 2009-02-18 04:06:28
i want to sync data from my fileserver (192.168.89.55) to my backup server (192.168.89.100) through network. Fileserver running ubuntu 6.06 server and backup server running ubuntu 8.04 LTS. So, kevin can u give the step to run the sync data for this…..coz i'm new to LINUX and still learning on this…….
on 2009-02-18 03:59:37
i want to sync data from my fileserver to my backup server through network Both also using Ubuntu
on 2009-02-11 02:24:15
@ Jeff: You are absolutely right and I have just forgotten to update the article. I will soon come with a new article on some unknown rsync options to make up ;) stay tuned!
on 2009-02-07 06:49:00
One thing to note: the archive (-a) option for rsync includes the recursive (-r) option, so the added -r on the command line isn't necessary.
on 2008-11-03 10:18:35
@ Steve: Thanks again, I've replaced the outdated text with a link to the other article to avoid duplication.
on 2008-10-22 19:34:50
The ssh installation section should explain how to send keys to a firstname.lastname@example.org other than the root user.
Your script can do this because it allows input of a second argument (the user name) after the server name. I didn't realize this and used the email@example.com single argument. It doesn't work because the script prepends \&
quot; thus trying to connect to root@firstname.lastname@example.org
You don't mention the use of two arguments however.
Properly a user should enter:
I wrote about this also on the ssh page.
on 2008-10-20 12:59:23
great article.. espesialy the automation..i want to give it a try later for my backup.
i love rsync too.. great toolls..
on 2008-01-02 18:29:00
rsnapshot is a great tool for simplifying typical backup scenarios using rsync. With rsnapshot, you can create incremental backups, run remote programs before/after backups, etc.
Also, you can drop your key on the remote server using one command. As mentioned on http://www.bytejar.com/
# replace user@host with the remote user and host you want
# the first time run, this will require you authenticate as user@host.
# subsequent times will be password free.
# this requires that you use a Bourne-derivative shell and GNU mkdir
on 2007-10-28 23:16:13
@ Logan: Maybe a rights issue? Otherwise type rsync –help to see if the mac version even supports the –delete option. Maybe it's called differently, I don't really know mac.
on 2007-10-28 19:04:05
I'm using this command on the OSX Terminal program and everything works except the –delete parameter. I've tried with files and directories and it just doesn't want to delete them.
on 2007-10-14 12:45:14
maybe try adding an asterisk to the path you want to exclude like this: /home/jerry/test-rsync/exclude/*
on 2007-10-13 08:52:46
I am trying to synchronize
except this folder in
which will include all its subfolders and files.
I have tried my command but still the folder \&
quot; gets synchronized.
I am thinking could it be my badly written command or wrong parameter placement.
on 2007-10-12 11:33:11
@ Jerry: In this case your excluding every file/dir that starts with /home/[user]/test-rsync/exclude