mirroring

You are currently browsing articles tagged mirroring.

If you are ever in need of copying large amounts of data from one server to another, You have a few options:

Option #1: Download data to local machine then re-upload files back to your other server (stupid)
Option #2: Use secure-copy (scp) to copy the necessary files securely from target server (not bad)
Option #3: Just use rsync to make an exact mirror/copy of your data from target server (best)

Actually, even though many folks use scp (Option #2) to accomplish our task – it’s actually a bad idea to do so when you have a large amount of data to copy.

I regularly mirror/copy over files in excess of 100GB+ and if I use scp and lose the connection mid-stream, I am essentially fucked…and I will have to restart the entire process all over again.

If you use rsync however (Option #3:), You won’t have such problem. In the event of any interruptions – you could simply repeat the command again and it will pickup where it left off and clean up any differences with ease.

So the verdict is: If you are copying over a few small files, use scp. If you need to mirror/backup large systems/files, use rsync. Got that buttercup? Good. Let’s move on.

When using rsync, we will be using the following options:

-r recurse into directories
-v Increase verbosity
–progress Show progress during transfer

So the command will basically look like this:

rsync -r -v --progress SER@REMOTE-HOST:~/address/to/remote/source/ /home/destination/

I personally use the following:

rsync -avvr --progress USER@REMOTE-HOST:/address/to/remote/source/ /home/local/mirror/

This will gather an index of all the files needed to copy (from the source server) and will make an exact copy (on your destination server) like you wanted.

That’s essentially it.

#

Troubleshooting:

If you get an error like this when attempting to use rsync:

Zsh: No Matches Found

..You will need to set zsh option to NOMATCH. This can be done by running the following command before executing our rsync command above:

zsh -3

*-3 is actually the option for NOMATCH. See here for more details.

You can also avoid entering a password each time when using rsync and setup automated backups between remote servers using crontab.