Backup entire filesystem using rsync

<yambe:breadcrumb self="Backup entire filesystem using rsync">Rsync|Rsync</yambe:breadcrumb>

Converting physical to virtual using rsync

To backup entire filesystem basically root (/) using rsync first update the version of rsync on both machines to at least 3.0. This is necessary so that we can use incremental recursive rsync without which we would most likely get "too many files error" or run out of memory.

During upgrade make sure that the new rsync becomes default version by running 'rsync --version'. In case new version has not overwritten older version use 'whereis rsync' to find both the current version and new version installed. Then again configure rsync source using something like './configure --prefix=/usr' in case the older rsync file is in /usr/bin. Then again 'make' and 'make install'. Verify that the default rsync now is at least 3.0.

To backup data after version on both ends is 3.0 use

rsync -avzrxHAX --partial --delete --delete-during /home root@10.4.2.20:/mnt/virtual_hosts/www_server/server_root

Here,

• -a is used so that all timestamps are preserved.

  This is important to preserver ownership etc. too

• -v is used for verbose so that we know where we are in copying.
• -z is used for compression.
• -r is used for recursive copying.

  In version 3.0 and later it is incremental so that we do not face memory issues.

• -x is to stay on current filesystem.

  This is extremely important when copying root '/' so that '/proc', '/dev' and other such filesystems do not cause any problem

• -H is for making sure hard-links on source are turned into hard-links on destination
• -A is for preserving ACLs
• -X is for preserving extended attributes
• --partial is to keep partial files and continue from same location if rsync gets interrupted in between. It is better to use --partial than using --inplace.
• --delete is to delete files which are not present on source but present on destination
• --delete-during is to support incremental recursive copying by deleting files during transfer.

After this if we have other filesystems like /bin, /lib etc. then we can copy them using something like

time rsync -avzrxHAX --delete --delete-during /bin root@10.4.2.20:/mnt/virtual_hosts/www_server/server_root
time rsync -avzrxHAX --delete --delete-during /lib root@10.4.2.20:/mnt/virtual_hosts/www_server/server_root
time rsync -avzrxHAX --delete --delete-during /lib64 root@10.4.2.20:/mnt/virtual_hosts/www_server/server_root
time rsync -avzrxHAX --delete --delete-during /opt root@10.4.2.20:/mnt/virtual_hosts/www_server/server_root
time rsync -avzrxHAX --delete --delete-during /root root@10.4.2.20:/mnt/virtual_hosts/www_server/server_root
time rsync -avzrxHAX --delete --delete-during /sbin root@10.4.2.20:/mnt/virtual_hosts/www_server/server_root
time rsync -avzrxHAX --delete --delete-during /usr root@10.4.2.20:/mnt/virtual_hosts/www_server/server_root
time rsync -avzrxHAX --delete --delete-during /var root@10.4.2.20:/mnt/virtual_hosts/www_server/server_root

Warning: Do not use 'rsync -avzrxHAX --delete --delete-during /home root@10.4.2.20:/mnt/virtual_hosts/www_server/server_root/home' as that would copy home folder inside home folder. Also do not use 'rsync -avzrxHAX --delete --delete-during /home/* root@10.4.2.20:/mnt/virtual_hosts/www_server/server_root/home' as delete may not work properly. Read man page carefully and verify lot of times in middle when copying disks.

Note: Do not pull this stunt with /etc, /dev or /boot etc. where there are fstab files, lvm configuration, kernels, device files, grub configuration etc. as any of these can break the virtual machine, especially if it is paravirtualized.

Now use '-n' option of rsync for other partitions and manually choose which files / directories of /boot, /dev or /etc should be synced. You might want to backup '/' partition using dd or by copying entire virtual image before attempting configuration sync.

Syncing /etc

Create a empty file with name exclude.txt and first as suggested above run rsync with -n option and see which files should not be rsynced. The example command is

rsync -avzrxHAXn --partial --delete --delete-during --exclude-from exclude.txt  /etc root@10.4.2.20:/mnt/virtual_hosts/www_server/server_root | less

Now add list of files which you do not want to rsync and run the above command again. Typical list is (contents of exclude.txt)

/etc/X11*
/etc/fstab
/etc/NetworkManager*
/etc/lvm*
/etc/makedev.d*
/etc/rc.d/init.d/rawdevices
/etc/rc.d/init.d/fcoe
/etc/sysconfig/grub
/etc/sysconfig/xenhotplug
/etc/sysconfig/rawdevices
/etc/sysconfig/raid-check
/etc/sysconfig/network-scripts*
/etc/sysconfig/networking*
/etc/sysconfig/udev-stw
/etc/udev*
/etc/xen*

Then if you feel the output with '-n' and less combination is safe. Use

rsync -avzrxHAX --partial --delete --delete-during --exclude-from exclude.txt  /etc root@10.4.2.20:/mnt/virtual_hosts/www_server/server_root

as in any other case, having backups and restore points is always a good idea.

Case studies

www.iiit.ac.in

Above method was used to convert www.iiit.ac.in using CentOS-5 from physical to virtual machine and things worked perfectly. All databases, users, server configuration, etc. got migrated successfully.

pascal.iiit.ac.in

Similar method was used to migrate pascal.iiit.ac.in using Debian Lenny 5.0 from physical to virtual machine. Things worked perfectly in this case too. Various scripts used for this migration were:

update_root.sh

cd /mnt/pascal
time rsync -avzrxHAX --partial --delete --delete-during root@10.4.7.64:/mnt/pascal/root/bin/ root/bin/
time rsync -avzrxHAX --partial --delete --delete-during root@10.4.7.64:/mnt/pascal/root/lib/ root/lib/
time rsync -avzrxHAX --partial --delete --delete-during root@10.4.7.64:/mnt/pascal/root/root/ root/root/
time rsync -avzrxHAX --partial --delete --delete-during root@10.4.7.64:/mnt/pascal/root/sbin/ root/sbin/
time rsync -avzrxHAX --partial --delete --delete-during root@10.4.7.64:/mnt/pascal/root/usr/ root/usr/
time rsync -avzrxHAX --partial --delete --delete-during root@10.4.7.64:/mnt/pascal/root/var/ root/var/

update_home.sh

cd /mnt/pascal
time rsync -avzrxHAX --partial --delete --delete-during root@10.4.7.64:/mnt/pascal/home/ home/

update_etc_boot.sh

cd /mnt/pascal
time rsync -avzrxHAX --partial --delete --delete-during --exclude-from etc_exclude.txt root@10.4.7.64:/mnt/pascal/root/etc/ root/etc/
time rsync -avzrxHAX --partial --delete --delete-during --exclude-from boot_exclude.txt root@10.4.7.64:/mnt/pascal/root/boot/ root/boot/
#time rsync -avzrxHAXn --partial --delete --delete-during --exclude-from dev_exclude.txt root@10.4.7.64:/mnt/pascal/root/dev/ root/dev/

Note that:

• /dev was not migrated and things worked perfectly.
• etc_exclude.txt contained

fstab
mtab
network/interfaces

• boot_exclude.txt was empty