stevepedwards.com/DebianAdmin linux mint IT admin tips info

Cool Command #3 – Rsync

Cool Cmd #3 – Rsync

man rsync

DESCRIPTION

Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behaviour and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.

As seen with dd, copying a whole drive or SD media is a handy tool for cloning one system to another drive including the MBR. If you only want the OS files, rsync can be used to recursively copy an OS file structure, between different mounted drive types, even if it still means the MBR has to be written correctly after, maybe by installing GRUB as a boot loader or using gpart or Gparted to write another correct type.

Rsync is about the best tool for doing incremental backups also, either locally or over network, and encrypted using SSH if required.

There is a great free GUI version for Windows that I use, called Syncback, by 2BrightSparks:

http://www.2brightsparks.com/

For this example, I'm going to clone the directory structure of a working Linux install I have on an external 250GB USB drive to an internal 320GB SATA drive.

A couple of things to remember when copying a working OS, like instantaneous memory processes aren't wanted and won't copy obviously, some of which in Linux are "mounted" in the /proc (process) directory, and network shares should be unmounted also else their contents will be physically copied also. It may be quicker in that case to just run rsync on a required folder by folder basis with a simpler command line, than a more complex one with multiple "--exclude /xxx" folder names.

The MOST important folder to exclude is the new system folder itself that is being copied to, else you get a feedback loop and an infinite copying until the new drive is filled, because the target folder is mounted under the main root / directory!

First, the target drive is prepared as a blank drive, by deleting and reformatting as a new Linux partition, with a swap drive of about 2.5 times memory size, say 5GB (with cfdisk and mkfs.ext4 for example), which is larger than the current OS structure to be copied. I'll assume it is known how to do this.

The basic command structure of rsync is standard, though switch placement is more flexible.

CMD – OPTIONS – SOURCE_FOLDER – DEST_FOLDER e.g.

rsync --dry-run -vr /content /backup

This would run a test copy – not actually transfer files - recursively and verbosely, of the files in /content to /backup.

I created a folder called /320GB which is the empty target SATA drive partition, and mounted the physical partition on it:

mount –t ext4 /dev/sdb /320GB

Once mounted, the folder /320GB is the target folder. I need to copy every folder and file that makes up the current working Linux OS, with the exceptions above – the target folder itself must not be recopied, and the processes folder /proc is unnecessary as it works for this current system only, and may cause rsync problems. This folder can be added empty to the mirrored directory afterwards for when the new system boots.

The actual command I'm using becomes:

rsync –dry-run –vr / /320GB/ --exclude /320GB/ --exclude /proc/

It is a good habit to always use the –dry-run switch to check the command will do what you want before you run it for real. If all looks right then run without it. Note the –exclude options can be after the destination folder, but the source must always be before the destination obviously so you get the copying in the right direction.

Once the dry run has been done and you are happy it works as you want, go for real:

rsync -vr / /320GB/ --exclude /320GB/ --exclude /proc/

var/www/www2/zones

var/www/www2/grepegsamples_files/

var/www/www2/two-interface_files/

sent 37655478872 bytes received 3163246 bytes 16193782.89 bytes/sec

total size is 37061299540 speedup is 0.98

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9]

root@DellLinux:~#

This copied about 37GB of data, and a listing of the original root, then the /320GB folder differences shows:

root@DellLinux:~# ls -l /

total 124

drwxr-xr-x 31 root root 4096 Apr 17 00:04 320GB

drwxr-xr-x 2 root root 4096 Apr 7 20:22 Acer1

drwxr-xr-x 2 root root 4096 Mar 29 02:03 bin

drwxr-xr-x 3 root root 4096 Mar 21 22:02 boot

drwxr-xr-x 14 root root 4096 Mar 22 00:19 debian

drwxr-xr-x 14 root root 3620 Apr 17 00:00 dev

drwxr-xr-x 150 root root 12288 Apr 16 23:56 etc

drwxr-xr-x 3 root root 4096 Mar 21 21:28 home

lrwxrwxrwx 1 root root 30 Mar 21 21:15 initrd.img -> /boot/initrd.img-3.2.0-4-amd64

drwxr-xr-x 17 root root 4096 Apr 8 03:34 lib

drwxr-xr-x 2 root root 4096 Mar 21 21:14 lib64

drwx------ 2 root root 16384 Mar 21 21:13 lost+found

drwxr-xr-x 3 root root 4096 Apr 15 19:02 media

drwxr-xr-x 2 root root 4096 Jan 28 10:53 mnt

drwxr-xr-x 3 root root 4096 Apr 16 18:26 opt

dr-xr-xr-x 128 root root 0 Apr 16 23:56 proc

drwxr-xr-x 2 root root 4096 Mar 31 17:03 Quadra

drwx------ 26 root root 4096 Apr 16 19:56 root

drwxr-xr-x 25 root root 1000 Apr 16 23:58 run

drwxr-xr-x 2 root root 4096 Mar 29 02:03 sbin

drwxr-xr-x 2 root root 4096 Jun 10 2012 selinux

drwxrwxrwx 9 stevee stevee 4096 Apr 12 11:07 share

drwxr-xr-x 2 root root 4096 Mar 21 21:14 srv

drwxr-xr-x 2 root root 4096 Apr 12 16:19 SSDWin7

drwxr-xr-x 2 root root 4096 Mar 24 19:02 Storebird

drwxr-xr-x 13 root root 0 Apr 16 23:56 sys

drwxrwxrwt 10 root root 4096 Apr 17 01:00 tmp

drwxr-xr-x 10 root root 4096 Mar 21 21:14 usr

drwxr-xr-x 12 root root 4096 Mar 23 00:59 var

-rw-r--r-- 1 root root 1276 Apr 13 01:19 varspace.sh

lrwxrwxrwx 1 root root 26 Mar 21 21:15 vmlinuz -> boot/vmlinuz-3.2.0-4-amd64

drwxr-xr-x 2 root root 4096 Apr 11 17:23 W7

root@DellLinux:~#
ls –l /320GB/

total 132

drwxr-xr-x 29 root root 4096 Apr 17 00:02 Acer1

drwx------ 2 root root 4096 Apr 17 00:04 bin

drwx------ 3 root root 4096 Apr 17 00:04 boot

drwx------ 14 root root 4096 Apr 17 00:04 debian

drwx------ 14 root root 4096 Apr 17 00:04 dev

drwx------ 150 root root 12288 Apr 17 00:04 etc

drwx------ 3 root root 4096 Apr 17 00:04 home

drwx------ 17 root root 4096 Apr 17 00:06 lib

drwx------ 2 root root 4096 Apr 17 00:02 lib64

drwx------ 2 root root 16384 Apr 17 00:00 lost+found

drwx------ 3 root root 4096 Apr 17 00:07 media

drwx------ 2 root root 4096 Apr 17 00:02 mnt

drwx------ 3 root root 4096 Apr 17 00:07 opt

drwx------ 2 root root 4096 Apr 17 00:02 Quadra

drwx------ 26 root root 4096 Apr 17 00:07 root

drwx------ 25 root root 4096 Apr 17 00:07 run

drwx------ 2 root root 4096 Apr 17 00:07 sbin

drwx------ 2 root root 4096 Apr 17 00:02 selinux

drwx------ 9 root root 4096 Apr 17 00:15 share

drwx------ 2 root root 4096 Apr 17 00:02 srv

drwx------ 2 root root 4096 Apr 17 00:02 SSDWin7

drwx------ 2 root root 4096 Apr 17 00:04 Storebird

drwx------ 13 root root 4096 Apr 17 00:26 sys

drwx------ 10 root root 4096 Apr 17 00:26 tmp

drwx------ 10 root root 4096 Apr 17 00:26 usr

drwx------ 12 root root 4096 Apr 17 00:40 var

-rw-r--r-- 1 root root 1276 Apr 17 00:04 varspace.sh

drwx------ 2 root root 4096 Apr 17 00:02 W7

Recreating /proc:

mkdir /320GB/proc

and giving it the same permissions:

chmod ugo+rx,ugo-w /320GB/proc/

root@DellLinux:~# ls -al /320GB/proc

dr-xr-xr-x 2 root root 4096 Apr 17 01:07 .

drwxr-xr-x 32 root root 4096 Apr 17 01:07 ..

Ownership is root anyway.

Just to check recursive copy took place:

root@DellLinux:~# ls /320GB/lib

crda libip6tc.so.0.1.0 lsb

discover libipq.so.0.0.0 modprobe.d

firmware libiptc.so.0.0.0 modules

hdparm libnss_mdns4_minimal.so.2 oss-compat

init libnss_mdns4.so.2 security

klibc-2xtYrByCrj5OEwaInv4tMSjej98.so libnss_mdns6_minimal.so.2 systemd

libaudit.so.0.0.0 libnss_mdns6.so.2 terminfo

libauparse.so.0.0.0 libnss_mdns_minimal.so.2 udev

libcryptsetup.so.4.2.0 libnss_mdns.so.2 x86_64-linux-gnu

libip4tc.so.0.1.0 libxtables.so.7.0.0 xtables

That seems ok, so now you have the major task of researching and installing a suitable boot loader like grub2 on the MBR so that the new drive will boot...

Needless to say, I won't be doing a Post on GRUB2 under a ~Cool Command~ heading – because it never was cool at all – just a complete overly complex headache in my view - particularly for such a useful backup scenario as this...Good Luck with that!

Good How To's seem hard to find also...

Comments are closed.

Post Navigation