Friday, April 2, 2010

Using tar with ssh effectively

We know that tar could be used to take backup. We may like to transfer backup to some remote box. Also some time,it is needed to transfer file with directory structure in the remote box.
Lets say we have to transfer many files in the remote box while keeping file permissions and directory structure intact.
We may choose to tar the source directory ,compress it and scp to destination box. We will need to create a v big temp file only to be transferred to the destination box and and removed from source box. Once the file is scp'd to destination box, we will extract the tar'd file to create files in the same directory structure.

Steps :-

(In Src box)

1) Tar the directory in the source file ( tar -cvzf tarfilename.tar.gz /path/of/dir )
2) transfer file to destination box (scp tarfilename.tar.gz user@remote-host)
3) Delete the file ( rm tarfilename.tar.gz)

(In Dest Box)

4) Untar the file (tar -xvzf tarfilename.tar.gz)
5) Remote the file (rm tarfilename.tar.gz)

We can accomplish all these steps in a single command.

Tar has the great ability to send data to stdout/stdin using the "-" (dash) as a filename in the command line. So using that we can string together pipes to send the data to a remote server

tar -cvzf - /dir/to/be/transferred/ | ssh remoteuser@remotehost 'tar -C /path/of/dir -xvzf -'

IN NUTSHELL, WE HAVE REPLACED tarfilename.tar.gz WITH -(DASH)

What this does is pretty simple: it creates a compressed tar file of the /dir/to/be/transferred/ and sends it to stdout (-). We catch stdout with the pipe (|) and then call ssh to connect to a remote server where we execute the remote command tar. Using the remote tar we run a decompress extraction of stdin (-). tar -c changes directory

Great way to transfer a bunch of files securely, as well maintain ownership and permissions.
Its the example of PUSH method.
Following example will show the reverse method.

Lets say we have a backup file in remote host and we want to restore it in local box.
(PULL Method)

ssh user@remote-host "cat /path/to/backup/backupfile.tar.bz2" |tar jpxf -

In this command, we are cat'ing zipped file and output is being redirected to local tar using pipe. So local tar command will replace -(dash) with zipped file and will untar it.

Many other scenarios could be found in http://www.lamolabs.org/blog/1766/pushing-pulling-files-around-using-tar-ssh-scp-rsync/