how to synchronize two directory?

Question

I want to using crontab to synchronize two directory between my linux partion and windows partion like this:

24 9 * * * cp -r /home/fan/Data /media/T/Data

But it would create a directory named Data in the origin Data directory, instead of copy the missing file from the source directory. I can't find a proper option at the cp manual to perfectly solve this. How can i copy the missing file(they do exist at the destination directory) from the src to dir.

And by the way, seems it need the T disk have been mounted to run the copy command, how to automatically mount the disk when i need to run the command(the mount command should run as root).

And how can i get the error message if the command have an exception?

Answer 1

The go to tool for syncing data is rsync. You can sync either at the directory level or just the contents of a directory like so:

Examples

directory sync

24 9 * * * rsync -a /home/fan/Data /media/T/

contents sync

24 9 * * * rsync -a /home/fan/Data/ /media/T/Data/

The first example will sync the directory Data from /home/fan to the directory /media/T. The second example will sync the contents of Data to the directory /media/T/Data.

As an additional tip I'll sync to target directories using this notation:

24 9 * * * rsync -a /home/fan/Data/ /media/T/Data/.

The /. I find more obvious when reading through my rsync commands.

Behavior Examples

Using rsync is probably one of the most confused issues I deal with when training new users.

Sample data:

$ tree home/fan/Data/
home/fan/Data/
|-- file1
|-- file2
|-- file3
|-- file4
`-- file5

1. syncing contents of dir Data

$ rsync -a home/fan/Data/ media/T/

Notice we put the contents of Data into T.

$ tree media/T/
media/T/
|-- file1
|-- file2
|-- file3
|-- file4
`-- file5

0 directories, 5 files

2. syncing the directory

$ rsync -a home/fan/Data media/T/

Notice we put the directory Data into T.

$ tree media/T/
media/T/
`-- Data
    |-- file1
    |-- file2
    |-- file3
    |-- file4
    `-- file5

1 directory, 5 files

Answer 2

Using rsync:

24 9 * * * rsync -a /home/fan/Data/ /media/T/Data

Add the extra / at the end of the name of the source directory will tell rsync to copy the contents rather than the full directory. rsync will also avoid copying files that have the same timestamps/size (ie ones that don't need to be copied in 99.99% of cases).

Also consider using the --inplace option for rsync to speed things up if you are syncing files on the same system. From the man page:

--inplace
    This option changes how rsync transfers a file when its data needs to be
    updated: instead of the default method of creating a new copy of the file and
    moving it into place when it is complete, rsync instead writes the updated
    data directly to the destination file.

There are pros and cons to this option though, probably the biggest con is that if the sync is interrupted, the target file will be left in an inconsistent state which can only be fixed by starting over again.

Answer 3

To address the error-message portion of the question, you might choose to run a script from cron instead of the system command.

24 9 * * * /usr/local/sbin/sync_data.sh

Create the file as /usr/local/sbin/sync_data.sh, giving root ownership and execute permission: chown root:root /usr/local/sbin/sync_data.sh && chmod 0700 /usr/local/sbin/sync_data.sh. The contents of the script are below.

#!/usr/bin/env bash
if [[ $EUID -ne 0 ]]; then
    echo "This script must be run as root." 1>&2
    exit 1
fi

# [ ] vs [[ ]] : http://mywiki.wooledge.org/BashFAQ/031

dir_src="/home/fan/Data/"
dir_dst="/media/T/"

# A Boolean to know if running interactively or not
b_interactive=1
if [[ -v PS1 ]] ; then # The "-v PS1" method is for BASH 4.2+
    $b_interactive=0   # Could also use [[ -z "$PS1" ]]
fi

# Send messages to console or syslog.
function notify() {
    message="$1"
    if [[ $b_interactive -eq 1 ]] ; then
        echo "$message"
    else
        # eval combines args into a single string for execution...
        eval $cmd_logger -p err "$0: $message"
    fi
}

# If the mount point if not currently mounted...
if [[ "$(grep $dir_dst /proc/mounts)" = "" ]] ; then
    # Try to mount the directory.
    mount $dir_dst

    # Send a message to console or syslog.
    if [[ $? -ne 0 ]] ; then
        notify("$0 failed to mount $dir_dst")
        exit 1;
    fi
fi

# Create a backup directory if it does not exist
if [[ -d $dir_dst ]] ; then
    mkdir -p $dir_dst 2>/dev/null
fi

# A one-way sync to dir_dst, deleting files that no longer exist in DIR_SRC...
rsync -a --delete $dir_src $dir_dst &> /dev/null

# Check the return status of last command...
if [ $? -eq 0 ]; then
    notify("$0: the rsync process succeeded.");
else
    notify("$0: the rsync process failed.");
fi

unset dir_src
unset dir_dst
unset b_interactive

Answer 4

mkdir /office
mount /dev/xvdf /office
mkdir /home
mount /dev/xvdf1 /home

rsync -aHAXxSP /office /home