RSYNC to copy new files in different directory

Question

In the remote directory, I have the below files and need to copy only the current day's files + previous run failures to my local directory.

Example: in Remote:

aa.txt
aa-18-04-2017.txt
aa-17-04-2017.txt

In local, I have to rsync the file aa.txt to my new directory.

If the previous run failed, I need to rsync two files (aa.txt and aa-18-04-2017.txt).

Can someone help me here to resolve this?

Currently, RSYNC is copying all the files to our local directory and this causing storage issue on daily basis.

Answer 1

rsync has a very comprehensive manual page, there's even some examples which indicate how to do this. Here's my take along the lines of what you want.

rsync -auv remotemachine:/path/to/srcdir/ /path/to/dest_dir

I have made some assumptions, update your question with additional information if your setup differs and you need more help.

Assumptions:

remotemachine has rsync server set up. (ssh access is possible but then the syntax is slightly different)

-a archive mode sets several options automatically, but it is usually what you want for backups

-u update mode, only copy files that don't exist or size or last modification time differ (source files are newer than what you have already)

srcdir/ The trailing slash means copy files within src_dir to dest_dir to avoid an extra level in destination.

-v verbose mode so you can see what is happening.

To Test this, use the -n option in addition, this skips the copy but shows you what it would do.

But I do recommend, reading the manual page man rsync and testing.

[edit] After @Dev 's 1st comment below.

If you have an intermediate step, ie dest_dir is really a temporary local storage location, used for offline archiving. Then, you could possibly build a list of files that have been archived successfully, and use the --exclude-from-file=/some/where/exclude_list.txt option.