1

I'd like to copy from /A to /C only paths missing on /B and /C. (Please assume those are paths, and can represent also remote locations e.g. mounted via sshfs)

I wonder... Is there more concise way then writing a loop, iterating over filesystem and making check, file by file ?

Example contents of tree directories /A, /B, /C :

/A:

/A/1abc/qwe
/A/2abc/asd
/A/3abc/zxc
/A/4abc/rty

/B:

/A/2abc/asd
/A/3abc/zxc

/C:

/C/1abc/qwe

expected: to copy from /A to /C only path 4abc/rty:

/A/4abc/rty -> /C/4abc/rty

To illustrate practical examples of /A, /B, /C, leading to such scenario:

  • you make backup, copy from some /source (/A) to some /destination (/B) and some paths failed to be copied. So you want to have copy anyway of missing ones. So you find /another_destination (/C) that can accept them, and you want to copy only missing ones. Here are example limitations why copy from /A to /B could fail: disc got full; limitation of filesystem of /B (like filename lenght), while not present on /C filesystem; etc.
Improve this question
4
  • A,B & C are on same machine or are they remotely resided ? Commented Jun 20, 2016 at 10:35
  • Please assume that all are mounted, but might be over sshfs (so remotely over not fastest connection). Commented Jun 20, 2016 at 11:04
  • your example is bit confusing, regardless of your title. If you can edit more it would be nice for me or for others too. Commented Jun 20, 2016 at 11:26
  • Sure! Let me rewrite a bit once I am back at desktop :). Commented Jun 20, 2016 at 11:29

1 Answer 1

Reset to default
1

One solution is to make a union mount of B and C with C as the write branch. Then the problem reduces to copying from A to B+C only files missing on B+C, which can be done with

rsync -a --ignore-existing /A /B+C

For the union mount, one possibility is unionfs-fuse.

unionfs-fuse -o cow /C=RW:/B=RO /B+C

Or unionfs:

mount -t unionfs -o dirs=/C=rw:/B=ro unionfs /B+C
Improve this answer
5
  • By curiosity : why unionfs-fuse instead of in-krenel unionfs ? I guess, it was just hany to write for you but both would do the job - or are there some edge cases where uninfs-fuse would be better for this kind of application ? Commented Jun 21, 2016 at 12:49
  • 1
    @GrzegorzWierzowiecki It has just handy for me to write, most union mount systems can handle this. For a discussion of union mount systems, see this thread — when somebody answers! Commented Jun 21, 2016 at 12:56
  • Too many open files (24) by rsync. I wonder if setting higher like -o max_files=32000 option for unionfs-fuse is good strategy? Commented Jun 26, 2016 at 21:31
  • No, it doesn't help. Still I get rsync: chown "/.../xyz" failed: Too many open files (24) at some point and rsync does not rsync everything, and unionfs-fuse directory is not possible to ls anymore. Maybe I should try with unionfs or AuFS instead. I couldn't find rsync flag that could help for that. Commented Jun 27, 2016 at 14:05
  • @GrzegorzWierzowiecki Oh, I'd never encountered that problem with unionfs-fuse. Presuming that unionfs-fuse is the culprit (the open file limit is per-user), another union mount implementation might help, and an in-kernel one (overlayfs, aufs) would not run into this problem. Commented Jun 27, 2016 at 18:39

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.