0

I've got 2 very old Backups of a friends computer. They were simply copied into a folder each on an external Harddrive. Both are about 300GB in Size and the contents are very much alike but not identical and the folder-structure is different. I want to free that space and make one single Backup of those two. I think about 90% of the files are douplicates, but i dont want to miss the files that are not.

So what I need is a program that compares the files in two directories with all their subdirectories but ignoring these subdirectories. All files within Folder A should be compared with All Files in Folder B. All exact douplicates in Folder B should be marked/moved(/deleted). I will handle the remains in Folder B manually.

I've tried meld, I've tried Gnome-Commander (I'm using Xubuntu with XFCE)

I would enjoy a gui-solution but I should be able to handle terminal and scripts too. I thought it may be possible to build a file-list for both sides and pipe these to some diff-program, but how to do it exactly is out of my capabilities.

Well, looking forward to your answers, turtle purple

Improve this question
5
  • What do you mean by ignore (it looks like a contradiction to me)? Commented Jan 26, 2020 at 14:20
  • 2
    What about two identical file but different names? What about identical files in different directories? Commented Jan 26, 2020 at 14:21
  • How about using the configurable diff algorihtm (via diffopts=) in star? It works nicely together with the builtin find feature from libfind. Commented Jan 26, 2020 at 14:34
  • Look at finddup.sourceforge.net Commented Jan 26, 2020 at 16:03
  • Ignoring the Subdirectories is a Trade-Off: I would like to preserve them, but whats actually important is the comparison of all acutal data (files). As the directory structure is different (the backups are from different years) it is not suitable to compare directories. Commented Apr 10, 2020 at 11:57

2 Answers 2

Reset to default
1

If the aim is to preserve file content (avoid losing data), I would concentrate on file equality, not the naming of directories of files.

Start with running this on each of the top-level folders, and save the output (it will run for a while!).

find FolderA -type f -print0 | xargs -0 cksum > FoldA.cksum
find FolderB -type f -print0 | xargs -0 cksum > FoldB.cksum

Sort the two outputs together, which brings any identical file contents together. Then start writing awk to group identical content based on the first two fields (checksum and size).

(a) Any one-line group is a unique file to be kept.
(b) Any larger group is a list of identical files. May as well keep the top one, and write the other names to a list for deletion (these may be duplicates between A and B, or within A, or within B, or both.

All duplicate files will now only be in your FolderA name, as will be about half of the unique files. What do you do with files where the selected copy is from FolderB (assuming you need to merge the remnants)?

If their pathname (from after FolderB down to the lowest directory) exists in FolderA, that's probably where you need to mv them to, via another output list.

If their pathname at FolderA does not exist, you would be guessing where they really belong. You could make the appropriate directory (with all its parents) and risk mislaying it, or eyeball it to see if it corresponds to anywhere else.

Both those last steps need an extra check: the possibility that you have two (or more) non-identical files with the same name. In that case, you need to choose some resolution (like always keep the later version), or extend the filename to make it unique, or examine each case individually.

My approach would be to work this incrementally: deal with the exact duplicate files first (90% in your estimation), then evaluate the discrepancies for any pattern you can use to reconcile the remainder.

Improve this answer
6
  • This is much slower than using star -c -diff -v -C dir1 . dir2 with s suitable diffopts= setup andprobably using the right -find .... arg list for star. Commented Jan 26, 2020 at 16:02
  • @schily. I can see you named as the author and contact for star and ustar, although the only man pages I can find say "For now (late 2002), ...". SourceForge has V6.1, but it is not in my Mint/Ubuntu repository. I appreciate cksum is going to be darn slow. But comparing by pathname, where the OP already knows the directory structures are different, cuts no ice. And missing out some of the levels of directory makes for ambiguous name matches. Cksum is going to identify where those levels are significant -- there should be classes of pairs that highlight them. Commented Jan 26, 2020 at 17:10
  • Recent is star version 1.6.1 and it should be easy to find tat the most recent source is in schilytools. If you install a OSS friendly distro like Suse, you get the most recent version as binary package. Commented Jan 26, 2020 at 17:34
  • BTW: here is the recent man page: schilytools.sourceforge.net/man/man1/star.1.html and in case it is not known, star is older than gtar and more actively maintained. I guess you checked man page collections from Linux. Commented Jan 27, 2020 at 8:56
  • This Solution is pretty much what I hoped for. Unfortunatley it's currently of reach for me to do AWK-scripting. I do know how to use a commandline, run scripts and set some parameters and AWK looks manageable but I can't dig into it right now, I would need an example. If you feel like doing one I'd be glad. Otherwise I will probably find GUI-Tools that do the job of sorting the outputs. So Thanks for the Quick answer, hope I'll find time to try it soon. Commented Apr 10, 2020 at 12:11
0

Do something simple as copy one directory onto another and ignore/replace identical (by path) files. If this will produce some duplicates (different path, same file name) you can try to find them later.

Want "gui" solution? Use mc. When asked what do do with duplicates you can choose "update" or "if size differs".

Or use archiver like zip/7z/rar/tar pack dir1 into an archive and update with dir2. This will avoid duplicates and you'll get one, updated backup.

PS. recursively ignoring sub-directories is contradiction. Can you explain that as it's more than unclear.

Improve this answer
1
  • I think it parses as "Compare large directories recursively, ignoring sub-directories". This is, compare pathnames only as: FolderX/.../basename.ext. Of course, chances are that will generate apparent duplicates between both archives, and also in each archive separately, which then also needs resolving. Commented Jan 26, 2020 at 17:17

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.