remove duplicate files

Question

On my Mac OS X 10.6.7, iTunes duplicated every single file in my music library. Now I have 3,840 files instead of 1,920. The problem is that iTunes didn't simply duplicate the whole folder, but each file in each folder, adding a 1 to the duplicate. It looks like this:

├── album1
│   ├── track1.mp3
│   ├── track1 1.mp3
│   ├── track2.mp3
│   └── track2 1.mp3
└── album2
    ├── track1.m4a
    └── track1 1.m4a

Now rather than clicking on 1,920 duplicates, I want to enter some smart line in the Terminal to get rid of the extra files:

find path/to/music/ -name "* 1.*" -delete

Though this will work in most cases, it will mess up things in the following situation:

└── album2
    ├── track 1.mp3
    └── track 1 1.mp3

Maybe I should exclude "* 1 1.*" and later on rename those files removing the extra 1?

How do I do this?

Is there a simple solution to this?

Answer 1

I'd also add that to successfully find (and remove) exact file duplicates, you have to compare files by hash:

#!/bin/sh -eu
find "${1:-.}" -type f ! -empty -print0 | xargs -0 md5 -r | \
    awk '$1 in a{sub("^.{33}","");printf "%s\0",$0}a[$1]+=1{}' | \
    xargs -0 rm -v --

Save this as deldupes.sh and run it giving dirname as 1st parameter (otherwise $PWD would be used).

Tested on OS X, works for long whitespaced filenames.

Answer 2

Something like this, haven't tested; it's in bash so you may have to convert some syntax:

IFS=$'\n'              # so that only newlines separate words, not spaces
set -f                 # disable globbing
FILES=$(find path/to/music/ -name "* 1.*")

for FILE in ${FILES}; do
    if [[ -f "${FILE% 1.*}" ]] ; do
        echo "Matched ${FILE}."
        # rm "${FILE}" # Uncomment me once you have confirmed it would do what you intend.
    fi
done

${FILE% 1.*} strips the last match of the 1.* syntax from the end, [[ -f ... ]] checks whether that file would exist; therefore, it would remove the files for which the file without the syntax at the end would exist. Please test before uncommenting rm, to be sure it is correct.