Find and delete partially duplicate lines

Question

https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html
https://www.domain2.com/dl/G5SPNDOF/JHCGTS/AAA-1080p.mp4.html
https://www.domain.com/files/ZQWL80BG/AAA-1080p.mp4.html
https://www.domain.com/files/SVSRS0AD/BBB-1080p.mp4.html
https://www.domain.com/files/UCIONEMA/BBB-1080p.mp4.html

Given a file with above lines, how do I delete the ones that have duplicate files, but links overall are different, to end up with:

https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html
https://www.domain.com/files/SVSRS0AD/BBB-1080p.mp4.html

Answer 1

Using awk :

awk -F'/' '!seen[$NF]++' file

You can also do it using sort. As your file contain a number of fields that change you need to reverse it first :

rev file | sort -u -t/ -k1,1 | rev

Answer 2

Using any sed, you'd choose the lazy "throw all known file names on the hold space" approach:

sed 'G;\_/\(.*\)\n.*\1_d;P;s_.*/__;h;d' file

• G appends the hold space (with all known file names) to the pattern space
• The pattern /\(.*\)\n.*\1 fits a filename between / and the newline, which gets repeated (\1) in the collected file names; in this case, delete it, no output, no changes to the hold space
• At this point, we know we found a new word, so Print the line without the appended hold space
• s_.*/__ removes the path, so the new path list can be copied to the hold space. d suppresses further output

Answer 3

If your list is huge then to avoid storing all of the file names in memory in awk:

$ sort file | awk -F'/' '$NF != prev{print} {prev=$NF}'
https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html
https://www.domain.com/files/SVSRS0AD/BBB-1080p.mp4.html
https://www.domain.com/files/ZQWL80BG/AAA-1080p.mp4.html

Otherwise see @MariusCouet's awk answer.

Answer 4

Using Raku (formerly known as Perl_6)

~$ raku -e '.put for lines.unique(as => *.IO.basename);'  file

Sample Input:

https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html
https://www.domain2.com/dl/G5SPNDOF/JHCGTS/AAA-1080p.mp4.html
https://www.domain.com/files/ZQWL80BG/AAA-1080p.mp4.html
https://www.domain.com/files/SVSRS0AD/BBB-1080p.mp4.html
https://www.domain.com/files/UCIONEMA/BBB-1080p.mp4.html

Sample Output:

https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html
https://www.domain.com/files/SVSRS0AD/BBB-1080p.mp4.html

Raku understands directory conventions, so you can use .IO to convert your string to an IO object, and get the filename with basename. Raku's unique function is clever enought to take an as parameter, so the full file path is returned.

https://docs.raku.org/routine/unique
https://docs.raku.org/routine/IO
https://docs.raku.org/routine/basename
https://raku.org

Answer 5

Using GNU datamash to prefix the original data with the basename (filename at the end) from each URL, and then to group the data by the added column and extract the first URL from each group. The added field is then removed with cut at the end of the pipeline. This is reminiscent of how one might do a Schwartzian transform.

$ datamash basename 1 cut 1 <file | datamash groupby 1 first 2 | cut -f 2
https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html
https://www.domain.com/files/SVSRS0AD/BBB-1080p.mp4.html

You can instead get the last URL from each group:

$ datamash basename 1 cut 1 <file | datamash -s groupby 1 last 2 | cut -f 2
https://www.domain.com/files/ZQWL80BG/AAA-1080p.mp4.html
https://www.domain.com/files/UCIONEMA/BBB-1080p.mp4.html

... or a random one:

$ datamash basename 1 cut 1 <file | datamash -s groupby 1 rand 2 | cut -f 2
https://www.domain2.com/dl/G5SPNDOF/JHCGTS/AAA-1080p.mp4.html
https://www.domain.com/files/SVSRS0AD/BBB-1080p.mp4.html
$ datamash basename 1 cut 1 <file | datamash -s groupby 1 rand 2 | cut -f 2
https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html
https://www.domain.com/files/SVSRS0AD/BBB-1080p.mp4.html
$ datamash basename 1 cut 1 <file | datamash -s groupby 1 rand 2 | cut -f 2
https://www.domain.com/files/ZQWL80BG/AAA-1080p.mp4.html
https://www.domain.com/files/UCIONEMA/BBB-1080p.mp4.html