4

I always have been using sort -u to get rid of duplicates until now.
But I am having a real doubt about a list generated by a software tool.
The question is: is the output of sort -u |wc the same as uniq -u |wc?

Because they don't yield the same results. The manual for uniq specifies:

-u, --unique only print unique lines

My output consists of 1110 words for which sort -u keeps 1020 lines and uniq -u 1110 lines, the correct amount. The issue is that I cannot visually spot any duplicates on the list which is generated by using > at the end of the command line, and that there IS an issue with the total cracked passwords (in the context of customizing john the ripper).

Improve this question
1

1 Answer 1

Reset to default
7

No, they're not the same. For one, sort would sort the list first; and second, uniq -u prints only those lines that are "unique" in each given run, the ones that don't have an identical input line before or after them.

$ printf "%s\n"  3 3 2 1 2 | sort -u
1
2
3
$ printf "%s\n"  3 3 2 1 2 | uniq -u
2
1
2

See also:

Improve this answer
3
  • Thanks for the links, I figured out the difference. It's much easier to look at an output passed through sort |uniq, than raw one for what I'm doing. Commented May 30, 2022 at 19:28
  • @roaima Are they? Probably. But as soon as -u was added to sort and there was no longer an imperative to pipe to uniq, divergence in output became possible. It made things faster, some may say cleaner, but it's harder to guarantee the consistency, much less prove it. But, -u has to be in sort when using keys (-k), since there's no clean way to tell uniq what they keys were. Commented May 30, 2022 at 21:41
  • @BlairHoughton I've withdrawn my comment after re-reading unix.stackexchange.com/a/76095/100397 Commented May 31, 2022 at 6:40

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.