I have a ksh script that returns a long list of values, newline separated, and I want to see only the unique/distinct values. It is possible to do this?
For example, say my output is file suffixes in a directory:
tar gz java gz java tar class class
I want to see a list like:
tar gz java class
You might want to look at the uniq and sort applications.
./yourscript.ksh | sort | uniq
(FYI, yes, the sort is necessary in this command line, uniq only strips duplicate lines that are immediately after each other)
EDIT:
Contrary to what has been posted by Aaron Digulla in relation to uniq's commandline options:
Given the following input:
class jar jar jar bin bin java
uniq will output all lines exactly once:
class jar bin java
uniq -d will output all lines that appear more than once, and it will print them once:
jar bin
uniq -u will output all lines that appear exactly once, and it will print them once:
class java
sort features a -u version for giving the unique values too.
uniq seams to process only adjacent lines (at least by default) meaning one may sort input before feeding uniq.
sort | uniq was 95 seconds - sort -u was 77 - awk '!a[$0]++' from @ajak6 was 9 seconds. So awk wins but also the hardest to remember../script.sh | sort -u
This is the same as monoxide's answer, but a bit more concise.
... | sort | uniq because it is performed in one shot
sort -u doesn't need to update the sorted list each time it finds an already encountered earlier value. while the sort | has to sort all items before it passes it to uniq
sort -u could be faster because any optimal comparison sort algorithm has O(n*log(n)) complexity, but it is possible to find all unique values with O(n) complexity using Hash Set data structure. Nonetheless, both sort -u and sort | uniq have almost the same performance and they both are slow. I have conducted some tests on my system, more info at gist.github.com/sda97ghb/690c227eb9a6b7fb9047913bfe0e431d
./script.sh | sort | uniq -u didn't output anything. Maybe because the output was too large? Although it wasn't so big, the output had 50_000 lines, with just 4 distinct values.With AWK you can do:
./yourscript.ksh | awk '!a[$0]++'
I find it faster than sort and uniq
With zsh you can do this:
% cat infile
tar
more than one word
gz
java
gz
java
tar
class
class
zsh-5.0.0[t]% print -l "${(fu)$(<infile)}"
tar
more than one word
gz
java
class
Or you can use AWK:
% awk '!_[$0]++' infile
tar
more than one word
gz
java
class
awk solution (see stackoverflow.com/a/21200722/45375 for an explanation) will work with large files as long as the number of unique lines is small enough (as unique lines are kept in memory). The zsh solution reads the entire file into memory first, which may not be an option with large files. Also, as written, only lines with no embedded spaces are handled correctly; to fix this, use IFS=$'\n' read -d '' -r -A u <file; print -l ${(u)u} instead.
(IFS=$'\n' u=($(<infile)); print -l "${(u)u[@]}")[@] suffix to reference all elements of an array - seems that - at least as of version 5 - it works without it; or did you just add it for clarity?print -l "${(fu)$(<infile)}"awk sample output, too.Pipe them through sort and uniq. This removes all duplicates.
uniq -d gives only the duplicates, uniq -u gives only the unique ones (strips duplicates).
uniq -u is NOT the default behaviour (see the edit in my answer for details)For larger data sets where sorting may not be desirable, you can also use the following perl script:
./yourscript.ksh | perl -ne 'if (!defined $x{$_}) { print $_; $x{$_} = 1; }'
This basically just remembers every line output so that it doesn't output it again.
It has the advantage over the "sort | uniq" solution in that there's no sorting required up front.
Unique, as requested, (but not sorted);
uses fewer system resources for less than ~70 elements (as tested with time);
written to take input from stdin,
(or modify and include in another script):
(Bash)
bag2set () {
# Reduce a_bag to a_set.
local -i i j n=${#a_bag[@]}
for ((i=0; i < n; i++)); do
if [[ -n ${a_bag[i]} ]]; then
a_set[i]=${a_bag[i]}
a_bag[i]=$'\0'
for ((j=i+1; j < n; j++)); do
[[ ${a_set[i]} == ${a_bag[j]} ]] && a_bag[j]=$'\0'
done
fi
done
}
declare -a a_bag=() a_set=()
stdin="$(</dev/stdin)"
declare -i i=0
for e in $stdin; do
a_bag[i]=$e
i=$i+1
done
bag2set
echo "${a_set[@]}"
I get a better tips to get non-duplicate entries in a file
awk '$0 != x ":FOO" && NR>1 {print x} {x=$0} END {print}' file_name | uniq -f1 -u
