1

I'm looking for a way to find non-repeated elements in an array in bash.

Simple example:

joined_arrays=(CVE-2015-4840 CVE-2015-4840 CVE-2015-4860 CVE-2015-4860 CVE-2016-3598)
<magic>
non_repeated=(CVE-2016-3598)

To give context, the goal here is to end up with an array of all package update CVEs that aren't generally available via 'yum update' on a host due to being excluded. The way I came up with doing such a thing is to populate 3 preliminary arrays:

  • available_updates=() #just what 'yum update' would provide
  • all_updates=() #including excluded ones
  • joined_updates=() # contents of both prior arrays Then apply logic to joined_updates=() that would return only elements that are included exactly once. Any element with two occurrences is one that can be updated normally and doesn't need to end up in the 'excluded_updates=()' array.

Hopefully this makes sense. As I was typing it out I'm wondering if it might be simpler to just remove all elements found in available_updates=() from all_updates=(), leaving the remaining ones as the excluded updates.

Thanks!

Improve this question
2
  • If your goal is to be able to do set differences, intersections, etc., you might consider using the comm utility. See BashFAQ #36. Commented Aug 31, 2016 at 15:12
  • @CharlesDuffy : In fact i didn't see your answer. Hmmm. but you're right, the question is something else :) Commented Aug 31, 2016 at 15:15

3 Answers 3

Reset to default
2

One pure-bash approach is to store a counter in an associative array, and then look for items where the counter is exactly one:

declare -A seen=( )                   # create an associative array (requires bash 4)
for item in "${joined_arrays[@]}"; do # iterate over original items
  (( seen[$item] += 1 ))              # increment value associated with item
done

declare -a non_repeated=( )
for item in "${!seen[@]}"; do         # iterate over keys
  if (( ${seen[$item]} == 1 )); then  # if counter for that key is 1...
    non_repeated+=( "$item" )         # ...add that item to the output array.
done

declare -p non_repeated               # print result

Another, terser (but buggier -- doesn't work with values containing newline literals) approach is to take advantage of standard text manipulation tools:

non_repeated=( )        # setup

# use uniq -c to count; filter for results with a count of 1
while read -r count value; do
  (( count == 1 )) && non_repeated+=( "$value" )
done < <(printf '%s\n' "${joined_arrays[@]}" | sort | uniq -c)

declare -p non_repeated # print result

...or, even terser (and buggier, requiring that the array value split into exactly one field in awk):

readarray -t non_repeated \
  < <(printf '%s\n' "${joined_arrays[@]}" | sort | uniq -c | awk '$1 == 1 { print $2; }'

To crib an answer I really should have come up myself from @Aaron (who deserves an upvote from anyone using this; do note that it retains the doesn't-work-with-values-with-newlines bug), one can also use uniq -u:

readarray -t non_repeated < <(printf '%s\n' "${joined_arrays[@]}" | sort | uniq -u)
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

I would rely on uniq.

It's -u option is made for this exact case, outputting only the uniques occurrences. It relies on the input to be a sorted linefeed-separated list of tokens, hence the need for IFS and sort :

$ my_test_array=( 1 2 3 2 1 0 )
$ printf '%s\n' "${my_test_array[@]}" | sort | uniq -u
0
3
Improve this answer

3 Comments

facepalm. I'm really quite embarrassed that I missed that possibility. (But why deal with IFS rather than printf '%s\n'? That way you work even on shells with non-bash implementations of echo, which have a rather huge amount of leeway in their behavior -- POSIX doesn't define echo's behavior at all if there's a backslash anywhere in its arguments, for instance... unless it's one with XSI extensions, in which case backslash-escape sequences must be interpolated).
Why deal with IFS rather than printf? Because I'm a noob with arrays and took the first thing that worked :) I've updated my answer to use printf
Please forgive the small updates here -- "${foo[@]}" avoids bugs (if, for instance, any of your array entries contain characters in IFS).
0

Here is a single awk based solution that doesn't require sort:

arr=( 1 2 3 2 1 0 )

printf '%s\n' "${arr[@]}" | 
awk '{++fq[$0]} END{for(i in fq) if (fq[i]==1) print i}'

0
3
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.