get unique without sorting in jq

Question

Array => ["cross", "base", "cross", "dunk"]

I want to delete all repeated values without sorting.

Command: jq '. | unique' file.json

Output: ["base", "cross", "dunk"]

Expected Output: ["cross", "base", "dunk"]

Answer 1

map({key:.,value:1})|from_entries|keys_unsorted

as an alternative to unique seems to work.

map turns the array into:

[{"key":"cross","value":1},{"key":"base","value":1},{"key":"cross","value":1},{"key":"dunk","value":1}]

from_entries turns that into:

{"cross":1,"base":1,"dunk":1}

Removing the duplicates at that point since two elements of an object can't have the same key.

And keys_unsorted returns the keys of that object, in the original order.

A shorter/simpler alternative working on the same principle:

map({(.):1})|add|keys_unsorted

---

Or you can use a more generic programming language such as perl:

perl -l -0777 -MJSON::PP -MList::Util=uniq -pe '
  $_ = encode_json[uniq @{decode_json $_}]' array.json

-p is the sed mode, pull each record of array.json into $_ for the eexpression to work on.

-0777 sets the record separator to something impossible, meaning there's only one record being the full file (see also -gobble in newer versions of perl).

decode_json $_ decodes that record and returns an array reference, @{...} makes that a list passed to uniq, [...] makes the result another array reference passed to encode_json, the resulting $_ is printed (followed by a newline with -l).

There are a number of JSON plugins for perl. JSON::PP (PP for pure perl) is part of the core of perl, so should always be available. JSON::XS would be more efficient and also comes with a json_xs which is handy to use in shell scripts:

json_xs -e 'use List::Util "uniq"; $_ = [uniq @$_]' < array.json

If you're already familiar with perl, that means you don't need to learn a new language like that of jqs and can reuse what you know to process other formats like XML, YAML, etc.

Answer 2

Using reduce() to build the resulting array of unique elements from the input:

jq 'reduce .[] as $a ([]; if IN(.[]; $a) then . else . += [$a] end)' file.json

This essentially iterates over the array in file.json, adding each element to a new array unless it's already present in that array.

Testing:

$ jq . file.json
[
  "cross",
  "base",
  "cross",
  "dunk"
]

$ jq 'reduce .[] as $a ([]; if IN(.[]; $a) then . else . += [$a] end)' file.json
[
  "cross",
  "base",
  "dunk"
]