Skip to main content
added 506 characters in body
Source Link
Stéphane Chazelas
  • 584.5k
  • 96
  • 1.1k
  • 1.7k
map({key:.,value:1})|from_entries|keys_unsorted

as an alternative to unique seems to work.

map turns the array into:

[{"key":"cross","value":1},{"key":"base","value":1},{"key":"cross","value":1},{"key":"dunk","value":1}]

from_entries turns that into:

{"cross":1,"base":1,"dunk":1}

Removing the duplicates at that point since two elements of an object can't have the same key.

And keys_unsorted returns the keys of that object, in the original order.

A shorter/simpler alternative working on the same principle:

map({(.):1})|add|keys_unsorted

Or you can use a more generic programming language such as perl:

perl -l -0777 -MJSON::PP -MList::Util=uniq -pe '
  $_ = encode_json[uniq @{decode_json $_}]' array.json

-p is the sed mode, pull each record of array.json into $_ for the eexpression to work on.

-0777 sets the record separator to something impossible, meaning there's only one record being the full file (see also -gobble in newer versions of perl).

decode_json $_ decodes that record and returns an array reference, @{...} makes that a list passed to uniq, [...] makes the result another array reference passed to encode_json, the resulting $_ is printed (followed by a newline with -l).

There are a number of JSON plugins for perl. JSON::PP (PP for pure perl) is part of the core of perl, so should always be available. JSON::XS would be more efficient and also comes with a json_xs which is handy to use in shell scripts:

json_xs -e 'use List::Util "uniq"; $_ = [uniq @$_]' < array.json

If you're already familiar with perl, that means you don't need to learn a new language like that of jqs and can reuse what you know to process other formats like XML, YAML, etc.

map({key:.,value:1})|from_entries|keys_unsorted

as an alternative to unique seems to work.

map turns the array into:

[{"key":"cross","value":1},{"key":"base","value":1},{"key":"cross","value":1},{"key":"dunk","value":1}]

from_entries turns that into:

{"cross":1,"base":1,"dunk":1}

Removing the duplicates at that point since two elements of an object can't have the same key.

And keys_unsorted returns the keys of that object, in the original order.

A shorter/simpler alternative working on the same principle:

map({(.):1})|add|keys_unsorted

Or you can use a more generic programming language such as perl:

perl -l -0777 -MJSON::PP -MList::Util=uniq -pe '
  $_ = encode_json[uniq @{decode_json $_}]' array.json

-p is the sed mode, pull each record of array.json into $_ for the eexpression to work on.

-0777 sets the record separator to something impossible, meaning there's only one record being the full file (see also -gobble in newer versions of perl).

decode_json $_ decodes that record and returns an array reference, @{...} makes that a list passed to uniq, [...] makes the result another array reference passed to encode_json, the resulting $_ is printed (followed by a newline with -l).

map({key:.,value:1})|from_entries|keys_unsorted

as an alternative to unique seems to work.

map turns the array into:

[{"key":"cross","value":1},{"key":"base","value":1},{"key":"cross","value":1},{"key":"dunk","value":1}]

from_entries turns that into:

{"cross":1,"base":1,"dunk":1}

Removing the duplicates at that point since two elements of an object can't have the same key.

And keys_unsorted returns the keys of that object, in the original order.

A shorter/simpler alternative working on the same principle:

map({(.):1})|add|keys_unsorted

Or you can use a more generic programming language such as perl:

perl -l -0777 -MJSON::PP -MList::Util=uniq -pe '
  $_ = encode_json[uniq @{decode_json $_}]' array.json

-p is the sed mode, pull each record of array.json into $_ for the eexpression to work on.

-0777 sets the record separator to something impossible, meaning there's only one record being the full file (see also -gobble in newer versions of perl).

decode_json $_ decodes that record and returns an array reference, @{...} makes that a list passed to uniq, [...] makes the result another array reference passed to encode_json, the resulting $_ is printed (followed by a newline with -l).

There are a number of JSON plugins for perl. JSON::PP (PP for pure perl) is part of the core of perl, so should always be available. JSON::XS would be more efficient and also comes with a json_xs which is handy to use in shell scripts:

json_xs -e 'use List::Util "uniq"; $_ = [uniq @$_]' < array.json

If you're already familiar with perl, that means you don't need to learn a new language like that of jqs and can reuse what you know to process other formats like XML, YAML, etc.

added 80 characters in body
Source Link
Stéphane Chazelas
  • 584.5k
  • 96
  • 1.1k
  • 1.7k
map({key:.,value:1})|from_entries|keys_unsorted

as an alternative to unique seems to work.

map turns the array into:

[{"key":"cross","value":1},{"key":"base","value":1},{"key":"cross","value":1},{"key":"dunk","value":1}]

from_entries turns that into:

{"cross":1,"base":1,"dunk":1}

Removing the duplicates at that point since two elements of an object can't have the same key.

And keys_unsorted returns the keys of that object, in the original order.

A shorter/simpler alternative working on the same principle:

map({(.):1})|add|keys_unsorted

Or you can use a more generic programming language such as perl:

perl -l -0777 -MJSON::PP -MList::Util=uniq -pe '
  $_ = encode_json[uniq @{decode_json $_}]' array.json

-p is the sed mode, pull each record of array.json into $_ for the eexpression to work on.

-0777 sets the record separator to something impossible, meaning there's only one record being the full file (see also -gobble in newer versions of perl).

decode_json $_ decodes that record and returns an array reference, @{...} makes that a list passed to uniq, [...] makes the result another array reference passed to encode_json, the resulting $_ is printed (followed by a newline with -l).

map({key:.,value:1})|from_entries|keys_unsorted

as an alternative to unique seems to work.

map turns the array into:

[{"key":"cross","value":1},{"key":"base","value":1},{"key":"cross","value":1},{"key":"dunk","value":1}]

from_entries turns that into:

{"cross":1,"base":1,"dunk":1}

Removing the duplicates at that point since two elements of an object can't have the same key.

And keys_unsorted returns the keys of that object, in the original order.

A shorter/simpler alternative working on the same principle:

map({(.):1})|add|keys_unsorted

Or you can use a more generic programming language such as perl:

perl -l -0777 -MJSON::PP -MList::Util=uniq -pe '
  $_ = encode_json[uniq @{decode_json $_}]' array.json
map({key:.,value:1})|from_entries|keys_unsorted

as an alternative to unique seems to work.

map turns the array into:

[{"key":"cross","value":1},{"key":"base","value":1},{"key":"cross","value":1},{"key":"dunk","value":1}]

from_entries turns that into:

{"cross":1,"base":1,"dunk":1}

Removing the duplicates at that point since two elements of an object can't have the same key.

And keys_unsorted returns the keys of that object, in the original order.

A shorter/simpler alternative working on the same principle:

map({(.):1})|add|keys_unsorted

Or you can use a more generic programming language such as perl:

perl -l -0777 -MJSON::PP -MList::Util=uniq -pe '
  $_ = encode_json[uniq @{decode_json $_}]' array.json

-p is the sed mode, pull each record of array.json into $_ for the eexpression to work on.

-0777 sets the record separator to something impossible, meaning there's only one record being the full file (see also -gobble in newer versions of perl).

decode_json $_ decodes that record and returns an array reference, @{...} makes that a list passed to uniq, [...] makes the result another array reference passed to encode_json, the resulting $_ is printed (followed by a newline with -l).

added 80 characters in body
Source Link
Stéphane Chazelas
  • 584.5k
  • 96
  • 1.1k
  • 1.7k
map({key:.,value:1})|from_entries|keys_unsorted

as an alternative to unique seems to work.

map turns the array into:

[{"key":"cross","value":1},{"key":"base","value":1},{"key":"cross","value":1},{"key":"dunk","value":1}]

from_entries turns that into:

{"cross":1,"base":1,"dunk":1}

Removing the duplicates at that point since two elements of an object can't have the same key.

And keys_unsorted returns the keys of that object, in the original order.

A shorter/simpler alternative working on the same principle:

map({(.):1})|add|keys_unsorted

Or you can use a more generic programming language such as perl:

perl -l -0777 -MJSON::PP -MList::Util=uniq -pe '
  $_ = encode_json[uniq @{decode_json $_}]' array.json
map({key:.,value:1})|from_entries|keys_unsorted

as an alternative to unique seems to work.

map turns the array into:

[{"key":"cross","value":1},{"key":"base","value":1},{"key":"cross","value":1},{"key":"dunk","value":1}]

from_entries turns that into:

{"cross":1,"base":1,"dunk":1}

Removing the duplicates at that point since two elements of an object can't have the same key.

And keys_unsorted returns the keys of that object, in the original order.

Or you can use a more generic programming language such as perl:

perl -l -0777 -MJSON::PP -MList::Util=uniq -pe '
  $_ = encode_json[uniq @{decode_json $_}]' array.json
map({key:.,value:1})|from_entries|keys_unsorted

as an alternative to unique seems to work.

map turns the array into:

[{"key":"cross","value":1},{"key":"base","value":1},{"key":"cross","value":1},{"key":"dunk","value":1}]

from_entries turns that into:

{"cross":1,"base":1,"dunk":1}

Removing the duplicates at that point since two elements of an object can't have the same key.

And keys_unsorted returns the keys of that object, in the original order.

A shorter/simpler alternative working on the same principle:

map({(.):1})|add|keys_unsorted

Or you can use a more generic programming language such as perl:

perl -l -0777 -MJSON::PP -MList::Util=uniq -pe '
  $_ = encode_json[uniq @{decode_json $_}]' array.json
added 188 characters in body
Source Link
Stéphane Chazelas
  • 584.5k
  • 96
  • 1.1k
  • 1.7k
Loading
Source Link
Stéphane Chazelas
  • 584.5k
  • 96
  • 1.1k
  • 1.7k
Loading