2

During everyday shell sessions, I very often find myself needing to assign data from a JSON document (extracted via some jq filter) to a Zsh shell parameter: JSON scalars to Zsh scalars, JSON arrays to Zsh arrays, and JSON objects to Zsh associative arrays. The problem—and what previously asked questions don’t seem to tackle—is that the data often contains newlines (and even NUL bytes), making this a rather nontrivial task.

Here is what I have come up with so far:

function assign-from-json {
    local -A opts && zparseopts -A opts -D -F -M -- a A && typeset -r opts
    if [[ $# -ne 3 || ( -v opts[-a] && -v opts[-A] ) ]] ; then
        >&2 printf 'Usage: %s [-a|-A] NAME FILTER JSON\n' $0
        return 2
    fi
    if [[ -v opts[-a] ]] ; then
        local -a lengths && { lengths=( "${(@f)$( jq -r "$2 | .[] | tostring | length" <<< $3 )}" ) || return $? } && typeset -r lengths
        local data && { data="$( jq -j "$2 | .[] | tostring" <<< $3 )" || return $? } && typeset -r data
        local elem
        local -a elems
        for length in "${lengths[@]}" ; do
            read -u 0 -k $length elem
            elems+=$elem
        done <<< $data
        eval "${(q)1}"='( "${elems[@]}" )'
    elif [[ -v opts[-A] ]] ; then
        local transformed_json && { transformed_json="$( jq "$2 | to_entries | map(.key, .value)" <<< $3 )" || return $? } && typeset -r transformed_json
        assign-from-json -a $1 "." $transformed_json
    else
        eval "${(q)1}"="${(q)$( jq -r $2 <<< $3 )}"
    fi
}

In most cases it works quite well:

% json='
{
    "scalar": "Hello, world",
    "array": [1, 2, 3],
    "scary_scalar": "\nNewlines\u0000NUL bytes\ttabs",
    "scary_array": [
        "A\nvery\u0000scary\nvalue",
        "A less scary value",
        "eh"
    ]
}
'
% assign-from-json scalar '.scalar' $json && printf '%q\n' $scalar
Hello,\ world
% typeset -a array && assign-from-json -a array '.array' $json && printf '%q\n' "${array[@]}"
1
2
3
% assign-from-json scary_scalar '.scary_scalar' $json && printf '%q\n' $scary_scalar
$'\n'Newlines$'\0'NUL\ bytes$'\t'tabs
% typeset -a scary_array && assign-from-json -a scary_array '.scary_array' $json && printf '%q\n' "${scary_array[@]}"
A$'\n'very$'\0'scary$'\n'value
A\ less\ scary\ value
eh
% typeset -A assoc && assign-from-json -A assoc '.' $json && printf '%q -> %q\n' "${(@kv)assoc}"
array -> \[1,2,3\]
scary_array -> \[\"A\\nvery\\u0000scary\\nvalue\",\"A\ less\ scary\ value\",\"eh\"\]
scary_scalar -> $'\n'Newlines$'\0'NUL\ bytes$'\t'tabs
scalar -> Hello,\ world

Unfortunately it seems to struggle with trailing newlines:

% assign-from-json bad_scalar '.' '"foo\n"' && printf '%q\n$ $bad_scalar
foo
# expected: foo$'\n'
  1. I assume the problem with trailing newlines is due to command substitution removing them. Do you see an easy way to fix it?
  2. One can do assign-from-json -A assoc ... even if assoc is not typeset as an associative array. How can I prevent that from being possible?
  3. Do you see any other problems with the code?
Improve this question
3
  • While zsh doesn't suck quite as much as bash for programming. This is still a task that would be better done in perl or python or some other language actually intended for programming. e.g. with the perl JSON module, just give it a string containing json text and it will return you a perl data structure containing all the scalar, array, and hash elements within the json document. You can access them as you would any other perl data structure - e.g. $j->{scary array}->[1] is A less scary value. I'll post a minimal example below. Commented Jul 6, 2022 at 9:41
  • @cas I fully agree with your sentiment and always use Python for more involved tasks, but I still prefer the convenience of Zsh for one-off things. Commented Jul 6, 2022 at 9:56
  • I get that, I really do. But if it's an f-ing PITA to work with, it's not really "convenience", is it? (btw, i write lots of little throwaway perl scripts for one-off things. I like the convenience :-). I guess I use perl for the things you use zsh scripting for). Commented Jul 6, 2022 at 10:06

2 Answers 2

Reset to default
1

A common workaround for keeping trailing newlines in a command substitution is to append a value, so that the newlines are no longer trailing. The dummy value is then removed from the variable:

v1=$'line1\nline2\n\n\n'
v2=$(print $v1)
v3=$(printf $v1; print X);v3=${v3%X}
typeset -p v1 v2 v3

Output:

typeset v1=$'line1\nline2\n\n\n'
typeset v2=$'line1\nline2'
typeset v3=$'line1\nline2\n\n\n'

This does complicate handling return codes, so you might need something like this:

local data \
  && {data="$( jq -j "$2 | .[] | tostring" <<< $3;
    ret=$?;
    print X;
    return $ret)" \
    || return $? } \
  && data=${data:%X} \
  && typeset -r data

Some other options for retaining trailing newlines are listed in this answer. Unfortunately, I don't think zsh has a 'command substitution flag' for situations like this (yet).

The t parameter expansion flag can be used to test the type of a variable and determine if a name references an associative array or something else:

function vtype {
  tt=${(Pt)1}
  if [[ $tt == association* ]]; then
      print "$1: YES [$tt]"
  else
      print "$1: no  [$tt]"
  fi
}
typeset -A a1; vtype a1
typeset -a a2; vtype a2
integer i1; vtype i1
typeset -AlxRr a3; vtype a3

The code above also uses the P expansion flag to interpret the positional parameter value in $1 as yet another parameter name. The test uses a wildcard (...*) because the type identifier from t may have multiple components. The output:

a1: YES [association]
a2: no  [array]
i1: no  [integer]
a3: YES [association-right_blanks-lower-readonly-export]

Good luck getting zsh to do what you need. I often find that stretching a tool slightly beyond its comfortable limits works better than completely rebuilding with a separate, less familiar tool. The hard part is defining 'slightly beyond'.

Improve this answer
0

You should use a language like perl or python instead of a shell (even zsh).

For example, using the perl JSON module:

$ cat json-example.pl 
#!/usr/bin/perl

use strict;
use JSON;

my $json_text;
# slurp the entire input file into a single string.
do { $/=''; $json_text=<> };

my $j = decode_json($json_text);

print $j->{scary_array}->[1], "\n";

Sample run:

$ chmod +x json-example.pl

$ ./json-example.pl input.json 
A less scary value

See the man pages for perldata, perllol, perldsc, perlref, perlreftut for details on perl data structures.

json data actually maps extremely well to perl data structures - both when writing code and if dumped for debugging with modules like Data::Dumper or Data::Dump, the representation is almost identical to the json data.

e.g. here's what $j looks like if dumped with Data::Dump's dd function:

{
  array => [1, 2, 3],
  scalar => "Hello, world",
  scary_array => ["A\nvery\0scary\nvalue", "A less scary value", "eh"],
  scary_scalar => "\nNewlines\0NUL bytes\ttabs",
}

That is actually how you would define a hash containing that data in perl syntax. You could copy-paste that into your script, assign it to a variable (e.g. my $var = { ... };), and it would compile without a problem.

my $var = {
  array => [1, 2, 3],
  scalar => "Hello, world",
  scary_array => ["A\nvery\0scary\nvalue", "A less scary value", "eh"],
  scary_scalar => "\nNewlines\0NUL bytes\ttabs",
};

print $var->{array}->[0], "\n";

would print 1.

More importantly, this data is not in any way scary in perl. It's just data.

Module notes:

  • Data::Dumper is a core perl module and is included with perl.
  • Data::Dump is not and needs to be installed separately (e.g. apt install libdata-dump-perl on Debian etc, or install with perl's cpan utility). I prefer it to the core Data::Dumper module.
  • JSON is not a core perl module either (on Debian, install with apt install libjson-perl. Optionally install libjson-xs-perl too, this speeds up the JSON module with compiled C code.
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.