Why is the JSON content from heredoc not parsable?

Question

I have a JSON fragment.

The following does not work:

VALUE=<<PERSON
{
  "type": "account",
  "customer_id": "1234",
  "customer_email": "[email protected]"  
}
PERSON
echo -n "$VALUE" | python -m json.tool

The result is:

No JSON object could be decoded

Doing the same with jq, i. e.

echo -n "$VALUE" | jq '.'

There is no output.

There is the same behavior for the following:

VALUE=<<PERSON
'{
  "type": "account",
  "customer_id": "1234",
  "customer_email": "[email protected]"  
}'
PERSON
echo -n "$VALUE" | python -m json.tool

Response:

No JSON object could be decoded

But the following works:

VALUE='{
  "type": "account",
  "customer_id": "1234",
  "customer_email": "[email protected]"
}'
echo -n "$VALUE" | jq '.'
echo -n "$VALUE" | python -m json.tool

Answer 1

VALUE=<<PERSON
some data
PERSON

echo "$VALUE"

No output.

A here-document is a redirection, you can't redirect into a variable.

When the command line is parsed, redirections are handled in a separate step from variable assignments. Your command is therefore equivalent to (note the space)

VALUE= <<PERSON
some data
PERSON

That is, it assigns an empty string to your variable, then redirects standard input from the here-string into the command (but there is no command, so nothing happens).

Note that

<<PERSON
some data
PERSON

is valid, as is

<somefile

It's just that there is no command whose standard input stream can be set to contain the data, so it's just lost.

This would work though:

VALUE=$(cat <<PERSON
some data
PERSON
)

Here, the command that receives the here-document is cat, and it copies it to its standard output. This is then what is assigned to the variable by means of the command substitution.

In your case, you could instead use

python -m json.tool <<END_JSON
JSON data here
END_JSON

without taking the extra step of storing the data in a variable.

---

It may also be worth while to look into tools like jo to create the JSON data with the correct encoding:

For example:

jo type=account customer_id=1234 [email protected] random_data="some^Wdata"

... where ^W is a literal Ctrl+W character, would output

{"type":"account","customer_id":1234,"customer_email":"[email protected]","random_data":"some\u0017data"}

So the command in the question could be written

jo type=account customer_id=1234 [email protected] |
python -m json.tool

Answer 2

Because the variable isn't being set by your heredoc:

$ VALUE=<<PERSON  
> {    
>   "type": "account",  
>   "customer_id": "1234",  
>   "customer_email": "[email protected]",  
> }  
> PERSON
$ echo "$VALUE" 

$

If you want to use a heredoc to assign a value to a variable, you need something like:

$ read -d '' -r VALUE <<PERSON  
{    
  "type": "account",  
  "customer_id": "1234",  
  "customer_email": "[email protected]",  
}   
PERSON

Answer 3

It is because the way you have defined a here-doc to use with a JSON is wrong. You need to use it as

VALUE=$(cat <<EOF
{  
  "type": "account",  
  "customer_id": "1234",  
  "customer_email": "[email protected]",  
}
EOF
)

and doing printf "$VALUE" should dump the JSON as expected.

Answer 4

Heredocs and variables don't mix well or at least not in this way. You can either…

Pass the heredoc as the standard input of an application

python -m json.tool <<PERSON  
{
  "type": "account",
  "customer_id": "1234",
  "customer_email": "[email protected]",
}
PERSON

or…

Store multi-line text in a shell variable

VALUE='{
  "type": "account",
  "customer_id": "1234",
  "customer_email": "[email protected]",
}'

I used single quotes to avoid the need to escape the inner double quotes. Of course you can also use double quotes, e. g. if you need to expand parameters:

VALUE="{
  \"type\": \"account\",
  \"customer_id\": ${ID},
  \"customer_email\": \"${EMAIL}\",
}"

Then you can use the variable value later on.

echo -n "$VALUE" | python -m json.tool