Bash script traversing a multi-line JSON object using jq

Question

I have to curl to a site (statuscake.com) that sends multiple items back in a JSON, each line of which contains multiple items. I want to extract from each line two of them, WebsiteName and TestID, so I can check if WebsiteName matches the one I'm interested in, get the TestID out and pass this to a second curl statement to delete the test.

Although it's more complex, the JSON that comes back is essentially of the form

[{"TestID": 123, "WebsiteName": "SomeSite1"}, {"TestID": 1234, "WebsiteName": "SomeSite2"}]

I can't seem to find a magic jq command to do it all in one - if there is one, I'd be really happy to see it.

I've got

cat $data | jq '[.[] | .WebsiteName]'

to get an array of the website names (and a very similar one for the TestIDs, but I think I've done something daft. data is the information coming back from the curl to get the JSON and that's populated OK.

I want to be able to assign these to two arrays, names and ids, then search names for the index of the relevant name, grab the id from ids and pass that to the curl. Unless there's a better way.

Any advice please?

What version of bash? Do you have any guarantee about what your website names are going to look like? (i.e. Can they have spaces in them? Can they have other shell metacharacters?) — Etan Reisner
– Etan Reisner, Commented Apr 14, 2015 at 13:21
Spaces, yes (not all of them, but many). They shouldn't have other meta characters, they're designed to be human friendly names for scanning in a large display of tests. — Lewis
– Lewis, Commented Apr 14, 2015 at 15:25

BeniBela · Accepted Answer · 2015-04-14 13:36:19Z

3

My Xidel can do it all at once by selecting the JSON with a XPath-like query:

E.g. return all ids where the WebsiteName contains "site2" from an array of objects:

xidel /tmp/x.json -e '$json()[contains((.).WebsiteName, "site2")]/TestID'

Or e.g. to download the original JSON and then make the HTTP request with the ids:

xidel http://statuscake.com/your-url... -f '$json()[contains((.).WebsiteName, "site2")]/TestID!x"/your-delete-url{.}..."'

answered Apr 14, 2015 at 13:36

BeniBela

17.1k4 gold badges48 silver badges55 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Lewis Over a year ago

If the delete URL has to end with the TestID, some url/?TestID=myIDToDelete how do I pass that from this statement?

Lewis Over a year ago

Sorry, should have read the documentation more thoroughly, the {.} is the TestID isn't it. Thank you.

BeniBela Over a year ago

Yes. More precisely . is the TestID and {} marks it as something that should be evaluated and inserted

user3899165 · Accepted Answer · 2015-04-15 19:34:53Z

If I'm getting your question right, it sounds like what you want is to, for each element, select those where .WebsiteName == "needle", and then get .TestID from it. You can do just that:

.[] | select(.WebsiteName == "needle") | .TestID

If you want an array as the result, you can wrap the above script in square brackets.

The jq filters startswith and endswith may be of interest to you. If you're going to pass the result back to cURL, you may also be interested in the @sh formatting filter and the -r command-line flag.

Etan Reisner · Accepted Answer · 2015-04-14 13:39:47Z

Assuming you have a bash 4+ and assuming the json is valid (does not contain newlines in strings, etc.) this works:

$ echo "$data"
[{"TestID": 123, "WebsiteName": "SomeSite1"}, {"TestID": 1234, "WebsiteName":
"SomeSite2"}, {"TestID": 555, "WebsiteName": "foo*ba@r blah[54]quux{4,5,6}"}]
$ declare -A arr
$ while IFS= read -r line; do
    eval "$line"
done < <(jq -M -r '.[] | @sh "arr[\(.WebsiteName)]+=\(.TestID)"' <<<"$data")
$ declare -p arr
declare -A arr='(["foo*ba@r blah[54]quux{4,5,6}"]="555" [SomeSite2]="1234" [SomeSite1]="123" )'

jq170727 · Accepted Answer · 2017-08-25 15:52:40Z

1

Here is a solution using only jq primitives.

  .[]
| if .WebsiteName == "SomeSite1" then .TestID else empty end

This is essentially the same as Santiago's answer but if you are new to jq it may be informative because select/1 is defined as

 def select(f): if f then . else empty end;

answered Aug 25, 2017 at 15:52

jq170727

15k6 gold badges52 silver badges61 bronze badges

Collectives™ on Stack Overflow

Bash script traversing a multi-line JSON object using jq

4 Answers 4

3 Comments

Comments

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

3 Comments

Comments

Comments

Comments

Related