Bash - loop through files and pull JSON value from corresponding keys in a master key file

Question

I'm trying to make a bash script that loops through a file that contains a list of files (so it goes through each one of these files). While the files are open, I want to use another program (jq) to extract the values from the key value pair of the JSON files. Here's what I'm working with so far, kept pretty basic so I can try to figure out where I'm going wrong. I'm not concerned about the jq command, I've had it working where I am able to loop through all the files and pull out the values of a single key, but this ends up being extremely tedious because I have to manually change it for each key.

TL;DR: 'master' is the file that contains the full path of each one of the JSON files I want to open. 'key.txt' is a file that contains all of the possible keys within these JSON files. $KEY is the syntax for jq to search through the open file for that specific key and return the value of that key. I want to open up a file in the master list, loop through all the possible keys, and return the values of these keys into a text file called 'list.txt'.

Any help would be greatly appreciated.

#!/bin/bash

master="path/masterlist.txt"
while read master
do
        for KEY in 'key.txt'
        do
                jq '. .'$KEY'' $f
                echo $KEY
        done >> list.txt
        echo $master
done < list.txt

Answer 1

This is me revisiting an old answer after gaining a bit more experience with jq.

The user in the question says

I'm not concerned about the jq command [...]

... but in fact, the jq command is the key to making this work efficiently.

To recap: The user wants to extract all the values for all keys listed in some text file, keys.txt. They want to do this for all JSON files listed in another file, list.txt.

To get all (decoded) values for a set of keys, a, b, c, from a set of files, file1, file2, file3, using jq, you may do it like this:

jq -r '.[$ARGS.positional[]] // empty' file1 file2 file3 --args a b c

This passes the strings to be used as keys into jq, which accepts them as elements in the array $ARGS.positional. That array is expanded, and its elements are used as the keys for getting the values from the input. Here, I've chosen to return nothing (empty) if the key's value does not exist or is null.

Adding the shell plumbing around this kernel of operation, assuming that both the list of files and the list of keys may be arbitrarily long and that the two files contain lines that are quoted as necessary:

xargs sh -c '
    xargs jq -r ".[\$ARGS.positional[]] // empty" "$@" --args <keys.txt
' sh <list.txt >result.txt

This takes the list of filenames and calls a short inline shell script with batches of these. The inline script uses xargs to call jq with the filenames given ("$@") and set of keys read from the key list.

The resulting list of values is written to result.txt.

---

Old answer is below:

After reading what it is you say you want to do:

master="path/masterlist.txt"

while read json_path; do
    while read key; do
        printf 'File="%s", Key="%s"\n' "$json_path" "$key"
        jq ". .'$key'" "$json_path"
    done <key.txt
done <"$master" >list.txt

The list of JSON files are read from $master and each path is assigned to json_path in turn.

In the inner loop, the keys are read from key.txt and assigned to key in turn. The jq utility is invoked on $json_path with the argument . .'$key' where $key is expanded with the value of the current key.

All output is put into list.txt.

There might be some optimization to be done with the inner loop. It seem unnecessary to read all the keys in every single iteration of the outer loop, for example.

---

Annotated version of your script:

master="path/masterlist.txt"        # this is never used
while read master                   # this will set $master to something from list.txt
do
        for KEY in 'key.txt'        # $KEY will be the string 'key.txt' (nothing else)
        do
                jq '. .'$KEY'' $f   # $f is empty
                echo $KEY
        done >> list.txt            # better to >list.txt after "done" of outer loop
        echo $master
done < list.txt                     # you probably don't want to read from this file

Answer 2

From reading the script you posted, I wouldn't think that $f is being assigned properly. Also, the quoting in the jq command may be causing a problem. And then there's the fact that you are attempting to read a bare word master in a while loop, which may also be trying to read in list.txt at the same time as it's trying to append out to list.txt inside the loop.

#!/bin/bash

master="path/masterlist.txt"

cat $master | while read f; do
    for KEY in 'key.txt'; do
        echo $(jq ". .'$KEY'" $f) >> list.txt
    done
    echo $master
done

This will take each line from the $master file, and assign that into the variable $f in the loop, and run the jq query, appending the output to list.txt.