jq filter for only 'null' values

Question

I have a json file with a load of AWS CloudWatch logs (generated from a CLI command). I'm trying to use jq to only return values for entries that don't have a 'retentionInDays' field. I have the following which returns everything as I want, but I can't seem to filter out the results that do have retentionInDays.

# Working output (unfiltered)
jq ".logGroups[] | { log_name: .logGroupName, log_arn: .arn, retention_scheme: .retentionInDays }" cwlogs.json

I've tried a couple of things, but either get an error, or it completes and outputs nothing:

# Doesn't return anything
jq '.logGroups[] | { log_name: .logGroupName, log_arn: .arn, retention_scheme: select(.retentionInDays | contains ("null")?) }' cwlogs.json

# Errors with "jq: error (at cwlogs.json:760): number (7) and string ("null") cannot have their containment checked"
jq '.logGroups[] | { log_name: .logGroupName, log_arn: .arn, retention_scheme: select(.retentionInDays | contains ("null")) }' cwlogs.json

# Hangs forever
jq '.logGroups[] | select(.retentionInDays != "null").type'

Update: Testable segment of JSON I'm using

{
    "logGroups": [
        {
            "storedBytes": 0,
            "metricFilterCount": 0,
            "creationTime": 1234,
            "logGroupName": "/aws/elasticbeanstalk/docker",
            "retentionInDays": 7,
            "arn": "longarnhere"
        },
        {
            "storedBytes": 0,
            "metricFilterCount": 0,
            "creationTime": 1245,
            "logGroupName": "/aws/elasticbeanstalk/nginx",
            "arn": "longarnhere"
        }
    ]
}

Are you talking about extracting records that don't have a specific field, or records that has a null value for an existing field? The title and the text in the question are contradictory on this point. Also, by "filter out", do you mean "extract" or "discard"? — Kusalananda
– Kusalananda ♦, Commented Nov 21, 2019 at 9:57

Kusalananda · Accepted Answer · 2022-08-18 17:25:14Z

I'm assuming you want to get the logGroups entries that don't have a retentionInDays key at all.

$ jq '.logGroups[] | select( has("retentionInDays") == false )' file.json
{
  "storedBytes": 0,
  "metricFilterCount": 0,
  "creationTime": 1245,
  "logGroupName": "/aws/elasticbeanstalk/nginx",
  "arn": "longarnhere"
}

If you want an array of these (likely, if there may be more than one):

$ jq '.logGroups | map(select( has("retentionInDays") == false ))' file.json
[
  {
    "storedBytes": 0,
    "metricFilterCount": 0,
    "creationTime": 1245,
    "logGroupName": "/aws/elasticbeanstalk/nginx",
    "arn": "longarnhere"
  }
]

You could also use has("retentionInDays") | not in place of has("retentionInDays") == false.

magor · Accepted Answer · 2019-11-21 09:58:29Z

8

You can use the alternative operator : // .

For example:

echo '{"a" : "b"}' | jq '. | .c // "Null"'

Or in your example filtering can be done by adding type too to the filter:

jq '.logGroups[] | select (.retentionInDays.type != null)'

edited Nov 21, 2019 at 9:58

answered Nov 21, 2019 at 9:52

magor

3,9223 gold badges16 silver badges30 bronze badges

1

I've tried your example but I get jq: error (at cwlogs.json:760): Cannot index number with string "type"

Mcar49
– Mcar49

2019-11-21 10:01:05 +00:00
Commented Nov 21, 2019 at 10:01
could you post an example json so that we see what we need to parse? or if it''s too big, just minimize it to the part we need

magor
– magor

2019-11-21 10:05:17 +00:00
Commented Nov 21, 2019 at 10:05
I get the same error as @Mcar49; this can be fixed for me by removing the reference to type and just saying jq '.logGroups[] | select (.retentionInDays != null)'

awerchniak
– awerchniak

2021-03-08 14:57:18 +00:00
Commented Mar 8, 2021 at 14:57

Add a comment |

Stack Exchange Network

jq filter for only 'null' values

2 Answers 2

You must log in to answer this question.

Hot Network Questions

jq filter for only 'null' values

2 Answers 2

You must log in to answer this question.

Related

Hot Network Questions