Python elasticsearch return only entries that have a certain field

Question

I'm trying to query an Elasticsearch, and only get results that have a certain field.

How do I query for documents that have field fields.EventData.PGID and ignore ones that don't?

datadict = es.search(index=idx1, 
                     q='run_id:"Run001" AND "fields.EventData.PGID exists"', 
                     sort='fields.System.TimeCreated.SystemTime',
                     size=1000)

The way events are logged in the ES is inconsistent and such I need to find only ones where a PGID was logged. I tried doing a try block in the Python code trying to access the field from the returned values and ignoring it if I get a KeyError, but due to the limit on how many items you can receive as a query result, in some cases I have all my results lacking a PGID so I just end up wasting a query and am unable to access actual results, so I would like this filtering to happen at the query level.

kgf3JfUtW · Accepted Answer · 2017-03-31 19:27:36Z

1

You can try with this filtered query

{
  "size": 1000,
  "query": {
    "filtered": {
      "filter": {
        "bool": {
          "must": {
            "exists": {
              "field": "fields.EventData.PGID"
            },
            "term": {
              "run_id": "Run001"
            }
          }
        }
      }
    }
  }
}

I think you can also add the sorting to the query

edited Mar 31, 2017 at 19:27

kgf3JfUtW

15.1k12 gold badges63 silver badges92 bronze badges

answered Mar 31, 2017 at 15:37

ML_TN

7296 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Peter Mortensen Over a year ago

Is that valid JSON? What about "size": 1000"query"?

Collectives™ on Stack Overflow

Python elasticsearch return only entries that have a certain field

1 Answer 1

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Related