19

TL;DR - How do I check whether one-of and all-of a nested array meet specified criteria?

I have a document. Each document has an array of nested outer objects, who themselves have a list of nested inner objects. I need to perform a filter for all documents where at least one of the document's outer nested objects match. When I say match, I mean that all the outer nested objects' inner objects match in some way. Here's an example mapping for reference;

{ "document" : {
    "properties" : {
      "name" : {
        "type" : "string"
      },
      "outer" : {
        "type" : "nested",
        "properties" : {
          "inner" : {
            "type" : "nested",
            "properties" : {
              "match" : {
                "type" : "string",
                "index" : "not_analyzed"
              },
              "type" : {
                "type" : "string",
                "index" : "not_analyzed"
              }
    }}}}}}
}

If the document has no outer/inner objects it is considered to match. But to make things worse the inner objects need to be considered to match differently depending on the type in a kind of conditional logic manner (eg CASE in SQL). For example, if the type were the term "Country" then inner object would be considered to match if the match were a specified country code such as ES. A document may have inner objects of varying type and there is not guarantee that specific types will exist.

Coming from a imperative (Java) programming background I am having incredible trouble figuring out how to implement this kind of filtering. Nothing I can think of even vaguely matches this behaviour. Thus far all I have is the filtered query;

"filtered" : {
      "query" : {
        "match_all" : { }
      },
      "filter" : {
        "bool" : {
          "should" : {
            "missing" : {
              "field" : "outer.inner.type"
            }
    }}}}
}

So, the question is...

How can I filter to documents who have at least one outer object which has all inner objects matching based on the type of inner object?

Further details By Request -

Example Document JSON

{
    "name":"First",
    "outer":[
        {
            "inner":[
                {"match":"ES","type":"Country"},
                {"match":"Elite","type":"Market"}
            ]
        },{
            "inner":[
                {"match":"GBR","type":"Country"},
                {"match":"1st Class","type":"Market"},
                {"match":"Admin","type":"Role"}
            ]
        }
    ],
    "lockVersion":0,"sourceId":"1"
}

The above example should come through the filter if we were to provide "1st Class" market and the country "GRB" because the second of the two outer objects would be considered a match because both inner objects match. If, however, we provided the country country "GRB" and the market "Elite" then we would not have this document returned because neither of the outer objects would have bother of their inner objects match in their entirety. If we wanted the second outer object to match then all three inner would need to match. Take note that there is an extra type in the third inner. This leads to a situation where if a type exists then it needs to have a match for it else it doesn't need to match because it is absent.

Improve this question
3
  • Could you post some example data, maybe in a gist or something? It needs to include enough variety to cover your various cases. I think I might know how to solve this, but I don't want to spend all morning constructing dummy data to test it. Commented Sep 16, 2015 at 15:51
  • 1
    @SloanAhrens - I've posted up a single example document with some details, and if that's still not enough let me know and I'll throw together some more examples. I'm not actually using production data/mapping in the question above so I will have to manually convert the data to match the question. Commented Sep 16, 2015 at 16:24
  • Nope, that did it, I think. Commented Sep 16, 2015 at 16:37

2 Answers 2

Reset to default
20

One of Nested Array

Having one of a nested array matching some criteria turns out to be very simple. A nested filter evaluates to matching/true if any of the array of nested objects match the specified inner filters. For example, given an array of outer objects where one of those objects has a field match with the value "matching" the following would be considered true.

"nested": {
   "path": "outer",
   "filter": {
       "term" : { "match" : "matching" } 
   }
}

The above will be considered true/matching if one of the nested outer objects has a field called match with the value "matching".

All of Nested Array

Having a nested filter only be considered matching if all of the nested objects in an array match is more interesting. In fact, it's impossible. But given that it is considered matching if only one of the nested objects match a filter we can reverse the logic and say "If none of the nested objects don't match" to achieve what we need. For example, given an array of nested outer.inner objects where all of those objects has a field match with the value "matching" the following would be considered true.

"not" : {
   "nested": {
      "path": "outer.inner",
      "filter": {
          "not" : {
              "term" : { "match" : "matching" } 
          }
      }
   }
}

The above will be considered true/matching because none of the nested outer.inner objects don't (double negative) have a field called match with the value "matching". This, of course, is the same as all of the nested inner objects having a field match with the value "matching".

Missing Any Nested Objects

You can't check whether a field containing nested objects is missing using the traditional missing filter. This is because nested objects aren't actually in the document at all, they are stored somewhere else. As such missing filters will always be considered true. What you can do however, is check that a match_all filter returns no results like so;

"not": {
   "nested": {
      "path": "outer",
      "filter": {
          "match_all": {}
       }
    }
 }

This is considered true/matching if match_all finds no results.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

6

Well, it's a doozy, but this query seems to do what you want:

POST /test_index/_search
{
   "query": {
      "filtered": {
         "filter": {
            "nested": {
               "path": "outer",
               "filter": {
                  "bool": {
                     "must": [
                        {
                           "nested": {
                              "path": "outer.inner",
                              "filter": {
                                 "bool": {
                                    "must": [
                                       { "term": { "outer.inner.type": "Market" } },
                                       { "term": { "outer.inner.match": "1st Class" } }
                                    ]
                                 }
                              }
                           }
                        },
                        {
                           "nested": {
                              "path": "outer.inner",
                              "filter": {
                                 "bool": {
                                    "must": [
                                       { "term": { "outer.inner.type": "Country" } },
                                       { "term": { "outer.inner.match": "GBR" } }
                                    ]
                                 }
                              }
                           }
                        }
                     ]
                  }
               }
            }
         }
      }
   }
}

Here is some code I used to test it:

http://sense.qbox.io/gist/f554c2ad2ef2c7e6f5b94b1ddb907813370f4edc

Let me know if you need some explanation of the logic; it is kind of involved.

Improve this answer

2 Comments

Sorry for the late reply and thank you for your help, but this wont find results that don't have any outer objects. Also, because we can't be sure specific type will definitely exist this wont find any document if the Country type is not there. I think maybe I need to put more clarifying details into my question. This does help me a lot though.
+1 - I actually figured it out in the end, but thank you again for all your help. I posted an answer with the details of what I needed.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.