Why does pipeline aggs query fail if it includes filter aggs?

Question

I am using Elasticsearch as a database.

I am going to use aggregation.

    POST new_logs/_search
    {
      "query": {
        "bool": {
          "must": [
            {
              "term": {
                "base.logClass.keyword": "Access"
              }
            }
          ]
        }
      },
      "size": 0, 
      "aggs": {
        "Rule1": {
          "terms": { "field": "source.srcIp" },
          "aggs": {
            "MinTime": {
              "min": { "field": "base.receiveTime" }
            },
            "MaxTime": {
              "max": { "field": "base.receiveTime" }
            }  
          }
        },
        "Rule2": {
          "filter": { "range": { "base.receiveTime": { "gte": "2022-06-22 11:27:00", "lte": "2022-06-22 11:29:00" } } 
          },
          "aggs": {
            "SubFilter": {
              "filter": { "term": { "base.subLogClass.keyword": "Login" }
              },
              "aggs": {
                "SourceIP": { 
                  "terms": { "field": "source.srcIp" },
                  "aggs": {
                      "DestinationIP": { "terms": { "field": "destination.dstIp" } 
                    }
                  }
                },
                "MinTime": {
                  "min": { "field": "base.receiveTime" }
                },
                "MaxTime": {
                  "max": { "field": "base.receiveTime" }
                }
              }
            }
          }
        },
        "Logic1": {
          "max_bucket": {
            "buckets_path": "Rule1>MinTime"
          }
        },
        "Logic2": {
          "min_bucket": {
            "buckets_path": "Rule2>SubFilter>MinTime"
          }
        }
      }
    }

As you can see in query, there are two aggs - Rule1 and Rule2.

Rule2 is using filter aggs and Rule1 is not using.

When i am going to use pipeline aggs, Logic1 is ok but Logic2 is failed.

This is the error message.

    {
      "error" : {
        "root_cause" : [
          {
            "type" : "action_request_validation_exception",
            "reason" : "Validation Failed: 1: The first aggregation in buckets_path must be a multi-bucket aggregation for aggregation [Logic2] found :org.elasticsearch.search.aggregations.bucket.filter.FilterAggregationBuilder for buckets path: Rule2>SubFilter>MinTime;"
          }
        ],
        "type" : "action_request_validation_exception",
        "reason" : "Validation Failed: 1: The first aggregation in buckets_path must be a multi-bucket aggregation for aggregation [Logic2] found :org.elasticsearch.search.aggregations.bucket.filter.FilterAggregationBuilder for buckets path: Rule2>SubFilter>MinTime;"
      },
      "status" : 400
    }

I'm not sure what went wrong.

If there is a filter aggs, is it not possible to use the pipeline aggs?

I am asking for help from people who have a lot of experience with Elasticsearch.

Thank you for help.

Answer 1

The filter aggregation is a single bucket aggregation. The min_bucket complains that it needs a multi-bucket aggregation at first level of input path.

You might be able to use the filters aggregation, which is a multi-bucket filter or nest the filter aggregations under Rule1, because you're already doing these aggregations and you could filter a subset from Rule1.