Optionally include 'Set Difference' for filter aggregation.

[Kallin]

A project that I'm working on involves breaking down an index via many nested filter aggregations. Imagine it being for something like a website visitor funnel:

1. Of all the visitors, bucket the ones who created an account
2. Of account creators, bucket those over age 30
3. etc..

What I also end up doing are creating the Difference filters, albeit manually.

1b. Of all the visitors, bucket the ones who didn't create an account.
2b. Of those account creators under 30, show me those who liked dogs.
3b. etc..

This has proven very powerful for segmenting data, but it is very verbose and error prone to create all these 'difference' filters manually. What would be great is if I could optionally have the difference filter created for me whenever I create a filter.

For example, in the filter agg doc it suggests:

{
    "aggs" : {
        "in_stock_products" : {
            "filter" : { "range" : { "stock" : { "gt" : 0 } } },
            "aggs" : {
                "avg_price" : { "avg" : { "field" : "price" } }
            }
        }
    }
}

What if we allowed a parameter on the filter agg like:

"filter" : { "range" : { "stock" : { "gt" : 0 } }, includeDifference: true },

And that would automatically create a bucket that includes all the docs not included in the main filter, perhaps automatically naming it in this case 'not_in_stock_products'.

The response might then look like:

{
    ...

    "aggs" : {
        "in_stock_products" : {
            "doc_count" : 100,
            "avg_price" : { "value" : 56.3 }
        },
        "not_in_stock_products" : {
            "doc_count" : 50
        }
    }
}

creating further sub-aggs on the auto-created agg could be done either inline with the original agg:

            "filter" : { "range" : { "stock" : { "gt" : 0 } } ,
            includeDifference:true, differenceAggs: {
                "avg_price" : { "avg" : { "field" : "price" } }
            } 
            },
            "aggs" : {
                "avg_price" : { "avg" : { "field" : "price" } }
            }

or perhaps simply by specifying the name of the agg it would map it to the original by convention:

{
    "aggs" : {
        "in_stock_products" : {
            "filter" : { "range" : { "stock" : { "gt" : 0 } }  includeDifference:true},
            "aggs" : {
                "avg_price" : { "avg" : { "field" : "price" } }
            }
        },
        "not_in_stock_products" : {
            "avg_price" : { "avg" : { "field" : "price" } }
        }
    }
}

What do people think about something like this? I'm sure I'm not the only one who tries to segment their data like this.

[Kallin]

I'd be happy to contribute code for this, just wonder if the feature would be welcomed.

[clintongormley]

Hi @Kallin

Honestly, I don't like the includeDifference syntax. I'm wondering if this would be handled generically with the "other" option described in #5324

[Kallin]

This does sounds like it has similarities to the changes suggested in the 'missing' or 'other' options, though from what I gather those were specific to terms aggregations or bucketing aggregations. In this particular case I'm concerned about filter aggregations. If there's a way that it could be rolled into some other work that would be ideal. I don't see why the 'not_*' dynamic aggregation I described above couldn't be replaced with 'other', though that issue (#6804) looks to have been closed. Should I push for inclusion of this in #5324 ?

[clintongormley]

It does sound like the other bucket would be a good fit to solve this.

See #5324 for discussion

[Kallin]

as long as #5324 will handle '_other' bucket on filter aggs then I'm happy to close this.

[clintongormley]

@Kallin as I'm sure you've seen in the other thread, it's not an easy change, but we will be working on it :)

[Kallin]

Nothing worth having ever is :)