Complex filters

[mourner]

Example from v0 style for native:

"source": "outdoors",
"layer": "poi_label",
"filter": [
  "and",
  { "field": "scalerank", "value": 4 },
  { "field": "localrank", "operator": "<=", "value": 16 },
  { "field": "localrank", "operator": ">", "value": 4 }
]

For the new style format, I suggest implementing the following syntax:

"filter": { // "and" at the top level
  "source": "outdoors",
  "layer": "poi_label",
  "scalerank": 4,
  ">": {"localrank": 4},
  "<=": {"localrank": 16}
}

Nested filters would look like this:

"filter": {
  "$or": {
    "maki": "park",
    "$and": {
        "foo": "bar",
        "$not": {"maki": "airport"}
    },
    "$match": {"maki": ".*"}
  },
  "bar": 5
}

Or with &&, ||, !, =~ instead of $and, $or, $not, $match.

It looks pretty clear, easy to read, and very compact for most cases. Consider that we're optimizing for common filters that are usually used in a style, not super-complex tree-like filters, because they are uncommon (but should still be possible).

@ansis @kkaefer @edenh @nickidlugash @incanus thoughts? Related discussions: #356

[ansis]

Looks good! I'm seeing a couple possible issues

or

With the suggested $or: { ... } you can't do an or on multiple values for the same key. For example, checking if class is either motorway or main. These are very common in current styles.

Instead we could do:

{
    "$or": [
        { "class": "motorway" },
        { "class": "motorway_link" } ] }

and possibly have a $in operator to make this shorter

{
    "class": { "$in": [ "motorway", "motorway_link" ] }
}

magic keywords

The magic keywords source, layer and type aren't clearly special, and reserving them make it impossible for these words to be used as feature keys. type is currently used as a key in mapbox streets, and seems like something that would also be commonly used in custom data

We could prefix them: $source, $layer, $type. I'm not a huge fan of having $$$$$ everywhere, but it does nicely distinguish them and I can't think of anything better.

---

The suggested filter format, especially with these changes, is very similar to mongodb's comparison and logical operators. Should we follow @tmcw's suggestion and just full adopt those?

guide and list of comparison and logical operators

The remaining differences are mostly renames.

[mourner]

@ansis it would be an extension of the current syntax, so one field with multiple values is specified simply like this:

"class": ["motorway", "motorway_link"]

So if value is an array, it's an "or" relation.

We could fully adopt the Mongo syntax, but it wouldn't look as compact I think, and we would be more limited in how we can customize it specifically for GL styles.

[mourner]

We could also add some more syntactic sugar and allow multiple operators per field like this:

"localrank": {">": 4, "<=": 16}

[kkaefer]

"filter": { // "and" at the top level
  "source": "outdoors",
  "layer": "poi_label",
  "scalerank": 4,
  ">": {"localrank": 4},
  "<=": {"localrank": 16}
}

is problematic because it'd make it impossible to select the attributes source and layers because they are, magic words; agreeing with @ansis here.

I think we should go with MongoDB's query syntax; it looks pretty concise. Instead of having the source and layer properties in the filter object, we should just revert to having them at the top level object. What was the reason to move them to the filter object?

A modified MongoDB syntax would look like this:

{
    "scalerank": 4,
    "localrank": { "<=": 16, ">": 4 }
}

and the original MongoDB syntax is

{
    "scalerank": 4,
    "localrank": { "$lte": 16, "$gt": 4 }
}

[mourner]

I agree we can go with MongoDB-style, but with some more differences. E.g. I'm convinced we need to keep the current "field": ["foo", "bar"] logic (OR) instead of having MongoDB-like exact matching of an array.

Instead of having the source and layer properties in the filter object, we should just revert to having them at the top level object. What was the reason to move them to the filter object?

Because it is a part of a filter semantically. Having it at the top of the layer object is confusing for users.

[tmcw]

E.g. I'm convinced we need to keep the current "field": ["foo", "bar"] logic (OR) instead of having MongoDB-like exact matching of an array.

I think what you're looking for is the $in query

// currently
{ "field": ["foo", "bar"] }
// mongo
{ "field": { "$in": ["foo", "bar"] } }

[kkaefer]

I think @mourner is proposing making $in the default and instead using something like $and to designate the mongodb semantic.

[mourner]

Yes. I think we don't need exact array matching at all (although we can have a special operator for it), just checking that a field contains each of the multiple fields, e.g.:

{"food": {"$has": ["restaurant", "chinese"]}}

[mourner]

In #430

[mourner]

Instead of having the source and layer properties in the filter object, we should just revert to having them at the top level object. What was the reason to move them to the filter object?

Forgot to mention one more reason: filter properties are inherited, so you can make styles more compact by combining layers with shared source, layer or feature_type into groups and specifying the property once for the whole group.

[kkaefer]

Wait, we support nested sources that inherit from their parents? Is there an example for this?

[mourner]

No example currently since migration scripts don't group layers — you have to do that manually if you want, but basically, lets say we have:

{
  "id": "landcover_snow",
  "filter": {"source": "outdoors", "layer": "landcover", "class": "snow"}
}, {
  "id": "landcover_crop",
  "filter": {"source": "outdoors", "layer": "landcover", "class": "crop"}
}, {
  "id": "landcover_grass",
  "filter": {"source": "outdoors", "layer": "landcover", "class": "grass"}
}, {
  "id": "landcover_scrub",
  "filter": {"source": "outdoors", "layer": "landcover", "class": "scrub"}
}, {
  "id": "landcover_wood",
  "filter": {"source": "outdoors", "layer": "landcover", "class": "wood"}
}

We can group them like this:

{
  "id": "landcover",
  "filter": {"source": "outdoors", "layer": "landcover"},
  "layers": [{
    "id": "landcover_snow",
    "filter": {"class": "snow"}
  }, {
    "id": "landcover_crop",
    "filter": {"class": "crop"}
  }, {
    "id": "landcover_grass",
    "filter": {"class": "grass"}
  }, {
    "id": "landcover_scrub",
    "filter": {"class": "scrub"}
  }, {
    "id": "landcover_wood",
    "filter": {"class": "wood"}
  }]
}

When preprocessing, the group filter gets propagated to children layers.

[kkaefer]

How are we going to represent nested layers then? As I understand it, all of the landcover_* layers are still at the top level?

[mourner]

Whenever styles indicate that the group will be composited, e.g. with opacity, prerender, comp-ops etc. Do you see any issues with this approach?