Consumer payload-based filtering

[bruth]

This feature may be out of scope since NATS is intentionally payload agnostic, however, related discussions have come up in Slack a handful of times so I wanted to start a discussion to document ideas.

What prompted me this time, was that I came across a recent Tim Bray post and Go library which implements JSON-based structural pattern matching. He had worked for AWS and help implement the EventBridge pattern matcher. This new library is a re-implementation of that and he boasts that it is quite fast (but is it NATS fast 😉).

Currently, a consumer supports filtering on a single subject (multiple subjects have been requested). For users using JSON-encoded payloads, one or more JSON patterns could be defined on the consumer for additional layer of server-side filtering. Here is an example from the post.

The big trade-off here, is that this introduces runtime cost and possible defects (bad JSON payload) although there may be ways to prevent bad data by checking for JSON validity when a published message is received. That said, this would be opt-in and the default fast path would remain the same. Only if the pattern matching configuration is specified would this additional runtime work need to be performed.

Another idea that had been discussed before was to use a wasm module, however that would require more effort (IMO) on both server-side integration and on developers at this time.

[derekcollison]

We have been thinking quite a bit on JetStream consumers and filtering, especially beyond just subjects. We have thought down the path of headers a bit, callouts, etc. And the WASM piece still flowing around for sure.

This looks pretty interesting, I think json might be one or two cool projects away from just winning 95+% of encodings.

I was concerned on this one around the pre-processing step though, but good t explore regardless.

Was also looking at these for ideas/inspiration as well.

https://github.com/SnellerInc/sneller
https://github.com/dflemstr/rq

/cc @aricart @tbeets

[bruth]

Thanks. Agreed regarding JSON, I have dabbled with MsgPack and Protobuf, but the ubiquity is hard to argue. The tooling is pretty fragmented which causes various quality/coverage issues (say with JSON Schema).. so I agree a couple projects to solidify the ecosystem of tooling. I have been keeping an eye on cue to see if it could be the winner of the schema definition and validation world.. but we will see.

Was also looking at these for ideas/inspiration as well.

Nice, wasn't aware of sneller.

[gedw99]

agree also.
I have used protobuf, message pack, etc also. I like the lightness of Cue and its power to also have logic / validation or not.

I like the idea of quick filtering..

#3266 is a discussion i just made, and explains how we can use Cue with JSON and JSON Schema and get evolvable schema. Grafana is already using it. Its called Thema and not going away..
I have been using Cue with NATS, and when i found Thema it made my day.

We can take the existing JSON Schema we have today and gen it into Cue with evolvable hooks ( called lineage in Thema).
Or just take your open API and turn into cue with linages, and then gen the openapi back out

[goireu]

I do believe WASM would be a brilliant idea, not only for header and payload processing but also maybe for extending NATS in many ways (fine grained permissions, custom logic, ...).
I recall Microsoft introducing a way to run custom .NET code in their SQL Server 15 years ago (in addition to PL/SQL), it was a really nice piece of innovation.

Of course this comes with many challenges and probably is a mid/long-term move.
On the other hand it probably is the way to go to remain payload-agnostic and provides decent performance.

One could imagine a generic JSON validator implemented as a WASM module for example.

My 2 cents 😄

[bruth]

@goireu I don't disagree with wasm being the future.. I think it is still early days on the tooling side and wasi, for example. So I guess the question is whether a short-term solution is introduced that is more accessible/simpler (for now) or we wait until there is more maturity. They aren't mutually exclusive and I could see an embedded solution (JSON validator/filter) be converted to a wasm-based solution later transparently.

[goireu]

@bruth 100% agreed on tooling/WASI maturity and short term possibilities.
My point was to broaden the field WASM modules could possibly be used to enhance NATS. Sorry if I didn't make it clear in the first place :)

[bruth]

For sure, no worries, we are in agreement! Wasm is exciting and there is a lot of potential.

[gedw99]

Regarding WASI and WASM i have been using wazero with NATS. Its a great marriage so far.

https://github.com/tetratelabs/wazero

NO CGO too !!! LIfe is too short for CGO ..

[lgothard]

Any update on extending NATS with WASM UDFs (user-defined function)?

[schmurfy]

I am evaluating nats as a potential replacement for google pubsub and I was a bit disappointed to see how filtering works in nats.

in pubsub you have your message with metadata attached to it (in nats that would be Headers I think), and you can set filters on these metadata and only them which works well. I was looking at something similar in nats but couldn't find it.

What is the current state of filtering ? is there anything available right now beside subject based filtering ?

[derekcollison]

At this time you are limited to subject based filtering.

[Jarema]

However keep in mind that NATS subject filtering is pretty powerful, with possible token filtering with wildcards etc.

[schmurfy]

I agree subject filtering in nats is already really powerful, I am just investigating what is th best approach to translate what we have with pubsub.

[tpihl]

I see benefits of more server side filtering that cannot be solved in the subject. Same subject is the core concept for many parts of jetstream (like rollup and max msgs per subject and last msg per subject and so on).

There are however many situations where only some of those messages are relevant for a specific consumer (in actionable events world).

In the end, the filtering needed would depend on the content of Data. One way to make it more light-weight and avoiding unpacking Data in the server processing would be to use headers and some kind of tagging. So subscription on subjects(!) (with or without wildcards) in combination with server-side evaluation of headers would be beneficial in my cases, specially when the latency between server and client increases or the bandwidth decreases (we do some sat com, low bandwidth and high latency). Currently, we try to solve it with different subjects and looses all optimizations like rollup or "Nats-Expected-Last-Subject-Sequence" that is very useful in sat-com.

The solution to first create a header-only consumer and the request the msgs we need creates a lot more roundtrips, and consuming all irrelevant msgs chokes limitated bandwidths as well as drives cost.

A first implementation where we can register a callback when we run the server embedded would help a lot. Or at least that would make cost/benefit/impact much clearer and maybe it will becode easier to reason about

[derekcollison]

We are considering header filtering. And this can be a bit more detailed but is similar to hoisting context into subjects via additional tokens.

[tpihl]

Not sure I understand the last part, how do you do to use max-msgs-per-subject with discard new to protect initial snapshots to not drop off, while also filter on wildcard tokens?

Yeah, basically asking for more details when time permits

[tpihl]

Any progress in this part of jetstream?

[derekcollison]

This will most likely follow from generic callouts that would instruct the server to callout to a co-located micro-service that the user/customer is in complete control of to provide rich filtering semantics.

We may build in header filtering as mentioned above, but for full payload semantics / filtering that will be using this feature. This will also allow changing the message itself based on consumer destination, bnot just binary allow/deny.

[sdktr]

Any hints on where to read up on the 'generic callout' ideas?

[derekcollison]

Nothing public yet, but imagine a subject mapping of sorts similar to subject mappings, but this one will use the secondary subject to send the message to and await a response. The response could be 1) send it through on original subject. 2) Send modified version. 3) Deny sending.

The entries will have settings for how long to wait, and whether or not these have to be processed serially. These would only happen on first entrance to a NATS system via client or leafnode connection, or via a service import to another account etc.