Engine for streaming queries

[kenwenzel]

Hello,

is it possible to use the evaluation strategy for processing of (virtual) RDF streams like
https://github.com/joshsh/stream42

We integrate time-series data from multiple sources and it would be really cool to run streaming queries on the data.

Thank you and best regards
Ken

[pulquero]

Yes, you could add support for a custom SERVICE url, eg.
SELECT * {
SERVICE stream://1 { ?s ?p ?o}
SERVICE stream://2 {?s ?u ?v}
}
and have an implementation of BindingSetCallbackFederatedService that does the stream reading (it would only support a single StatementPattern). Then you could use BindingSetCallbackSailConnection to stream it out.

[kenwenzel]

OK, that sounds interesting. Is it also possible to register hundreds of queries in parallel without blocking a thread for each query?

[pulquero]

Yes, it is effectively event-based. So data is "pushed" up the query tree by the threads reading the data. It needs to block if you use the conventional RDF4J "iterator"-based API, but if you stick to the BindingSetCallback interfaces, it should push all the way through.