Add streaming option

[rmarx]

Current thinking is to allow http://ndjson.org and to log the qlog header information as a separate object on top.
This would remove the possibility to log multiple traces in the same qlog file if it uses the streaming option, so users would need to use group_id in those cases.

Would also require the addition of a "format" field to the qlog metadata header (though we probably need that anyway if we want to keep support for other JSON-based optimizations like event_fields.

We first need to check how this plays with oboe.js in qvis though, to see if we need to add another parser in the mix.

See also #106 and #2 for more discussion.

For comparison:

Draft-01 would look like this (if, like me, JSON serializers would be too lazy to add quotes):
{
    version: draft-01,
    traces: [
        {
             vantage_point: client,
             common_fields: [ ... ],
             event_fields: [ ... ], 
             events: [
                 [ 55, transport, packet_sent, { ... } ],
                 [ 66, transport, packet_received, { ... } ],
                 ...
             ]
        }
    ]
}

Draft-02 would be more like:
{
    version: draft-02,
    format: "ndjson",
    trace: {
             vantage_point: client,
             common_fields: [ ... ]
     }
}
{ time: 55, name: "transport:packet_sent", data: { ... } }
{ time: 66, name: "transport:packet_received", data: { ... } }
...

CC @marten-seemann @huitema

[marten-seemann]

I like this idea. In the interest of keeping parsers simple, I would be supportive of completely replacing the current qlog format with this.
As I've said elsewhere, I don't think that the complexity that comes with putting multiple traces into the same file justifies the complexity.

[nibanks]

I agree with @marten-seemann. We should optimize for the primary case, which is definitely single machine/trace.

[huitema]

Streaming would definitely be useful for me too. Currently, I have to use a two stage approach to solve that: stream-supporting binary log, then convert to qlog once I am sure the end is reached. And yes, all usage is single machine/trace.

[LPardue]

support this, would be good to see a short example of the difference between new and old formats so I can start planning

[rmarx]

I've updated the issue with an example, PTAL.

I'm highly skeptical that that type of thing would be de-serializable automatically with something like @LPardue's serde, but a girl can dream.

[LPardue]

Thanks. FWIW, the current qlog format is effectively not deserializable using serde either, so nothing would be lost.

edit: hmm, actually the new format might make it more straightforward to use a semi-automated deserializer

[rmarx]

Initial support for "newline delimited JSON" qlog streaming is now in qvis per quiclog/qvis@8504c0e

An example draft-02 NDJSON file can be found in attachment (should load just fine in the live qvis atm).
Now that I'm certain this can be done without major performance issues, I will move forward with this for qlog draft-02.

In the meantime, further feedback based on the example file is of course welcome.

newline_delimited.zip

[rmarx]

This issues should have been resolved by the choice of NDJSON as streaming format as of f5db7cd