Intake v2: investigate parallelizing decode and validate

[roncohen]

As previously discussed, it would be interesting to investigate parallelizing the decoding and validation of documents. Transformation is already being done in parallel.

At the moment, decoding and validation is happens in parallel at the request level. E.g. incoming data from separate requests are being decoded and validated in parallel. As part of this issue, we should also seek to put a bound on that parallelization

[alvarolobato]

We agreed that it's a bit premature to implement this know, not even needed probably. Deferred from 6.5

[graphaelli]

@axw do the changes you've proposed and implemented make this obsolete?

[axw]

Nope.

#3551 will weave validation and decoding together such that parallelising them wouldn't make sense.

It might make sense to decode concurrently, e.g. have the stream processor send events to a channel to be decoded, and then have multiple goroutines processing those. This would enable parallel decoding within a single stream.

[stuartnelson3]

Closing this for now as we aren't prioritizing time to investigate whether it's worth implementing this