Streaming data

[ghost]

Please describe what you would like to see in this project

We need a way to stream data between steps instead of providing them in a bunch.

Please describe your use case

Some steps provide a lot of data, or provide data over a longer period of time. For example,a Prometheus scraper would do this. To guarantee safe storage and efficiency, it should be possible to stream data between two steps that support it, or stream data to a loop that processes the data.

Additional context

The implementation should consider batching with time-window to make data transfers efficient.

[dustinblack]

Superseded by https://github.com/arcalot/arcalot-round-table/blob/signals/art-decisions/proposals/2022-11-20-arcaflow-signals.md

[ghost]

Tracked as #44