Hi @nazarhussain. To be honest, this isn't well supported, but I think it should be possible for simple cases using thenStream. Does that work for you?

If not, can you explain more about what you're trying to do? I'm definitely interested in better supporting this in future, and it'd be useful to understand the use cases that are important.

Yes, I tried thenStream but could not help out well. Points are highlighted below:

1. The thenStream requires sending a pre-initialized stream
2. Unless we don't know exact query parameters we can't initialize a relevant stream

The use-case I have is the following:

1. I have my implementation of event-stream that I want to reuse
2. That implementation requires a handler to be passed for events of interest categorized by topics
3. When a particular event happens, my implementation checks if there is a handler for that topic or not and calls it.

We are using fastify for actual implementation and I wanted to mock to test some complex scenarios. Here you can find the reference to real handling of those event streams.

https://github.com/ChainSafe/lodestar/blob/384e3931c939b04353964524b52969e4f1a4f307/packages/api/src/beacon/server/events.ts#L15-L58

Ah, I see, that makes sense. Yes, I think that's not really supported right now. You could try creating many rules each matching different sets of parameters and returning different streams, but that won't work for many cases (e.g. completely unknown freeform query params) and it's quite awkward.

In future, I think this would probably be best supported by allowing it within thenCallback, with something like:

server.forGet('/')
  .thenCallback((req) => {
    return {
      statusCode: 200,
      body: buildCustomStream(req)
    };
  );

Doing this perfectly is a little harder than it sounds in some places - for example in the general case, the stream needs to be automatically encoded, whilst it's streaming, using the appropriate compression algorithm (e.g. gzip). I think we can avoid that initially, but we'd need to do so explicitly...

How about we extend the result type from callbacks, so that the rawBody field can be returned with either a Buffer, or a Buffer stream? rawBody is never encoded - if you use it, you're responsible for handling encoding yourself - but I think that's probably fine for your case. We could then add stream support with automatic encoding in the body parameter later on, if necessary, with no breaking changes.

I'm open to something like this. I'm unlikely to have much time in the short-term to look into it myself, but PRs are very welcome.

How about we have thenCallback support to return stream and where users will be responsible for writing to stream the way they wanted.

How about we have thenCallback support to return stream and where users will be responsible for writing to stream the way they wanted.

I'm not sure exactly what you mean, can you show an example?

Currently the thenCallback supports different options to return, e.g. json, body. What if we can return stream from there? We can prepare the stream within thenCallback and return like return { stream: myStream }.

Currently the thenCallback supports different options to return, e.g. json, body. What if we can return stream from there? We can prepare the stream within thenCallback and return like return { stream: myStream }.

Yes, exactly. I think maybe you've misunderstood my example? That's effectively the same as what I'm proposing in the example above.

The only difference is that in that case, we return the stream as another type of value for the existing body field, instead of a adding a new separate stream field. Does that make sense? It's useful that way because it's never meaningful to return both at the same time, and returning a stream there is unambiguous (because it's currently not a valid value for that field).

It's also better to start with just rawBody, not body, to make it clear that the stream data will not be automatically encoded (we can do that later, but it's not needed immediately, and it's not easy). We can do both later though, to support both pre-encoded & unencoded body streams.

Anyway, yust to make it clearer, I'm proposing that this would work:

server.forGet('/')
  .thenCallback((req) => {
    // Inside the callback, use 'req' to build a readable stream
    // for the body data:
    const yourStream = generateAStream(req);

    return {
        statusCode: 200,
        // If you return a stream as the body here, then the HTTP
        // response body will stream from this, and continue until
        // your stream ends:
        rawBody: yourStream
    };
  );

Is that clearer?

Hi @pimterry! I took a stab at this and tried to change CallbackResponseMessageResult.rawBody to Buffer | Uint8Array | Readable and I soon realized that this also affects the beforeRequest and beforeResponse paths, because buildOverriddenBody was written to expect a complete rawBody and not something that is still streaming.

Would having a stream be useful at all in beforeRequest and beforeResponse? I'm wondering if I could just add something like CallbackResponseMessageResult.bodyStream that's only used by writeResponseFromCallback. I was able to get that working rather quickly and have tests passing, but would it make sense to add a new property just for this? Do you see a better way to do this?

EDIT: Or, maybe read the whole stream into a buffer in buildOverriddenBody?

Opened a PR for this feature. I'd love to get some feedback on it. Thanks!