Jetstream durable queue-group subscription based MaxAckPending

[ccapuano-ssc]

Problem

Working on a recent project where we are moving from NATS streaming to JetStream. We have, and would like to maintain, async push subscriptions to a Jetstream Consumer created as a durable queue group. The issue we are facing is when additional subscriptions are created, the MaxAckPending property of the Consumer does not change based on the number of subscribers.

Proposed Solution

Our requirement would be to allow a subscription to define a MaxAckPending rather than on the consumer. This would allow a level of elastic scalability with push subscriptions we don't seem to be able to achieve in the current version (2.6.6).

In practice, we would like to have a additional subscriptions to the consumer take up to a configured amount of messages. In an example, if we specified a MaxSubscriptionAckPending of 5, and subscribed 3 times we should have a Consumer MaxAckPending of 15 with each subscription not receiving more than 5 messages at a time.

[derekcollison]

I might suggest that in JetStream this use case is better suited to pull consumers. They scale out by design and you can supply a batch size they are willing to take. In addition you can create a consumer with a MaxBatch to control how many messages any number of subscribers could ask for, say 5 as in above.

[ccapuano-ssc]

Thanks for the comeback Derek. I did see that the pull subscriptions may provide some level of what we are looking for, but from what I can tell its in batches with timeouts to achieve it, correct? In our use case, we are looking to respond to messages as they arrive in more of a push situation. I assumed that the NATs server/Consumer had all the info it needed to honor pushing only a configured number of messages to a given subscription as it seems its just a blend of info it already has (queue group and MaxAckPending).

I will look at the pull sub again, but think there is still value in allowing a subscription to define a MaxAckPending as it would resolve the gap in our current use case cleanly. Not sure how many others have a similar one.

Thanks

[derekcollison]

2.7.0 introduced the ability to do long polls, and 2.7.1, coming next week will solidify that such that push can be completely implemented via pull under the covers.

[zbindenren]

Is there an example on how to do long polls?_

Like this?

s.NextMsgWithContext(context.Background())

[derekcollison]

Let us know what you are trying to accomplish. Is this a work queue type system where you want to be able to scale the number of apps pulling requests etc? Or something else?

[zbindenren]

Yes exactly a work queue type system where I want to be able to scale the number of apps pulling requests.

[derekcollison]

Once you have a JetStream context you can create or bind an existing pull consumer as follows.

	sub, err := js.PullSubscribe("foo", "bar") // foo is subject, bar is durable name.

Then you can do..

msgs, err := sub.Fetch(1, nats.MaxWait(60*time.Minute))

That tells the system (server 2.7.2 or greater IIRC) to return 1 message waiting a maximum of 1hr..

[zbindenren]

thx