how to recover from DiscardNew policy?

[lukasGemela]

hey guys, have a question related to jetstream's discard policy new
The Discard Policy sets how messages are discarded when limits set by LimitsPolicy are reached. The DiscardOld option removes old messages making space for new, while DiscardNew refuses any new messages.
This to me somehow indicates that new messages wont be accepted until some old messages are ACKed and deleted from stream.

But I tested this scenario and apparently that's not how it works:
I have stream with 3 maxMsgs
I publish 3 messages
I consume and ACK 3 messages
I publish one more message, got exception that stream is full
I never consume more messages and messages are not cleared out from stream.

Is this expected behaviour?

I'm trying to archieve behaviour that our producer starts to fail when stream is full (return error to client or whatever), but can recover from failure when consumers catch up later. If I keep discard policy to old and consumers starts to fail for some reason we might start to loose messages

[derekcollison]

Streams have a separate state from consumers, unless the stream is in interest retention mode which then means when all consumers have ack'd a message it will be removed from the stream.

That is probably what you want.

[lukasGemela]

hi, yes that's what I want, I have two consumer groups reading different subjects from one stream and want to delete the message once they both ACK. REading the doc

In InterestPolicy messages will be removed as soon as there are no more Consumers.

that's not very explanatory, was thinking that I need to unsubscribe consumer to get the message deleted, but I guess it actually means when all interested consumers ACK the message then :) will test that, thanks!

[derekcollison]

Will see if we can clean up the docs a bit here.

[jnmoyne]

Docs improved in this commit https://github.com/jnmoyne/nats.docs/commit/dfdc6236745bf455062477e510bf8924717a5056 (which will be included in the next PR of my docs fork back to the nats-io docs repo