route_to_stream pipeline function silently dropping messages

[waab76]

Expected Behavior

When route_to_stream is called for a paused stream and the remove_from_default argument is set, an exception should be raised to indicate that processing has failed.

Current Behavior

When route_to_stream is called for a paused stream and the remove_from_default argument is set, the message will not be added to the target stream and it will be removed from the default stream, resulting in the message being silently lost.

Possible Solution

Update route_to_stream logic to raise an error instead of silently dropping messages from all streams.

[kingzacko1]

Feedback from Arch:

• Due to the possibility of high rates of messages coming through, logging an error each time this happens is likely not the best approach even with rate limited logging in place
• If the server has Failure Processing enabled then we should log these messages to the Graylog Message Failures index (if possible). These are considered system messages and not counted against licensed ingest limits.
• We should have a more clear disclaimer when pausing a stream. Right now the disclaimer is just a generic Are you sure? which doesn't let users know of the potential fallout from this action
• We should consider renaming the Pause action as it may imply that messages that should have been routed to this stream while it is in this state will be routed there once the stream is restarted. That is not the case. The stream is Stopped, will not receive any messages, and if there are messages that match the stream rules with Remove from default stream set then anything that should have made it into the stream while stopped is completely gone.
• With the previous point, we should clearly document what this state means for streams in our documentation.

[boosty]

Thanks @kingzacko1 for the summary!

I am wondering: If 3 + 4 get done (which are cheap to build), which will present a very clear warning that messages will be dropped if you pause the one and only target stream for a message, then maybe 2 is not that urgent to implement.

If the server has Failure Processing enabled then we should log these messages to the Graylog Message Failures index (if possible). These are considered system messages and not counted against licensed ingest limits.

Just a thought: If this gets implemented, maybe it should get its own config option ("Log Stream Routing Failures"). This would allow people to turn it off, and also document the fact that this behavior exists.

/cc @Graylog2/architecture

[tellistone]

IMO Streams having an "off" state that blocks messages routed from pipelines seems like a trap for users without a benefit.

I can understand the on/off controlling stream rules (the point of entry to pipeline flow), but impacting pipeline routing as it does means pressing this button has an impact the user can't anticipate without looking at every other active pipeline.

If we altered the on/off state to not impact pipeline routing (and changed the name of the column from "status" to "Stream Rules"), then we wouldn't need this exception handling.