snuba-subscription-consumer-events doesn't process events

[gromnsk]

Environment

snuba-

Steps to Reproduce

not sure how to reproduce it, but I increased partitions in ingest-events topic.
I had a problem with KeyError which was fixed by adding this in config:
TOPIC_PARTITION_COUNTS = { "ingest-events": 20, "events": 20, // didn't increase partitions in this topic, but see that it scaled to ingest-events partition amount }

and now it's not crashing, I tried to enable debug logs, but don't see any activity in service and according on Kafka Lag it doesn't do anything, I have lag on snuba-commit-log topic and also on events in snuba-events-subscriptions-consumers group. Right now I have no any errors or so, but events topic doesn't have consumers and snuba-commit-log does, but lag is increasing on both topics.

Expected Result

Have no any lags on topics and events processing

Actual Result

Have lags on events and snuba-commit-log topics in snuba-events-subscriptions-consumers group

[kiper-prog]

Same problem

[untitaker]

can you be more precise? you are mentioning three different topics in one bugreport, and one of them (ingest-events) is a completely separate system from the rest of the two (events and snuba-commit-log). Did you encounter this issue as a direct result of scaling up ingest-events?

[gromnsk]

not sure that it was a direct result of scaling up ingest-events, but it could be so. At this moment snuba-subscription-consumer-events doesn't connect to it's topic and doesn't process any messages there. Debug logs doesn't show anything, according on Kafka there is no consumers for this topic.
That's my problem, all other stuff I thought could be helpful to define the problem. I have no any additional information about it because have no any logs, looks like it's just not consuming messages and that's it

[untitaker]

how many partitions does each topic have? I remember some issues where commit-log had a too high partition count, but the issue manifested differently: #5855

[gromnsk]

snuba-commit-log topic for consumer group snuba-events-subscriptions-consumers has 1 partition, events topic has 20 partitions

[onewland]

Are you saying that snuba is not consuming any events? Or are you saying that the commit log topic isn't being consumed? Or both?

If snuba is not consuming events, then no messages will be published to the commit log. The commit log is a topic that records progress of the snuba event consumer for alerting purposes. Additionally, the subscription scheduler consumer (which consumes the commit log) waits for a message on every partition in the parent topic (snuba events) before advancing. So if your volume is low and you aren't receiving errors on all 20 partitions, then that consumer will never move forward.

[gromnsk]

@onewland I have lags on topics above, it means that there is data, but no one process this messages and snuba-subscription-consumer-events is working(no any errors), but doesn't process messages and topic doesn't have consumer-group connected to this topic

[onewland]

@gromnsk

Errors (which use the events topic) are partitioned by project_id. The subscriptions system which powers alerting will only advance the observed system time if messages are received on all partitions with sufficient timestamps (one of the design goals of the system is to be able to distinguish between data not yet ingested and empty time regions). My guess is that you don't have incoming traffic on all topics, and therefore the subscription system will never schedule subscriptions (consuming the commit log topic) because it cannot with confidence determine that it is "caught up" on all ingestion.

To test this hypothesis, try running this command on your Kafka broker:

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group errors_group

replacing errors_group with whatever the appropriate consumer group name is. If LOG-END-OFFSET is set to 0 on any partition, then the subscription scheduler won't be able to make progress until that partition receives traffic and therefore the "ingestion clock" can advance.

I think you probably only want one or two partitions on errors and snuba-events for this to function properly (unless you have a lot of projects that regularly receive error events).

[onkar]

Closing this issue based on the @onewland's comment above. It does not look like a product issue.