KeyError: 2 in snuba-subscription-consumer-metrics

[bkk-bcd]

Environment

Sentry self-hosted on Kubernetes

snuba:24.2.0

Steps to Reproduce

  extraProvisioningCommands:
   - /opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server ${KAFKA_SERVICE} --topic outcomes --alter --partitions 4
   - /opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server ${KAFKA_SERVICE} --topic snuba-metrics --alter --partitions 4
   - /opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server ${KAFKA_SERVICE} --topic ingest-events --alter --partitions 3
   - /opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server ${KAFKA_SERVICE} --topic ingest-attachments --alter --partitions 2
   - /opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server ${KAFKA_SERVICE} --topic ingest-transactions --alter --partitions 3

Expected Result

All services should be in ready state.

Actual Result

snuba-subscription-consumer-metrics in crash loop backoff:

2024-05-03 12:07:22,859 Initializing Snuba...
2024-05-03 12:07:25,945 Snuba initialization took 3.0845345419365913s
{"module": "builtins", "event": "Checking Clickhouse connections", "severity": "info", "timestamp": "2024-05-03T12:07:25.958708Z"}
2024-05-03 12:07:28,975 New partitions assigned: {Partition(topic=Topic(name='snuba-metrics-commit-log'), index=0): 0}
2024-05-03 12:07:29,484 Caught exception, shutting down...
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/arroyo/processing/processor.py", line 321, in run
    self._run_once()
  File "/usr/local/lib/python3.11/site-packages/arroyo/processing/processor.py", line 409, in _run_once
    self.__processing_strategy.submit(message)
  File "/usr/src/snuba/snuba/subscriptions/scheduler_processing_strategy.py", line 240, in submit
    self.__next_step.submit(message)
  File "/usr/src/snuba/snuba/subscriptions/combined_scheduler_executor.py", line 275, in submit
    tasks.extend([task for task in entity_scheduler[tick.partition].find(tick)])
                                   ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 2
2024-05-03 12:07:29,490 Closing <snuba.subscriptions.scheduler_consumer.CommitLogTickConsumer object at 0x7f1523546b90>...
2024-05-03 12:07:29,491 Partitions to revoke: [Partition(topic=Topic(name='snuba-metrics-commit-log'), index=0)]
2024-05-03 12:07:29,491 Partition revocation complete.
2024-05-03 12:07:29,494 Processor terminated
Traceback (most recent call last):
  File "/usr/local/bin/snuba", line 33, in <module>
    sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/snuba/snuba/cli/subscriptions_scheduler_executor.py", line 153, in subscriptions_scheduler_executor
    processor.run()
  File "/usr/local/lib/python3.11/site-packages/arroyo/processing/processor.py", line 321, in run
    self._run_once()
  File "/usr/local/lib/python3.11/site-packages/arroyo/processing/processor.py", line 409, in _run_once
    self.__processing_strategy.submit(message)
  File "/usr/src/snuba/snuba/subscriptions/scheduler_processing_strategy.py", line 240, in submit
    self.__next_step.submit(message)
  File "/usr/src/snuba/snuba/subscriptions/combined_scheduler_executor.py", line 275, in submit
    tasks.extend([task for task in entity_scheduler[tick.partition].find(tick)])
                                   ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 2
Stream closed EOF for sentry/sentry-snuba-subscription-consumer-metrics-865598d55d-lg9pt (sentry-snuba)

[bkk-bcd]

🙏

[untitaker]

From #5514 (comment)

My snuba-commit-log topic only has one partition

If you encounter this for the metrics subscription executor, you need to take a look at that corresponding commit log: snuba-metrics-commit-log

I believe that topic has more than 1 partition

Generally, if you run scheduler-executor-type consumers, all topics ending with -commit-log should have 1 partition and only 1 consumer replica.

If you want to horizontally scale that topic, the scheduler-executor consumer needs to be split up into two consumer types scheduler and executor, see also #5514 (comment)

[bkk-bcd]

From #5514 (comment)

My snuba-commit-log topic only has one partition

If you encounter this for the metrics subscription executor, you need to take a look at that corresponding commit log: snuba-metrics-commit-log

I believe that topic has more than 1 partition

Generally, if you run scheduler-executor-type consumers, all topics ending with -commit-log should have 1 partition and only 1 consumer replica.

If you want to horizontally scale that topic, the scheduler-executor consumer needs to be split up into two consumer types scheduler and executor, see also #5514 (comment)

Thanks, will parse this out and take a look!

[getsantry]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[Funghorn]

@untitaker speaking on behalf of @bkk-bcd.

If you encounter this for the metrics subscription executor, you need to take a look at that corresponding commit log: snuba-metrics-commit-log
I believe that topic has more than 1 partition

Looks like this topic has just one partition:

Topic: snuba-metrics-commit-log	TopicId: 0lNibaLGS7aXGj1lDqhxog	PartitionCount: 1	ReplicationFactor: 1	Configs: cleanup.policy=compact,delete,min.compaction.lag.ms=3600000,retention.bytes=21474836480
	Topic: snuba-metrics-commit-log	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

Could you suggest the next steps to debug this, please?