Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Batching failures when individual event errors bubble up #408

Merged
merged 4 commits into from
Apr 1, 2024

Conversation

bmtcril
Copy link
Contributor

@bmtcril bmtcril commented Mar 27, 2024

When errors such as when a backend isn't configured or there is no transformer for an event, bubbles up it causes the whole batch to get dumped to the dead letter queue. These changes cause those errors to be logged but otherwise just get dropped from the batch.

Error:

[2024-03-27 14:58:10,291: INFO/ForkPoolWorker-23] Event edx.course.enrollment.deactivated has been queued for batching. Queue size: 10
[2024-03-27 14:58:10,291: INFO/MainProcess] Task event_routing_backends.tasks.dispatch_bulk_events[c6ff920d-3d73-410d-ab73-bd41cfae0ec1] received
[2024-03-27 14:58:10,292: ERROR/ForkPoolWorker-23] Exception occurred while trying to bulk dispatch 10 events.
Traceback (most recent call last):
  File "/openedx/venv/lib/python3.8/site-packages/event_routing_backends/backends/events_router.py", line 187, in send
    self.bulk_send([json.loads(queued_event.decode('utf-8')) for queued_event in batch])
  File "/openedx/venv/lib/python3.8/site-packages/event_routing_backends/backends/events_router.py", line 153, in bulk_send
    event_routes = self.prepare_to_send(events)
  File "/openedx/venv/lib/python3.8/site-packages/event_routing_backends/backends/events_router.py", line 93, in prepare_to_send
    processed_events = self.process_event(event)
  File "/openedx/venv/lib/python3.8/site-packages/event_routing_backends/backends/events_router.py", line 256, in process_event
    events = processor(events)
  File "/openedx/venv/lib/python3.8/site-packages/event_routing_backends/processors/mixins/base_transformer_processor.py", line 37, in __call__
    transformed_event = self.transform_event(event)
  File "/openedx/venv/lib/python3.8/site-packages/event_routing_backends/processors/caliper/transformer_processor.py", line 43, in transform_event
    raise NoBackendEnabled
eventtracking.processors.exceptions.NoBackendEnabled
[2024-03-27 14:58:10,292: INFO/ForkPoolWorker-23] Pushing failed events to the dead queue: dead_queue_caliper

Comment on lines -178 to -180
if not required:
logger.warning('Could not get value for %s in event "%s"', key, self.event.get('name', None))
else:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great change here

@bmtcril bmtcril force-pushed the bmtcril/batching_fixes branch from 42dc174 to 57ae7bd Compare April 1, 2024 16:54
When errors bubble up, such as when a backend isn't configured, it causes the whole batch to get dumped to the dead letter queue. Now it just returns empty arrays and logs the errors.
@bmtcril bmtcril force-pushed the bmtcril/batching_fixes branch from 57ae7bd to fb71b0c Compare April 1, 2024 17:34
@bmtcril bmtcril marked this pull request as ready for review April 1, 2024 17:40
@bmtcril bmtcril changed the title fix: Batching failures when backend is not configured fix: Batching failures when individual event errors bubble up Apr 1, 2024
@bmtcril bmtcril requested a review from ziafazal April 1, 2024 17:42
These were added in testing the batching failures, but I suspect they're unnecessary since I can't find any place that would trigger them.
@Ian2012 Ian2012 merged commit a8952cf into master Apr 1, 2024
10 checks passed
@Ian2012 Ian2012 deleted the bmtcril/batching_fixes branch April 1, 2024 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants