Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: pubsub manager #1673

Merged
merged 7 commits into from
Jun 6, 2024
Merged

feat: pubsub manager #1673

merged 7 commits into from
Jun 6, 2024

Conversation

matt2e
Copy link
Collaborator

@matt2e matt2e commented Jun 5, 2024

closes #1596 closes #1642

How it works:

  • pubsub managers poll for subscriptions that are behind the topic's head
    • locks these subscriptions for the tx
    • skips subscriptions that are already locked
  • Then finds the next event for each subscription
    • If the event is newer than our artificial delay, we skip this subscription on this attempt

Pubsub also speeds things up by:

  • Polling once a second
  • When it schedules an async call, notifies controller so we can attempt to pick it up straight away
    • this is wrapped in a mutex so that we don't accidentally get more and more async calls running at once on a single controller
  • When the async call is completed, pubsub is notified so that it can immediately try progressing subscriptions again

@github-actions github-actions bot changed the title Matt2e/pubsub manager feat(pubsub): Matt2e/pubsub manager Jun 5, 2024
@ftl-robot ftl-robot mentioned this pull request Jun 5, 2024
@matt2e matt2e force-pushed the matt2e/pubsub-manager branch 3 times, most recently from 9ac848d to 1bc032b Compare June 6, 2024 02:03
@matt2e matt2e force-pushed the matt2e/pubsub-manager branch from 1bc032b to 933b492 Compare June 6, 2024 02:12
@matt2e matt2e marked this pull request as ready for review June 6, 2024 02:15
@matt2e matt2e requested a review from alecthomas as a code owner June 6, 2024 02:15
@matt2e matt2e requested review from a team and wesbillman and removed request for a team June 6, 2024 02:15
@matt2e matt2e changed the title feat(pubsub): Matt2e/pubsub manager feat: pubsub manager Jun 6, 2024
@matt2e matt2e merged commit c9a8561 into main Jun 6, 2024
35 checks passed
@matt2e matt2e deleted the matt2e/pubsub-manager branch June 6, 2024 03:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve guarantees for pubsub when 2 events are inserted at the same time PubSub controller changes
2 participants