PubSub

[matt2e]

Design doc is here.

Major steps:

• PubSub Schema Changes #1594
• Go schema extraction #1599
• feat: add topics, subscriptions & subscribers to db on deployment #1608
• PubSub grpc changes #1607
• PubSub DB changes #1595
• PubSub controller changes #1596
• ftltest support for PubSub #1598
• Retries for pubsub #1658
• PubSub Batching #1674 (when?)

Issues:

• Handle calling Publish() on another module's topic #1640
• Improve guarantees for pubsub when 2 events are inserted at the same time #1642
• Remove pubsub reliance on min_replicas #1660
• Pubsub can end up blocked by subscriptions with no subscribers #1685
• Add runtime safety checks when publishing to external topics #1703

[matt2e]

Case to handle:

• Deployment registers a topic with event A
• Updated deployment happens, now with the topic defined with event B
• ... what should happen?

[matt2e]

Case to handle:

• Deployment registers a subscription
  • Some events are consumed by the subscription. Curser points to event row A
• Replace deployment, now without the subscription
  • Quick thoughts: Does that subscription row just sit there untouched? Should we be deleting it? Will this work ok when there are multiple versions of a module? Don't worry about that for now?
• ... Wait a week
• Replace deployment again, now with a subscription with the same name as the old subscription
• ... is the cursor still pointing to a week ago?

[matt2e]

Thing to think about:
When a module is updated and removes a topic, should the topic just sit there?

• The module won't be able to publish any new events anyway...
• It allows time for subscribers to catch up to the remaining unhandled events, which is good

[matt2e]

Case to handle:

• Can not publish to external module's topic

[matt2e]

Queries are doing a lot of joins. Should we denormalize module name into topics?

[alecthomas]

For now we won't worry about catching any of these cases, we want a holistic approach to managing schema changes across FTL.

[alecthomas]

Queries are doing a lot of joins. Should we denormalize module name into topics?

Yeah, we do it in other places, so if you think it's necessary go for it.

[alecthomas]

I'm going to close this out. We can revisit later if we decide to do batching. Nice work Matt.