Outbox loader

[mrmasterplan]

Sometimes we have low throughput loading tasks to external systems. Examples of such systems are

• cosmos DB
• eventHub

A nice feature would be to address these issues:

• long (10h+) loading jobs that can fail halfways and should be resumed instead of retried
• rate limiting to below requisitioned throughput would help the connected system
• sometimes we don't know what we have sent, having a record of transmitted data could be useful in finding errors

All of these could be addressed with an "outbox" table.
The outbox table is strictly append-only. Every action taken towards the end system should be written to the outbox table first.
In this way, the outbox table forms a record of exported messages.
The outbox table should then be streamed to the external system. Streaming allows for rate limiting and resuming.