Exponential backoff problem

[tumb1er]

RMQ docs say that only a message from queue HEAD is expired via TTL, so if you have a 1024 seconds TTL message on top of retry queue, all your 100500 1-second TTL messages will still wait 1024 seconds to expire and re-queue to initial queue.

That makes exponential backoff settings almost useless.

[tumb1er]

One of solutions:

• Topic exchange recover
• Incoming queues are bound with same routing keys to both events and recover exchanges (separate recover exchange prevents retrying messages to another services)
• Incoming queues has DLX set to exchange delay.1 (retry for connection problems)
• Fanout exchanges delay.1 - delay.N with bound queues delay.1 - delay.N
• Each queue has x-message-ttl=2^N and DLX=recover

1. Events exchange receives a message some_event
2. It is routed by routing key some_event to queue consumer.some_event
3. In case of retry we increment retries counter and change exchange to delay.C where C is current number of retries
4. We publish message to exchange delay.1, where it is automatically routed with initial routing key some_event to queue delay.1
5. After one second the message in delay.1 queue is expired via x-message-ttl queue policy and is routed to DLX recover
6. From recover exchange it is routed via some_event routing key to queue consumer.some_event
7. On each retry we send new message with incremented retries counter to a new delay.N exchange, and that's exponential backoff with constant number of queues
8. If we have an error while re-publishing retried message (i.e. no more new connections could be opened), message can be rejected with basic.reject(requeue=false). This message will be routed to DLX delay.1 and retried after one second (constant backoff in case of RMQ send errors).