Automatic webhook retries feature

[liangyuanruo]

This issue is a catch-all for the automatic webhook retries feature

The ability to automatically attempt retries is a useful feature that can greatly reduce the amount of recovery work necessary for downstream webhook consumers. This can generally be implemented using a message queue with a visibility timeout, such as AWS SQS.

Design document

The design document should address the following:

1. Determine the queue message format, keeping an eye on future extensibility, such as retries with attachments
2. Identify and addressing risks to the system, such as potential for cascade failure
3. Clear communication to users of expectations and conditions under which retries will take place, to be spelled out in guide.form.gov.sg
4. Describe the configuration of both the messaging queue as well as the dead letter queue (DLQ)
5. Key metrics for monitoring for alerts, such as the number of messages in the DLQ
6. Recovery procedures for managing dropped messages

[r00dgirl]

is this closed? @mantariksh

[mantariksh]

@syan-syan yup but there isn't an issue open for the implementation yet, so we can either convert this issue to include implementation or open a new one

[karrui]

Mentioned the relevant PRs to this issue for future reference