Retry upload on error.

[fulmicoton]

Some object storage can fail from to time. In that case we should retry instead of restarting the pipeline and losing all of the work done.

[fulmicoton]

This is low priority.

[guilload]

We should already be retrying a few times (3?). Is that not working, or does the transient storage issue persist for a duration longer than our retry delay?

The number of retries was set to a low value because, on the search side, we wanted some quick feedback when a storage issue occurred.

We could have a dedicated retry policy for PUT requests.

[trinity-1686a]

we already have retries on the storage for writes, up to 5 times. Some error were not retried, but now are since #5384, so I think this ticket can be closed

[fulmicoton]

@trinity-1686a Do we retry on S3 internal errors?

[trinity-1686a]

we retry based on what the sdk defines as transient and throttling errors, list here: https://docs.rs/aws-runtime/1.4.3/src/aws_runtime/retries/classifiers.rs.html#18-36
It doesn't include InternalError, so we don't retry on that

[trinity-1686a]

we do retry upload now for all transient errors we could find. If more error conditions should be retried, that should be a separate ticket