dbworker: Log when all retries/resets have been exhausted

[mrnugget]

It would be nice if the worker would log in failure_message when all retries have been exhausted.

Right now the num_resets column is bumped when a record is dequeued on retry. So, in pseudo-code, we could do something like this:

func MarkErrored(record Record, err error) {
  if record.NumResets == MaxRetries {
    record.FailureMessage = errors.Wrap(err, "retries exhausted. won't retry.")
    return
  }
  record.FailureMessage = err.Error()
}

[efritz]

@mrnugget Just thought of this: this should probably be a presentation concern in either the UI or the GraphQL layer. We can add convenience methods to help (e.g. record.WouldRetry()), though.

The data we need is already there the record (our num resets) and in the max retires constant. When presenting the error message, simply prepend retries exhausted. won't retry. previous error: (or similar) to the last failure. This won't totally erase the error conditions (which I think should still be available somehow to aid in debugging the reason it failed so many times in the first place).

Thoughts on this?

[mrnugget]

That makes sense to me, especially since you could change the MaxRetries constant and the record would be retried.