Non-retryable Prometheus remote write errors should return 400

[murphyzlaw]

Following is a software issue for Prometheus remote write and occurs on all O/S, hardware type, and M3 and Prometheus configurations.

Once samples get older than value of the bufferPastDuration in the namespace, each sample will be rejected outright with a 500 error. However the Prometheus remote write API considers 5xx errors retryable: https://github.com/prometheus/prometheus/blob/master/storage/remote/client.go#L111

M3 should return 400 for the too old data instead, as the condition will never self-fix.

The Prometheus issue prometheus/prometheus#5636 makes the impact more severe, because failed (too old) samples will retry indefinitely, stopping metrics flow completely from the given Prometheus.

Issue observed for "timestamp too far in the past" but other non-retryable errors may also experience this.

[robskillington]

Agreed, I've opened PR #1692 to address this with a docker integration test to ensure this is checked by CI moving forward too.

[richardartoul]

@robskillington is it safe to close this or do we still have some pending work?

[robskillington]

Closing this, confirmed working as intended now