outputs.influxdb_v2 keeps looping on 500 response.

[eremmel]

Relevant telegraf.conf:

[[outputs.influxdb_v2]]
Any working config will do.

[[inputs.influxdb_listener]]
Any local input will do

System info:

telegraf 1.16.3
influxdb 2.x

Steps to reproduce:

I've configured a bucket with a retention policy of 2 weeks. When I write some measurements that are older than those two weeks, influxdb response with an 500 error.

1. setup influxdb v2
2. create bucket with a retention policy of N weeks
3. setup outputs.influxdb_v2 to connect to this bucket
4. start telegraf with debug
5. send some data to this bucket that is older than N weeks. E.g. via curl.
6. the outputs.influxdb_v2 will keep trying to resend the data to influxdb 2.
7. New measurements pile up after this request.

Expected behavior:

Report the error and discard the batch of measurements.

Actual behavior:

Error is reported but the batch of measurements is offered over and over again to influxdb

Error trace:

2020-12-16T11:02:09Z D! [outputs.influxdb_v2] Buffer fullness: 278 / 10000 metrics
2020-12-16T11:02:09Z E! [agent] Error writing to outputs.influxdb_v2: 500 Internal Server Error: internal error: unexpected error writing points to database: partial write: points beyond retention policy dropped=139
2020-12-16T11:02:19Z D! [outputs.http] Buffer fullness: 0 / 10000 metrics
2020-12-16T11:02:19Z D! [outputs.influxdb] Wrote batch of 139 metrics in 24.9356ms
2020-12-16T11:02:19Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2020-12-16T11:02:19Z D! [outputs.influxdb] Wrote batch of 139 metrics in 25.9331ms
2020-12-16T11:02:19Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2020-12-16T11:02:19Z E! [outputs.influxdb_v2] when writing to [http://54.226.129.163]: 500 Internal Server Error: internal error: unexpected error writing points to database: partial write: points beyond retention policy dropped=139
2020-12-16T11:02:19Z D! [outputs.influxdb_v2] Buffer fullness: 417 / 10000 metrics
2020-12-16T11:02:19Z E! [agent] Error writing to outputs.influxdb_v2: 500 Internal Server Error: internal error: unexpected error writing points to database: partial write: points beyond retention pol

[ssoroka]

Thanks! looking into this now