be able to persist failed probe history longer

[QingsongYao]

sometime, we only care about the failed probe logs, however since we many succeed logs, which cause the failed log was removed due to history.limit value.

[brian-brazil]

The history is only intended for low volume recent debugging. If you want something more medium to long term, you should enable debug logging.

[ktosiek]

I'm getting ~800kb/minute from debug logging on a pretty small instance (36 targets, 5s scrape interval).

I think having a log level that only shows basic info for failed probes would be desirable - I want to know which step failed (name resolution, connection, response, content) and maybe what IP was resolved, but I don't need much more. Would you consider a PR that changes results logging in such way?

[brian-brazil]

Info-level logging is only for blackbox-exporter level issues, which a probe failure is not, while debug logging is for everything so I'm not seeing the scope for that.

[QingsongYao]

I still feel that the failed probe record need to persist longer. it is not related to logging, but recent probe html page which is super help to debug what is wrong.

[brian-brazil]

You can always change history.limit

[QingsongYao]

when probing with large amount of urls, the history will be quickly override.
checking logs is not an option since log might not be available for end user, and it is hard to search jobs. It is a real scenario since user want to know why probe fail and history is the only option for them.

[jutley]

I'd like to see this feature be added. I'm running into the exact issue that @QingsongYao is describing. One of our probes has occasional failures that I would like to debug. The last instance of this was 1.5 hours ago. We currently have about 200 probes a minute, so to be able to keep this history, we'd need to keep the last 18,000 scrapes, of which 17,995 are uninteresting.

Similar to how Kubernetes CronJobs have a different limit for successes and failures, I think the blackbox-exporter should support a similar feature.

[brian-brazil]

Feel free to send a PR