Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement temp client banning on failed requests #38

Merged
merged 2 commits into from
May 7, 2024

Conversation

Starttoaster
Copy link
Owner

@Starttoaster Starttoaster commented May 7, 2024

For #35

Problem statement:
Prometheus needs quick exporter page loads for its scrape intervals, and attempting to load these metrics from a Proxmox server that is down can have the consequence of drastically slowing down page loads while waiting for the failed requests to that client to time out.

Proposed solution:
When a request fails against a particular client in the clients list, that client should be temporarily banned. A separate goroutine should run to inspect current bans, make a request against the Proxmox server, see if it succeeds, and unbans the client if the request succeeded. All API requests made for the exporter should check if the client they picked out at random was banned, and skip it if so.

Potential problems:

  • The first Prometheus scrape interval after a Proxmox host goes down, this exporter won't have banned the down client yet. So the first scrape interval that a Proxmox server is down for will still probably not get picked up by Prometheus. But all scrape intervals after that first one should succeed again. We could proactively ban clients on an interval, but Prometheus's scrape interval might beat us to it and find the down client first anyway.
  • If a request made by this exporter 404's, this banning mechanism as written here will ban the client. It would probably be better to only ban the client on 500+ status code responses, but this exporter doesn't accept human inputs for node names and such anyway, so a 404 shouldn't happen.

@Starttoaster Starttoaster merged commit de5af64 into main May 7, 2024
1 check passed
@Starttoaster Starttoaster deleted the client-banning branch May 7, 2024 22:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant