Feature request: Custom endpoint IP for http checks

[mblaschke]

Allow custom endpoint IP addresses for HTTP checks.
Sometimes services are exposed behind a NAT or not reachable via the published DNS address so it would be nice to specify a custom IP address for overriding the DNS request.

possible example:

  http_2xx:
    prober: http
    timeout: 30s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2"]
      preferred_ip_protocol: "ip4"
      endpoint: "10.1.2.3"

The current solution (creating a module for each check) is too much work and creates a huge configuration :(

[brian-brazil]

You can already override the host header, and running the blackbox exporter behind the NAT would also make things simpler.

The current solution (creating a module for each check) is too much work and creates a huge configuration :(

Automation is your friend.

[mblaschke]

We have around 5 LoadBalancer and several hundreds of host which we want to check.
So that means we have to create hundreds of seperated scraping configs AND blackbox modules.

Sure this is possible but it's not really elegant just to throw configuration against the system :)

When we can override the host header why not override the endpoint ip address? :)

[brian-brazil]

When we can override the host header why not override the endpoint ip address? :)

That's what the target URL parameter is.

[mblaschke]

Ah sure but we also need for each host one scape configuration as we also have to specify the module every time.

Also Prometheus supports service discovery (eg. kubernetes_sd or file_sd) with watches, but blackbox exporter lacks this feature, right?

In a cloud environment where the hosts are constantly changing this doesn't feel right to regenerate the configuration every time the cloud changes.

It would be nice to override the endpoint IP like we can override the headers instead of generating the configuration and reloading the blackbox exporter. Debugging in prometheus is much easier then in blackbox exporter.

[mblaschke]

Would also sponsor a beer or two 😊

[brian-brazil]

Also Prometheus supports service discovery (eg. kubernetes_sd or file_sd) with watches, but blackbox exporter lacks this feature, right?

Yes, that's Prometheus's responsibility.

This sounds like something that you can do with relabelling on the Prometheus end.

[mblaschke]

It's possible to set the Host header override via http parameter for the probe? hmm...haven't found it yet.

We just want to avoid a huge generated configuration if there is a possibility to override the endpoint.

[brian-brazil]

It's possible to set the Host header override via http parameter for the probe?

It's not.

[mblaschke]

But then it needs for every host an own configuration. That's exactly our problem and we cannot just use Prometheus's SD.

That's why I created this feature request :)

[brian-brazil]

If you've that complex a setup, then you'll have to handle it via your configuration management system.

You might also want to consider running the blackbox exporter behind the NAT.

[mblaschke]

We have also the same problem with some PaaS services in the cloud (eg. Azure) where you cannot put the blackbox exporter in between because it's not possible. Also putting the exporter behind the NAT is just not possible for several internal reasons (big company stuff and a lots of security issues).
We want to check every part of the services and detect possible outages, but for this we need to manipulate the endpoints. There a several services in between so we have to identify possible issues between.

We wanted to avoid throwing hundreds of configuration lines against the problem and wanted to solve this with a simple feature which can be used with Prometheus service discovery.

We want to avoid a fork and also think, that maybe other people could also have this issue.
If you say that this will never be a feature from blackbox exporter we have to deal with this statement and have to find another solution.

[brian-brazil]

No matter what changes are made here, I believe you'd need to have lots of automated config generation.

[mblaschke]

I don't understand your point.

If we can override the endpoint IP we can use Prometheus SD and the checks are fully automated with a single configuration line and one module in blackbox exporter.

Or we have to create one scraping config for prometheus AND one module config for blackbox exporter every time a new hosts is added.

So the second "solution" is better then using Prometheus SD? 🤔

[brian-brazil]

You need a target again per Host line though, so it's not that simple.

[mblaschke]

We already have a working solution for a lot of these checks using Prometheus SD, but not for "not directly reachable hosts" or "we want to just monitor the k8s ingress without PaaS service in front to check if the PaaS service is down again" using Prometheus service discovery.
Prometheus is already generating these configuration using service discovery - no we don't need a target per host if we can use service discovery and that's our use-case 😀

We just search for a solution to have a bunch of different blackbox exporter modules with different endpoints so we don't have to configure these targets without SD.

[brian-brazil]

I'm not seeing how you need this feature then, those use cases are fine without it.

[mblaschke]

Example:

To check if www.foobar.example is running using a Prometheus scape. Unfortunately www.foobar.example is exposed behind a NAT or a PaaS service so if you probe this URL you always check the result in the full chain of services.

What we want:
Check a subpart of the service chain (eg. only Ingress or a cloud LoadBalancer) and create an alert if one or more probes are failing to be informed if we have an issue with our cloud provider (PaaS services, networking ...) or if our application/loadbalancer is just failing.

Sure it's possible to override the endpoint using iptables or a dns masq but it's PITA in Kubernetes for just a bunch of Pods (blackbox exporter).

For exactly that usecase we want to override the endpoint. It's exactly the same usecase like the host header override but just the other way around.

[mblaschke]

It would also be fine to use the header override if it is settable via http params to make it possible to set this value using eg. annotations from the service discovery.

[brian-brazil]

Unfortunately www.foobar.example is exposed behind a NAT or a PaaS service so if you probe this URL you always check the result in the full chain of services.

Pass in the IP in the HTTP param, and override Host to www.foobar.example in the module configuration.

[mblaschke]

And that's exactly what we don't want as this makes Prometheus service discovery impossible to use because we need a different module configuration for every host in the system and blackbox exporter doesn't have any kind of service discovery.
The configuration needs to be in sync instead of a dynamic configuration.

It's very easy to achieve this feature with a possibilty to inject this configuration with a url parameter but it makes this very diffucult and very hard to debug if the configuration has generated every time the cloud is changing and keep that in sync with the service discovery in Prometheus.

I don't understand why this feature request is discussed in that way even if there are already three thumbs up 😞

[brian-brazil]

we need a different module configuration for every host in the system

Yes, you need a module for every test you wish to perform. Having otherwise would be unsustainable, and would turn this exporter into an open proxy.

[mblaschke]

That's why i asked for an endpoint configuration in the first place.

[brian-brazil]

The blackbox exporter is designed with the idea that you're preforming the same check against many many targets, not many many checks against one target.

[mblaschke]

And that's exactly what we want to avoid --> running the same check with only one target (and have for each target one unique module configuration).
That's why we want the endpoint IP over writable like it's possible for the Host header.

[brian-brazil]

That's not in line with how exporters are designed I'm afraid. Target discovery is the responsibility of Prometheus, not an exporter.

You'd end up having to generate the blackbox config with your proposal too, as you'd need a module per target.

[mblaschke]

I've implemented an example in PR #545
This solves a lot corporate challanges when using Prometheus service discovery and send the requests to a specific endpoint instead of using the IP address from DNS.

[brian-brazil]

As indicated on the PR, this doesn't make sense in terms of the overall design of this exporter.

[milunko]

Hi, we would require this feature as well.

I followed the entire discussion, and we are facing the exact same issues as the reporter of the issue.
Request really seems sensible, and argument that it breaks the design is not valid since as seen in the PR this adds just additional config field.

Could you reconsider adding this feature?

[2mf]

I think #795 would achieve that.
And I need that feature as well.