HTTP Probe fails due to DNS resolution when used with proxy

[sathiyajith]

Host operating system: Linux

blackbox_exporter version: 0.19.0

What is the blackbox.yml module config.

modules:
  http_2xx_probes:
    prober: http
    timeout: 90s
    http:
      valid_status_codes: []
      fail_if_body_matches_regexp:
        - ".*(ERROR).*"
      headers:
        Host: https://www.example.com:1234
      preferred_ip_protocol: "ip4"
      ip_protocol_fallback: false
      proxy_url: http://proxy-username:proxy-password@internal-proxy-server:3120
      tls_config:
        insecure_skip_verify: true

What is the prometheus.yml scrape config.

- job_name: test
  honor_timestamps: true
  params:
    domain:
    - test-domain
    module:
    - http_2xx_probes
  scrape_interval: 5m
  scrape_timeout: 4m
  metrics_path: /probe
  scheme: http
  file_sd_configs:
  - files:
    - /etc/prometheus/testfile.yaml
    refresh_interval: 1m
  proxy_url: http://proxy-username:proxy-password@internal-proxy-server:3120
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    target_label: __param_target
    replacement: $1
    action: replace
  - source_labels: [__param_target]
    separator: ;
    regex: (.*)
    target_label: instance
    replacement: $1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: __address__
    replacement: blackbox-exporter:9115
    action: replace

What logging output did you get from adding &debug=true to the probe URL?

target=https://www.example.com:1234 level=debug msg="Error for HTTP request" err="Get \"https://123.45.67.8:1234\": Forbidden"
ts=2022-01-27T06:38:39.363Z caller=main.go:180 module=http_2xx_probes 
target=https://www.example.com:1234 level=debug msg="Response timings for roundtrip" roundtrip=0 start=2022-01-27T06:38:39.348049772Z dnsDone=2022-01-27T06:38:39.355050347Z connectDone=2022-01-27T06:38:39.355463276Z gotConn=0001-01-01T00:00:00Z responseStart=0001-01-01T00:00:00Z tlsStart=0001-01-01T00:00:00Z tlsDone=0001-01-01T00:00:00Z end=0001-01-01T00:00:00Z
ts=2022-01-27T06:38:39.363Z caller=main.go:180 module=http_2xx_probes 
target=https://www.example.com:1234 level=debug msg="Probe failed" duration_seconds=0.023786742

What did you do that produced an error?

I added the proxy in http_probe to hit that specific url.

What did you expect to see?

I expected a 200 response from the http_probe.

What did you see instead?

The probe fails with 403 Forbidden error due to DNS resolution. From the above error logs, I found out that blackbox exporter pod is trying to do dns resolution and using the resolved ip address and port to pass through the proxy-server.

Note: i have replaced the actual DNS Name, IP Address and Proxy server (squid) details for privacy purpose in these logs.
Issue : The probe fails with 403 Forbidden error.
From the above error logs, I found out that blackbox exporter pod is trying to do DNS resolution for the public target (https://www.example.com:1234) and using the resolved ip address and port (https://123.45.67.8:1234) to pass through the squid proxy-server (http://proxy-username:proxy-password@internal-proxy-server:3120).
In our proxy server we only have the DNS url (https://www.example.com:1234) whitelisted and not it's IP Address.
Reason: IP Address keeps changing and the proxy server admins in any org will not add dynamic ip's onto the whitelist due to various reasons.

We tried adding host header (host: https://www.example.com:1234) also as a flag on the blackbox exporter config yaml. However, proxy is not able to recognize that and we continue to get the same error

To resolve this issue, we would request blackbox exporter source code owners/maintainers to add a feature/flag (Eg:- use_proxy_dns: true) to not resolve the DNS into IP Address. It will help various organizations, who are having similar proxy setup and is facing similar issues, so that they dont need to look into any other exporter/monitoring solutions.

[shabeermm]

We are also facing similar issue. Please help in addressing this issue.

[Venkat2694]

We are also facing similar issue. Please help in addressing this issue.

[jayninja]

+1 here Im sitting here troubleshooting hitting a health check URL through a squid proxy to test squid. Its failing because the target uses round robin dns. The squid has a whitelist with the dns name in it. Since blackbox is resolving and then querying the IP squid fails.

[ChristianCiach]

This will probably never be fixed. See

• Add use_proxy_dns to bypass local DNS resolution #554
• http probe: skip DNS lookup if using proxy #338

[roidelapluie]

Hello,

I think this is a reasonable feature request. It could be build on top of existing pull requests, but the effects of the settings are more than just "skip DNS", it also skips protocol selection, so it would need to be properly documented.

[roidelapluie]

Maybe it means that we need another probe type, proxy_http instead of http, WDYT ?

[SCLogo]

same issue here. any temp fix until this issue will be resolved?

[wikitops]

Same problem here. The issue does not seem to be only related to the proxy parameter. I can see from my configuration that Blackbox resolves all our endpoint to an IP and uses that IP to run the check. For HTTP verification, it adds the host to the header, I guess that’s why it works correctly on simple internal endpoints.

But for public endpoints behind dynamic IP (with or without proxy), adding the host won’t make it work (check this ticket for example: #886)

I don’t think skipping DNS is an option, this feature is important for Blackbox. But perhaps adding an option to use or not use the IP or DNS name during verification would be a good approach.

[mem]

@wikitops I'm missing something... why wouldn't adding a Host header work in the specific case of dynamic IPs? The name resolution happens each time the probe runs.

The reason why it performs DNS resolution first and then uses the resolved IP address + Host header is consistency: if we pass the unresolved host name to the HTTP client, it will do resolution a second time, possibly obtaining a different IP address.

[paceyuk]

I had this problem too, we use whitelisting in our outbound proxy based on destination hostname. When blackbox_exporter makes an HTTP request, it resolves the hostname to IP and makes the request to the IP which meant the proxy didn't have any idea what URL was originally requested, only the IP. We can't replace the whitelist with a list of IPs instead because things created in AWS aren't guaranteed to always remain on those same IPs (depending on the product).

[RickBankers]

We really need this feature too. Instead of wasting time trying to fix the problems with the existing http_proxy probe why not just create an additional probe type like "http_probe_proxy" and have it resolve by name only rather than IP. This way it won't break the other dependencies and could be used as a one-off for these situations until a longer term fix is found.

[ChristianCiach]

The maintainers of this project clearly have no interest in fixing this issue. We've switched to https://github.com/ricoberger/script_exporter to workaround this issue. Using this exporter we can just use curl and similar commands to do these checks.

[roidelapluie]

I already expressed that I would like to have this feature, see #877 (comment)

As an open source project, everyone can contribute this feature.

[RickBankers]

@ChristianCiach Do you have any examples on how to use script_exporter to check urls and return the same metrics as the normal probe? If you have any example configs for checking a single url that would help me get started.

[roidelapluie]

Could you please have a look at #944 ?

[roidelapluie]

Closed by blackbox exporter 0.22.0 and skip_resolve_phase_with_proxy

[senemamourr]

i use blackbox exporter version 0.24.0 with proxy_url and always get 403 status code.
my target is in format https://sitename/health.
Does it support this format target ?