Support proxy environment variables

[ThinkChaos]

Discussed in #1409

^{Originally posted by ezekieldas March 19, 2024}
I'm testing blocky in an environment which requires all http/https requests to use a forward proxy. I have the following env vars defined on OS:

no_proxy=localhost, .somewhere.io
ftp_proxy=ftp://proxy.somewhere.io:8118
https_proxy=http://proxy.somewhere.io:8118
HTTPS_PROXY=http://proxy.somewhere.io:8118
HTTP_PROXY=http://proxy.somewhere.io:8118
http_proxy=http://proxy.somewhere.io:8118

When I start blocky as a standalone binary, requests against blocklists fail:

[2024-03-19 17:19:17]  WARN list_cache: Temporary network err / Timeout occurred: temporary error occurred: Get "http://sysctl.org/cameleon/hosts": dial tcp 213.186.34.12:80: i/o timeout (Client.Timeout exceeded while awaiting headers) attempt=2/3 link=http://sysctl.org/cameleon/hosts
[2024-03-19 17:19:17]  WARN list_cache: Temporary network err / Timeout occurred: temporary error occurred: Get "https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts": context deadline exceeded (Client.Timeout exceeded while awaiting headers) attempt=2/3 link=https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

Yet when I test this in an environment where 80/443 are not blocked, blocky starts just fine:

[2024-03-19 17:31:44]  INFO list_cache: import succeeded count=20562 source=http://sysctl.org/cameleon/hosts
[2024-03-19 17:31:45]  INFO list_cache: import succeeded count=178085 source=https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

I'm not well versed in Go but is there a library which could be included in the blocky build to support forward proxy?

---

This shouldn't be hard to support. Go has support for the env proxy variables in the standard library. It's even enabled by default.
We use a custom "HTTP Transport" so we can override the name resolution, and provide the right SNI for DoH/DoT. That has the side effect of disabling the env proxy support.

I'll look at basing our transport on the default one instead of building it from scratch.

[ezekieldas]

Thanks for responding to this so quickly! Based on what you've noted above I don't think this will help but going to mention anyway... Prometheus, also Go based, has an option for scrapers behind proxies proxy_from_environment Details: https://github.com/search?q=org%3Aprometheus+proxy_from_environment&type=code

[ThinkChaos]

Yep that Prometheus option maps to Go's http.ProxyFromEnvironment, which is what Go uses by default, but was lost in the way we build our HTTP "Transport"/config.
I made a PR with the fix :)