tsh should optionally not try to use HTTPS_PROXY for ssh connections

[programmerq]

Expected behavior:

when HTTPS_PROXY is set, tsh should not make connections to the proxy ssh port via the HTTP CONNECT proxy. It should only make a connection to port 443 using the proxy. Related to #13048 and #12215.

Current behavior:

When tsh has HTTPS_PROXY set, it should affect the https connections. The ssh connections it makes should have a separate method for configuring it. Some corporate HTTP CONNECT style proxies simply do not allow connections to ports other than 80 and 443, and no other egress method is available for port 80/443.

Bug details:

• Teleport version - tsh 8.3.5-8.3.13 and tsh 9.0.2-9.3.3 are affected.
• Recreation steps

run a http connect proxy. Only allow connection requests to port 443. Here's a tinyproxy.conf config file that works with tinyproxy 1.11.1 installed via homebrew.

User nobody
Group nobody
Port 8888
Timeout 600
DefaultErrorFile "/usr/local/Cellar/tinyproxy/1.11.0/share/tinyproxy/default.html"
StatFile "/usr/local/Cellar/tinyproxy/1.11.0/share/tinyproxy/stats.html"
LogLevel Info
MaxClients 100
Allow 127.0.0.1
Allow ::1
ViaProxyName "tinyproxy"
ConnectPort 443

Set the proxy variable export HTTPS_PROXY=http://127.0.0.1:8888 and run a tsh login command against a cluster that has tls multiplexing disabled.

• Debug logs

jeff@mbp z5471 % HTTPS_PROXY=http://127.0.0.1:8888 tsh -d login --user jeff --proxy teleport.example.com
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/Users/jeff/.tsh/keys/teleport.example.com/jeff-x509.pem" valid until "2022-06-09 03:20:46 +0000 UTC". client/keystore.go:307
DEBU [KEYSTORE]  Reading certificates from path "/Users/jeff/.tsh/keys/teleport.example.com/jeff-ssh/teleport.example.com-cert.pub". client/keystore.go:330
INFO [CLIENT]    no host login given. defaulting to jeff client/api.go:1369
INFO [CLIENT]    [KEY AGENT] Connected to the system agent: "/private/tmp/com.apple.launchd.k7Xosnid64/Listeners" client/api.go:3393
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/Users/jeff/.tsh/keys/teleport.example.com/jeff-x509.pem" valid until "2022-06-09 03:20:46 +0000 UTC". client/keystore.go:307
DEBU [KEYSTORE]  Reading certificates from path "/Users/jeff/.tsh/keys/teleport.example.com/jeff-ssh/teleport.example.com-cert.pub". client/keystore.go:330
INFO [KEYAGENT]  Loading SSH key for user "jeff" and cluster "teleport.example.com". client/keyagent.go:197
DEBU [CLIENT]    not using loopback pool for remote proxy addr: teleport.example.com:443 client/api.go:3354
DEBU             Attempting GET teleport.example.com:443/webapi/ping webclient/webclient.go:118
Enter password for Teleport user jeff:
DEBU [CLIENT]    not using loopback pool for remote proxy addr: teleport.example.com:443 client/api.go:3354
DEBU [CLIENT]    HTTPS client init(proxyAddr=teleport.example.com:443, insecure=false) client/weblogin.go:232
DEBU [KEYAGENT]  Adding CA key for teleport.example.com client/keyagent.go:319
DEBU [KEYSTORE]  Adding known host teleport.example.com with proxy teleport.example.com and key: SHA256:ck0UQzcG2dg3kdkDoDAyaWujhAWHnBZBKppqNauSl9s client/keystore.go:578
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/Users/jeff/.tsh/keys/teleport.example.com/jeff-x509.pem" valid until "2022-06-09 03:20:46 +0000 UTC". client/keystore.go:307
DEBU [KEYAGENT]  Deleting obsolete stored key with index {ProxyHost:teleport.example.com Username:jeff ClusterName:teleport.example.com}. client/keyagent.go:520
INFO [KEYAGENT]  Loading SSH key for user "jeff" and cluster "teleport.example.com". client/keyagent.go:197
INFO [CLIENT]    Connecting to proxy=teleport.example.com:3023 login="root" client/api.go:2588
DEBU [HTTP:PROX] Found proxy "http://127.0.0.1:8888" in environment, returning proxy dialer. proxy/proxy.go:283

ERROR REPORT:
Original Error: *trace.BadParameterError unable to proxy connection: 403 Access violation
Stack Trace:
	/private/tmp/build-darwin-amd64/go/src/github.com/gravitational/teleport/api/client/proxy.go:72 github.com/gravitational/teleport/api/client.DialProxyWithDialer
	/private/tmp/build-darwin-amd64/go/src/github.com/gravitational/teleport/api/client/proxy.go:32 github.com/gravitational/teleport/api/client.DialProxy
	/private/tmp/build-darwin-amd64/go/src/github.com/gravitational/teleport/lib/utils/proxy/proxy.go:203 github.com/gravitational/teleport/lib/utils/proxy.proxyDial.Dial
	/private/tmp/build-darwin-amd64/go/src/github.com/gravitational/teleport/lib/client/api.go:2599 github.com/gravitational/teleport/lib/client.makeProxySSHClientDirect
	/private/tmp/build-darwin-amd64/go/src/github.com/gravitational/teleport/lib/client/api.go:2589 github.com/gravitational/teleport/lib/client.makeProxySSHClient
	/private/tmp/build-darwin-amd64/go/src/github.com/gravitational/teleport/lib/client/api.go:2533 github.com/gravitational/teleport/lib/client.(*TeleportClient).connectToProxy
	/private/tmp/build-darwin-amd64/go/src/github.com/gravitational/teleport/lib/client/api.go:2453 github.com/gravitational/teleport/lib/client.(*TeleportClient).ConnectToProxy.func1
	/var/folders/ys/8czjjsys38x504kj8172pd_m0000gp/T/drone-GI6ktjPX660KXBHj/home/drone/build-383-1654023581-toolchains/go/src/runtime/asm_amd64.s:1581 runtime.goexit
User Message: failed to authenticate with proxy teleport.example.com:3023
	unable to proxy connection: 403 Access violation
jeff@mbp z5471 %

gz#5471

[klizhentas]

@r0mant will check with @atburke to make sure that NO_PROXY respects the SSH port when using tsh

[atburke]

NO_PROXY does respect ports in tsh as of #10209.

[ollyg]

Our customer tested and reports that using NO_PROXY as "jumphost.example.com:22" does work around this. Thank you!

Still, applying HTTP[S]_PROXY to non-HTTP traffic is perhaps still wrong, per the original issue description, and should be patched to be configurable somehow.

[r0mant]

Glad to hear that using NO_PROXY addresses your use-case. We do have customers expecting the opposite behavior (respecting HTTP_PROXY for all connections) so unlikely going to provide an option to configure this since NO_PROXY workaround works.