tsh ignores HTTPS_PROXY #8108

programmerq · 2021-08-31T19:26:19Z

Description

With HTTPS_PROXY set, I would expect tsh login to use that. It seems to ignore that and instead try to make the connections directly.

Our docs do make reference to using HTTP_PROXY / HTTPS_PROXY environment variables, but not specifically for tsh.

Reproduction Steps

I have a clonable gist that illustrates the problem: https://gist.github.com/programmerq/d053ae8acde4c7e467b44a32817750ef

Clone the gist, put your own license.pem (since it is using teleport enterprise), and follow the instructions in the gist readme.

If not using the gist, the idea is to set up a teleport server and try to access it via an HTTP CONNECT style proxy with http_proxy, HTTP_PROXY, https_proxy, and HTTPS_PROXY all set to the same thing. In my test, I used a tinyproxy instance and confirmed that curl was able to access the teleport API endpoint successfully with those variables set. In a proper lab setup, unsetting the variables should cause a failure due to the teleport URL due to timing out.

Server Details

reproduced in the compose file with the following:

Teleport version (run teleport version): 7.1.0 (from the quay.io/gravitational/teleport-ent:7.1.0 docker image)
Server OS (e.g. from /etc/os-release): Ubuntu 20.04.3 LTS
Where are you running Teleport? (e.g. AWS, GCP, Dedicated Hardware): docker container via compose
Additional details: https://gist.github.com/programmerq/d053ae8acde4c7e467b44a32817750ef "teleport" container in compose file.

Client Details

Tsh version (tsh version): Teleport v7.1.0 git:v7.1.0-0-gb52a7d89f go1.16.2
Computer OS (e.g. Linux, macOS, Windows): CentOS Linux release 7.9.2009
Browser version (for UI-related issues): N/A
Installed via (e.g. apt, yum, brew, website download): https://get.gravitational.com/teleport-ent-7.1.0-1.x86_64.rpm
Additional details: https://gist.github.com/programmerq/d053ae8acde4c7e467b44a32817750ef "shell" container in compose file

Debug Logs

Please include or attach debug logs, when appropriate. Obfuscate sensitive information!

Start Teleport with --debug flag (teleport --debug)
Run tsh with --debug flag (tsh --debug)

[root@236b59f41927 ~]# tsh --debug login --user admin --proxy teleport:3080 --insecure
DEBU [CLIENT]    Failed to stat file: stat /root/.tsh: no such file or directory. client/api.go:679
INFO [CLIENT]    no host login given. defaulting to root client/api.go:1033
ERRO [CLIENT]    [KEY AGENT] Unable to connect to SSH agent on socket: "". client/api.go:2780
DEBU [CLIENT]    not using loopback pool for remote proxy addr: teleport:3080 client/api.go:2745

ERROR REPORT:
Original Error: *url.Error Get &#34;https://teleport:3080/webapi/ping&#34;: dial tcp 192.168.208.40:3080: connect: connection timed out
Stack Trace:
	/go/src/github.com/gravitational/teleport/vendor/github.com/gravitational/teleport/api/client/webclient/webclient.go:87 github.com/gravitational/teleport/api/client/webclient.Ping
	/go/src/github.com/gravitational/teleport/lib/client/api.go:2342 github.com/gravitational/teleport/lib/client.(*TeleportClient).Ping
	/go/src/github.com/gravitational/teleport/lib/client/api.go:2196 github.com/gravitational/teleport/lib/client.(*TeleportClient).Login
	/go/src/github.com/gravitational/teleport/tool/tsh/tsh.go:800 main.onLogin
	/go/src/github.com/gravitational/teleport/tool/tsh/tsh.go:588 main.Run
	/go/src/github.com/gravitational/teleport/tool/tsh/tsh.go:261 main.main
	/opt/go/src/runtime/proc.go:225 runtime.main
	/opt/go/src/runtime/asm_amd64.s:1371 runtime.goexit
User Message: Get &#34;https://teleport:3080/webapi/ping&#34;: dial tcp 192.168.208.40:3080: connect: connection timed out
[root@236b59f41927 ~]#

The text was updated successfully, but these errors were encountered:

russjones · 2022-01-29T00:45:26Z

Related see #9376

johns-carta · 2022-02-09T22:48:01Z

hey @atburke I'm the original reporter


tsh login --proxy=teleport.[redacted]:3080 --insecure --auth=local --user=admin -d
DEBU [CLIENT]    open /Users/john/.tsh/teleport.[redacted].yaml: no such file or directory client/api.go:751
INFO [CLIENT]    no host login given. defaulting to john client/api.go:1102
INFO [CLIENT]    [KEY AGENT] Connected to the system agent: "/private/tmp/com.apple.launchd.t061w1Pczn/Listeners" client/api.go:2963
DEBU [CLIENT]    not using loopback pool for remote proxy addr: teleport.[redacted]:3080 client/api.go:2924
DEBU             Attempting GET teleport.[redacted]:3080/webapi/ping/local webclient/webclient.go:63
Enter password for Teleport user admin:
Enter your OTP token:
231099
DEBU [CLIENT]    not using loopback pool for remote proxy addr: teleport.[redacted]:3080 client/api.go:2924
DEBU [CLIENT]    HTTPS client init(proxyAddr=teleport.[redacted]:3080, insecure=true) client/weblogin.go:221
WARNING: You are using insecure connection to SSH proxy https://teleport.[redacted]:3080
DEBU [KEYAGENT]  Adding CA key for teleport.[redacted] client/keyagent.go:302
DEBU [KEYSTORE]  Adding known host teleport.[redacted] with proxy teleport.[redacted] and key: SHA256:GrwFsPglKOhzHgSf7yMLGRadVIi/t0qH0XNTS6KMjsg client/keystore.go:544
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/Users/john/.tsh/keys/teleport.[redacted]/admin-x509.pem" valid until "2022-02-10 09:44:34 +0000 UTC". client/keystore.go:283
DEBU [KEYAGENT]  Deleting obsolete stored key with index {ProxyHost:teleport.[redacted] Username:admin ClusterName:teleport.[redacted]}. client/keyagent.go:479
INFO [KEYAGENT]  Loading SSH key for user "admin" and cluster "teleport.[redacted]". client/keyagent.go:180
INFO [CLIENT]    Connecting proxy=teleport.[redacted]:3080 login="-teleport-nologin-159b94f6-b10b-4f20-a9a1-dcfb43427e9d" client/api.go:2156
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/Users/john/.tsh/keys/teleport.[redacted]/admin-x509.pem" valid until "2022-02-10 09:45:38 +0000 UTC". client/keystore.go:283


ERROR REPORT:
Original Error: *net.OpError dial tcp [redacted IP]:3080: connect: operation timed out
Stack Trace:
    /tmp/build-darwin-amd64/go/src/github.com/gravitational/teleport/lib/client/api.go:2189 github.com/gravitational/teleport/lib/client.makeProxySSHClientWithTLSWrapper
    /tmp/build-darwin-amd64/go/src/github.com/gravitational/teleport/lib/client/api.go:2201 github.com/gravitational/teleport/lib/client.makeProxySSHClient
    /tmp/build-darwin-amd64/go/src/github.com/gravitational/teleport/lib/client/api.go:2158 github.com/gravitational/teleport/lib/client.(*TeleportClient).connectToProxy
    /tmp/build-darwin-amd64/go/src/github.com/gravitational/teleport/lib/client/api.go:2077 github.com/gravitational/teleport/lib/client.(*TeleportClient).ConnectToProxy.func1
    /var/folders/ys/8czjjsys38x504kj8172pd_m0000gp/T/drone-wsZk42ffif1Da26m/home/drone/build-10133-1644289590-toolchains/go/src/runtime/asm_amd64.s:1581 runtime.goexit
User Message: failed to dial tls teleport.[redacted]:3080
    dial tcp [redacted IP]:3080: connect: operation timed out
john@john ~ %
john@john ~ % nslookup teleport.[redacted]
Server:     127.0.0.1
Address:    127.0.0.1#53

Non-authoritative answer:
teleport.[redacted] canonical name = [redacted].elb.us-east-1.amazonaws.com.
Name:   [redacted].elb.us-east-1.amazonaws.com
Address: [redacted IP]

this tls.Dial here will fail:

teleport/lib/client/api.go

Line 2211 in d33f51d

tlsConn, err := tls.Dial("tcp", cfg.WebProxyAddr, clientTLSConf)

It should be using something like this:

teleport/lib/utils/proxy/proxy.go

Line 216 in e8f9220

func DialerFromEnvironment(addr string, opts ...DialerOptionFunc) Dialer {

russjones · 2022-02-09T22:56:12Z

@johns-carta This is with 8.1.6-dev.1?

johns-carta · 2022-02-09T23:02:13Z

Correct, Teleport v8.1.6-dev.1 git:v8.1.6-dev.1-0-g40cbd171f go1.17.3

johns-carta · 2022-02-10T01:49:11Z

this works

diff --git a/lib/client/api.go b/lib/client/api.go
index dadf89ae0..54a3b0535 100644
--- a/lib/client/api.go
+++ b/lib/client/api.go
@@ -55,6 +55,7 @@ import (
 	"github.com/gravitational/teleport/api/types/wrappers"
 	apiutils "github.com/gravitational/teleport/api/utils"
 	"github.com/gravitational/teleport/api/utils/keypaths"
+	"github.com/gravitational/teleport/lib"
 	"github.com/gravitational/teleport/lib/auth"
 	"github.com/gravitational/teleport/lib/client/terminal"
 	"github.com/gravitational/teleport/lib/defaults"
@@ -63,11 +64,11 @@ import (
 	"github.com/gravitational/teleport/lib/services"
 	"github.com/gravitational/teleport/lib/session"
 	"github.com/gravitational/teleport/lib/shell"
-	alpncommon "github.com/gravitational/teleport/lib/srv/alpnproxy/common"
 	"github.com/gravitational/teleport/lib/sshutils/scp"
 	"github.com/gravitational/teleport/lib/tlsca"
 	"github.com/gravitational/teleport/lib/utils"
 	"github.com/gravitational/teleport/lib/utils/agentconn"
+	"github.com/gravitational/teleport/lib/utils/proxy"
 
 	"github.com/gravitational/trace"
 
@@ -2198,31 +2199,15 @@ func (tc *TeleportClient) connectToProxy(ctx context.Context) (*ProxyClient, err
 	}, nil
 }
 
-func makeProxySSHClientWithTLSWrapper(tc *TeleportClient, sshConfig *ssh.ClientConfig) (*ssh.Client, error) {
-	cfg := tc.Config
-	clientTLSConf, err := tc.loadTLSConfig()
-	if err != nil {
-		return nil, trace.Wrap(err)
-	}
-
-	clientTLSConf.NextProtos = []string{string(alpncommon.ProtocolProxySSH)}
-	clientTLSConf.InsecureSkipVerify = cfg.InsecureSkipVerify
-
-	tlsConn, err := tls.Dial("tcp", cfg.WebProxyAddr, clientTLSConf)
-	if err != nil {
-		return nil, trace.Wrap(err, "failed to dial tls %v", cfg.WebProxyAddr)
-	}
-	c, chans, reqs, err := ssh.NewClientConn(tlsConn, cfg.WebProxyAddr, sshConfig)
-	if err != nil {
-		// tlsConn is closed inside ssh.NewClientConn function
-		return nil, trace.Wrap(err, "failed to authenticate with proxy %v", cfg.WebProxyAddr)
-	}
-	return ssh.NewClient(c, chans, reqs), nil
-}
-
 func makeProxySSHClient(tc *TeleportClient, sshConfig *ssh.ClientConfig) (*ssh.Client, error) {
 	if tc.Config.TLSRoutingEnabled {
-		return makeProxySSHClientWithTLSWrapper(tc, sshConfig)
+		if (tc.Config.InsecureSkipVerify) {
+			lib.SetInsecureDevMode(tc.Config.InsecureSkipVerify)
+        }
+		var opts []proxy.DialerOptionFunc
+		opts = append(opts, proxy.WithALPNDialer())
+		dialer := proxy.DialerFromEnvironment(tc.Config.WebProxyAddr, opts...)
+		return dialer.Dial("tcp", tc.Config.WebProxyAddr, sshConfig)
 	}
 	client, err := ssh.Dial("tcp", tc.Config.SSHProxyAddr, sshConfig)
 	if err != nil {

programmerq added bug c-ec Internal Customer Reference labels Aug 31, 2021

russjones assigned atburke Jan 29, 2022

pschisa added c-sq Internal Customer Reference c-ct Internal Customer Reference labels Jan 31, 2022

atburke mentioned this issue Feb 7, 2022

Respect HTTP_PROXY/HTTPS_PROXY #10209

Merged

atburke closed this as completed in #10209 Mar 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tsh ignores HTTPS_PROXY #8108

tsh ignores HTTPS_PROXY #8108

programmerq commented Aug 31, 2021

russjones commented Jan 29, 2022

johns-carta commented Feb 9, 2022

russjones commented Feb 9, 2022

johns-carta commented Feb 9, 2022

johns-carta commented Feb 10, 2022

tsh ignores HTTPS_PROXY #8108

tsh ignores HTTPS_PROXY #8108

Comments

programmerq commented Aug 31, 2021

Description

Reproduction Steps

Server Details

Client Details

Debug Logs

russjones commented Jan 29, 2022

johns-carta commented Feb 9, 2022

russjones commented Feb 9, 2022

johns-carta commented Feb 9, 2022

johns-carta commented Feb 10, 2022