kubernetes_service does not honour HTTP_PROXY for outbound requests to clusters

[webvictim]

Expected behavior

Teleport's kubernetes_service should use the HTTP_PROXY or HTTPS_PROXY set in the Teleport's process's environment when accessing remote Kubernetes clusters.

Current behavior

The supplied HTTP_PROXY seems to only be used when performing the initial SelfSubjectAccessReview as part of the Teleport process' startup. Regular kubectl commands do not go through the proxy and cause a connection error to be displayed when outbound traffic via the default gateway is blocked.

Logs

Teleport server:

Aug 16 10:36:19 ubuntu teleport[10452]: 2023-08-16T10:36:19-03:00 INFO [CA]        Generating TLS certificate 1.3.9999.1.15=#13046e6f6e65,1.3.9999.1.9=#13033a3a31,1.3.9999.1.7=#13067562756e7475,1.3.9999.1.3=#1303656b73,1.3.9999.1.2=#130e73797374656d3a6d617374657273,CN=gus,OU=usage:kube,O=access+O=editor+O=auditor,POSTALCODE={\"aws_role_arns\":null\,\"azure_identities\":null\,\"db_names\":null\,\"db_roles\":null\,\"db_users\":null\,\"gcp_service_accounts\":null\,\"kubernetes_groups\":null\,\"kubernetes_users\":null\,\"logins\":null\,\"windows_logins\":null},STREET=ubuntu,L=-teleport-internal-join+L=-teleport-nologin-e31467d3-7cfe-46a2-b40d-a32a503f1ce9 dns_names:[] key_usage:5 not_after:2023-08-17 01:36:19.40728909 +0000 UTC tlsca/ca.go:1111
Aug 16 10:36:19 ubuntu teleport[10452]: 2023-08-16T10:36:19-03:00 ERRO [KUBERNETE] Error forwarding to https://3FB75662E5F6A8FFBD1B44FBC603820D.gr7.us-east-2.eks.amazonaws.com:443/version?timeout=32s, err: dial tcp 3.131.140.41:443: connect: connection refused pid:10452.1 forward/fwd.go:176
Aug 16 10:36:19 ubuntu teleport[10452]: 2023-08-16T10:36:19-03:00 INFO [AUDIT]     kube.request addr.remote:[::1]:43200 cluster_name:ubuntu code:T3009I ei:0 event:kube.request kubernetes_cluster:eks kubernetes_groups:[system:masters system:authenticated] kubernetes_users:[gus] login:gus namespace:default proto:kube request_path:/version response_code:500 server_id:9e5b68f3-5939-457c-82e1-4385d19746f5 time:2023-08-16T13:36:19.468Z uid:878f93aa-bccd-45f8-9ac7-0e1f5f3db3db user:gus verb:GET events/emitter.go:265
Aug 16 10:36:19 ubuntu teleport[10452]: 2023-08-16T10:36:19-03:00 INFO [PROXY:PRO] Round trip: GET https://kube-teleport-proxy-alpn.teleport.cluster.local/version?timeout=32s, code: 500, duration: 57.691289ms tls:version: 304, tls:resume:false, tls:csuite:1301, tls:server:kube-teleport-proxy-alpn.127.0.0.1.nip.io pid:10452.1 forward/fwd.go:182

Client:

gus@apollo:~ % tsh login --proxy=127.0.0.1.nip.io:443 --insecure
Enter password for Teleport user gus:
WARNING: You are using insecure connection to Teleport proxy https://127.0.0.1.nip.io:443
Enter an OTP code from a device:
> Profile URL:        https://127.0.0.1.nip.io:443
  Logged in as:       gus
  Cluster:            ubuntu
  Roles:              access, auditor, editor
  Kubernetes:         enabled
  Kubernetes groups:  system:masters
  Valid until:        2023-08-16 22:34:16 -0300 ADT [valid for 12h0m0s]
  Extensions:         login-ip, permit-agent-forwarding, permit-port-forwarding, permit-pty, private-key-policy

Did you know? Teleport Connect offers the power of tsh in a desktop app.
Learn more at https://goteleport.com/docs/connect-your-client/teleport-connect/

gus@apollo:~ % tsh kube ls
Kube Cluster Name Labels Selected
----------------- ------ --------
eks

gus@apollo:~ % tsh kube login eks --insecure
Logged into Kubernetes cluster "eks". Try 'kubectl version' to test the connection.

gus@apollo:~ % tsh kubectl get nodes -v=7
I0816 10:43:51.532032   60189 loader.go:373] Config loaded from file:  /Users/gus/.kube/config
I0816 10:43:51.535463   60189 round_trippers.go:463] GET https://127.0.0.1.nip.io:443/api/v1/nodes?limit=500
I0816 10:43:51.535470   60189 round_trippers.go:469] Request Headers:
I0816 10:43:51.535475   60189 round_trippers.go:473]     Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json
I0816 10:43:51.535479   60189 round_trippers.go:473]     User-Agent: tsh/v0.0.0 (darwin/arm64) kubernetes/$Format
I0816 10:43:51.630809   60189 round_trippers.go:574] Response Status: 500 Internal Server Error in 95 milliseconds
I0816 10:43:51.631033   60189 helpers.go:246] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "dial tcp 3.138.166.42:443: connect: connection refused",
  "reason": "InternalError",
  "code": 500
}]
Error from server (InternalError): dial tcp 3.138.166.42:443: connect: connection refused

Outbound traffic was blocked from the VM using sudo iptables -A OUTPUT -p tcp -m tcp --dport 443 -j REJECT

Charles proxy:

Config files

/etc/teleport.yaml:

version: v3
teleport:
  nodename: ubuntu
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: INFO
    format:
      output: text
  ca_pin: ""
  diag_addr: ""
  # prevents spurious outbound connections to 0.0.0.0 via proxy on startup
  #auth_server: 127.0.0.1:3025
auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  proxy_listener_mode: multiplex
  authentication:
    second_factor: on
    webauthn:
      rp_id: 127.0.0.1.nip.io
ssh_service:
  enabled: "yes"
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
kubernetes_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3027
  kubeconfig_file: /etc/teleport/kubeconfig.yaml
proxy_service:
  enabled: "yes"
  https_keypairs: []
  https_keypairs_reload_interval: 0s
  acme: {}
  web_listen_addr: "0.0.0.0:443"
  public_addr: "127.0.0.1.nip.io:443"

/etc/default/teleport:

AWS_ACCESS_KEY_ID="<redacted>"
AWS_SECRET_ACCESS_KEY="<redacted>"
AWS_SESSION_TOKEN="<redacted>"
# charles proxy on host
HTTP_PROXY=http://198.19.249.3:8888
HTTPS_PROXY=http://198.19.249.3:8888
NO_PROXY=localhost,127.0.0.1,0.0.0.0,::,teleport.cluster.local,kube-teleport-proxy-alpn.teleport.cluster.local

Teleport process environment:

gus@ubuntu:/Users/gus$ sudo cat /proc/$(pidof teleport)/environ | xargs -0 -L1
LANG=en_US.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PIDFILE=/run/teleport.pid
INVOCATION_ID=8b8e5d0664e448f1bf5cfd581019160a
JOURNAL_STREAM=8:99027
SYSTEMD_EXEC_PID=10758
AWS_ACCESS_KEY_ID=<redacted>
AWS_SECRET_ACCESS_KEY=<redacted>
AWS_SESSION_TOKEN=<redacted>
HTTP_PROXY=http://198.19.249.3:8888
HTTPS_PROXY=http://198.19.249.3:8888
NO_PROXY=localhost,127.0.0.1,0.0.0.0,::,teleport.cluster.local,kube-teleport-proxy-alpn.teleport.cluster.local

/etc/teleport/kubeconfig.yaml:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <redacted>
    server: https://3FB75662E5F6A8FFBD1B44FBC603820D.gr7.us-east-2.eks.amazonaws.com
  name: eks
contexts:
- context:
    cluster: eks
    user: eks
  name: eks
current-context: eks
kind: Config
preferences: {}
users:
- name: eks
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      - us-east-2
      - eks
      - get-token
      - --cluster-name
      - <redacted>
      - --output
      - json
      command: aws

Bug details

• Teleport version: Teleport v13.2.3 git:v13.2.3-0-g4785e70 go1.20

This issue was reported by a customer and reproduced by me.

[tigrato]

Reopening because kubectl exec sessions aren't working.
Our SPDY upgrader doesn't support proxies and it won't be trivial to support them