Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

6.1.0 dialing teleport.cluster.local, works anyway #6308

Closed
ThisGuyCodes opened this issue Apr 5, 2021 · 4 comments · Fixed by #6471
Closed

6.1.0 dialing teleport.cluster.local, works anyway #6308

ThisGuyCodes opened this issue Apr 5, 2021 · 4 comments · Fixed by #6471
Assignees
@Joerger
Labels
bug
Milestone
6.2 "Buffalo"

Comments

@ThisGuyCodes
Copy link

Description

What happened:
Logs have a TLS error about teleport.cluster.local

What you expected to happen:
No TLS errors about teleport.cluster.local

Reproduction Steps

As minimally and precisely as possible, describe step-by-step how to reproduce the problem.

  1. Install 6.1.0-beta1
  2. use the below config yaml
  3. run with teleport start --config=config.yaml

Server Details

  • Teleport version (run teleport version): Teleport v6.1.0-beta.1 git:v6.1.0-beta.1-0-g693576f98 go1.15.5
  • Server OS (e.g. from /etc/os-release): Ubuntu 20.10
  • Where are you running Teleport? (e.g. AWS, GCP, Dedicated Hardware): dedicated hardware

Client Details

N/A

Debug Logs

Please include or attach debug logs, when appropriate. Obfuscate sensitive information!

  • Start Teleport with --debug flag (teleport --debug)
  • Run tsh with --debug flag (tsh --debug)
DEBU [SQLITE]    Connected to: file:/var/lib/teleport/proc/sqlite.db?_busy_timeout=10000&_sync=OFF, poll stream period: 1s lite/lite.go:172
DEBU [SQLITE]    Synchronous: 0, busy timeout: 10000 lite/lite.go:217
DEBU [KEYGEN]    SSH cert authority started with no keys pre-compute. native/native.go:103
DEBU [PROC:1]    Adding service to supervisor. service:register.app service/supervisor.go:184
DEBU [PROC:1]    Adding service to supervisor. service:apps.start service/supervisor.go:184
DEBU [PROC:1]    Adding service to supervisor. service:apps.stop service/supervisor.go:184
DEBU [PROC:1]    Adding service to supervisor. service:common.rotate service/supervisor.go:184
DEBU [PROC:1]    No signal pipe to import, must be first Teleport process. service/service.go:781
DEBU [PROC:1]    Service has started. service:apps.start service/supervisor.go:245
DEBU [PROC:1]    Service has started. service:register.app service/supervisor.go:245
DEBU [PROC:1]    Service has started. service:apps.stop service/supervisor.go:245
DEBU [PROC:1]    Service has started. service:common.rotate service/supervisor.go:245
DEBU [PROC:1]    Connected state: never updated. service/connect.go:100
INFO [PROC:1]    Connecting to the cluster REDACTED with TLS client certificate. service/connect.go:129
DEBU [PROC:1]    Attempting to connect to Auth Server directly. service/connect.go:798
WARN [PROC:1]    Failed to connect to Auth Server directly. error:[
ERROR REPORT:
Original Error: *trace.ConnectionProblemError Get "https://teleport.cluster.local/v2/domain": remote error: tls: internal error
Stack Trace:
        /go/src/github.com/gravitational/teleport/lib/httplib/httplib.go:127 github.com/gravitational/teleport/lib/httplib.ConvertResponse
        /go/src/github.com/gravitational/teleport/lib/auth/clt.go:312 github.com/gravitational/teleport/lib/auth.(*Client).Get
        /go/src/github.com/gravitational/teleport/lib/auth/clt.go:403 github.com/gravitational/teleport/lib/auth.(*Client).GetDomainName
        /go/src/github.com/gravitational/teleport/lib/auth/clt.go:2215 github.com/gravitational/teleport/lib/auth.(*Client).GetLocalClusterName
        /go/src/github.com/gravitational/teleport/lib/service/connect.go:799 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClient
        /go/src/github.com/gravitational/teleport/lib/service/connect.go:130 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connect
        /go/src/github.com/gravitational/teleport/lib/service/connect.go:79 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connectToAuthService
        /go/src/github.com/gravitational/teleport/lib/service/connect.go:49 github.com/gravitational/teleport/lib/service.(*TeleportProcess).reconnectToAuthService
        /go/src/github.com/gravitational/teleport/lib/service/service.go:1858 github.com/gravitational/teleport/lib/service.(*TeleportProcess).registerWithAuthServer.func1
        /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:457 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
        /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:246 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
        /opt/go/src/runtime/asm_amd64.s:1374 runtime.goexit
User Message: Get "https://teleport.cluster.local/v2/domain": remote error: tls: internal error] service/connect.go:801
DEBU [PROC:1]    Attempting to connect to Auth Server through tunnel. proxy-addr:REDACTED:3024 service/connect.go:819
DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:207
DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:207
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:122
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:122
DEBU [PROC:1]    Connected to Auth Server through tunnel. proxy-addr:REDACTED:3024 service/connect.go:825
DEBU [PROC:1]    Connected client: Identity(App, cert(f9deabc4-c92e-4877-93a9-e9d966b1a119.REDACTED issued by REDACTED:70055602065070603071210055665687550479),trust root(REDACTED:70055602065070603071210055665687550479)) service/connect.go:83
DEBU [PROC:1]    Connected server: Identity(App, cert(f9deabc4-c92e-4877-93a9-e9d966b1a119.REDACTED issued by REDACTED:70055602065070603071210055665687550479),trust root(REDACTED:70055602065070603071210055665687550479)) service/connect.go:84
DEBU [PROC:1]    Adding service to supervisor. service:auth.client.app service/supervisor.go:184
DEBU [PROC:1]    Broadcasting event. event:AppsIdentity service/supervisor.go:333
DEBU [PROC:1]    Service is completed and removed. service:register.app service/supervisor.go:222
DEBU [PROC:1]    Service has started. service:auth.client.app service/supervisor.go:245
DEBU [APP:SERVI] Received event "AppsIdentity". service/service.go:2912
DEBU [PROC:1]    Creating sqlite backend for [app:service:1]. service/service.go:1469
DEBU [SQLITE]    Connected to: file:/var/lib/teleport/cache/app:service:1/sqlite.db?_busy_timeout=10000&_sync=OFF, poll stream period: 100ms lite/lite.go:172
DEBU [SQLITE]    Synchronous: 0, busy timeout: 10000 lite/lite.go:217
WARN [APP:SERVI] Re-init the cache on error: watcher is closed. cache/cache.go:675
WARN [APP:SERVI] Cache "apps" first init failed, continuing re-init attempts in background. error:[
ERROR REPORT:
Original Error: *trace.ConnectionProblemError watcher is closed
Stack Trace:
        /go/src/github.com/gravitational/teleport/lib/cache/cache.go:811 github.com/gravitational/teleport/lib/cache.(*Cache).fetchAndWatch
        /go/src/github.com/gravitational/teleport/lib/cache/cache.go:669 github.com/gravitational/teleport/lib/cache.(*Cache).update
        /opt/go/src/runtime/asm_amd64.s:1374 runtime.goexit
User Message: watcher is closed] cache/cache.go:624
DEBU [APP:SERVI] Reloading Linear(attempt=0, duration=0s). cache/cache.go:681
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log. service/service.go:1895
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload. service/service.go:1895
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/sessions. service/service.go:1895
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/sessions/default. service/service.go:1895
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log. service/service.go:1895
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload. service/service.go:1895
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/streaming. service/service.go:1895
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/streaming/default. service/service.go:1895
DEBU [PROC:1]    Adding service to supervisor. service:uploader.service service/supervisor.go:184
DEBU [PROC:1]    Adding service to supervisor. service:uploader.shutdown service/supervisor.go:184
DEBU [PROC:1]    Service has started. service:uploader.service service/supervisor.go:245
DEBU [PROC:1]    Adding service to supervisor. service:fileuploader.service service/supervisor.go:184
DEBU [PROC:1]    Adding service to supervisor. service:fileuploader.shutdown service/supervisor.go:184
DEBU [PROC:1]    Service has started. service:uploader.shutdown service/supervisor.go:245
DEBU [PROC:1]    Service has started. service:fileuploader.shutdown service/supervisor.go:245
DEBU [PROC:1]    Service has started. service:fileuploader.service service/supervisor.go:245
WARN [APP:SERVI] Re-init the cache on error: watcher is closed. cache/cache.go:675
DEBU [APP:SERVI] Reloading Linear(attempt=1, duration=1s). cache/cache.go:681
DEBU [APP:SERVI] Starting App heartbeat with announce period: 1m0s, keep-alive period 7m18.479572894s, poll period: 5s srv/heartbeat.go:143
DEBU [PROXY:AGE] Starting agent pool f9deabc4-c92e-4877-93a9-e9d966b1a119.REDACTED.REDACTED... cluster:REDACTED reversetunnel/agentpool.go:165
DEBU [PROC:1]    Broadcasting event. event:AppsReady service/supervisor.go:333
DEBU [PROC:1]    Broadcasting mapped event. in:AppsReady out:EventMapping(in=[AppsReady], out=TeleportReady) service/supervisor.go:358
INFO [APP:SERVI] All applications successfully started. service/service.go:3073
INFO [PROC:1]    The new service has started successfully. Starting syncing rotation status with period 10m0s. service/connect.go:432
DEBU [PROXY:AGE] Seeking: {Addr:REDACTED:3024 AddrNetwork:tcp Path:}. cluster:REDACTED reversetunnel/agentpool.go:195
DEBU [PROXY:AGE] Adding agent(leaseID=1,state=connecting) -> REDACTED:REDACTED:3024. cluster:REDACTED reversetunnel/agentpool.go:281
DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:207
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:122
INFO [APP:SERVI] Connected. addr:192.168.4.34:32928 remote-addr:REDACTED:3024 leaseID:1 target:REDACTED:3024 reversetunnel/agent.go:349
DEBU [APP:SERVI] Agent connected to proxy: [2df041f7-c7fa-43dc-9b2a-28075cf5a821.REDACTED 2df041f7-c7fa-43dc-9b2a-28075cf5a821 ip-172-16-51-156.REDACTED ip-172-16-51-156 localhost 127.0.0.1 ::1 REDACTED remote.kube.proxy.teleport.cluster.local]. leaseID:1 target:REDACTED:3024 reversetunnel/agent.go:354
DEBU [APP:SERVI] Changing state connecting -> connected. leaseID:1 target:REDACTED:3024 reversetunnel/agent.go:192
DEBU [APP:SERVI] Discovery request channel opened: teleport-discovery. leaseID:1 target:REDACTED:3024 reversetunnel/agent.go:465
DEBU [APP:SERVI] handleDiscovery requests channel. leaseID:1 target:REDACTED:3024 reversetunnel/agent.go:483
WARN [APP:SERVI] Re-init the cache on error: watcher is closed. cache/cache.go:675
DEBU [APP:SERVI] Reloading Linear(attempt=2, duration=2s). cache/cache.go:681
WARN [APP:SERVI] Re-init the cache on error: watcher is closed. cache/cache.go:675
DEBU [APP:SERVI] Reloading Linear(attempt=3, duration=3s). cache/cache.go:681
WARN [APP:SERVI] Re-init the cache on error: watcher is closed. cache/cache.go:675
DEBU [APP:SERVI] Reloading Linear(attempt=4, duration=4s). cache/cache.go:681
WARN [APP:SERVI] Re-init the cache on error: watcher is closed. cache/cache.go:675
DEBU [APP:SERVI] Reloading Linear(attempt=5, duration=5s). cache/cache.go:681
WARN [APP:SERVI] Re-init the cache on error: watcher is closed. cache/cache.go:675
DEBU [APP:SERVI] Reloading Linear(attempt=6, duration=6s). cache/cache.go:681
WARN [APP:SERVI] Re-init the cache on error: watcher is closed. cache/cache.go:675
DEBU [APP:SERVI] Reloading Linear(attempt=7, duration=7s). cache/cache.go:681
WARN [APP:SERVI] Re-init the cache on error: watcher is closed. cache/cache.go:675
DEBU [APP:SERVI] Reloading Linear(attempt=8, duration=8s). cache/cache.go:681
DEBU [APP:SERVI] Transport request: teleport-transport. leaseID:1 target:REDACTED:3024 reversetunnel/agent.go:439
DEBU [APP:SERVI] Received out-of-band proxy transport request for @local-node [f9deabc4-c92e-4877-93a9-e9d966b1a119.REDACTED]. leaseID:1 target:REDACTED:3024 reversetunnel/transport.go:151
DEBU [APP:SERVI] Handing off connection to a local "app" service. leaseID:1 target:REDACTED:3024 reversetunnel/transport.go:226
DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:207
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:122
DEBU [AUTH]      ClientCertPool -> cert(REDACTED issued by REDACTED:70055602065070603071210055665687550479) auth/middleware.go:593
DEBU [AUTH]      ClientCertPool -> cert(REDACTED issued by REDACTED:250595971901550293603279960258630168876) auth/middleware.go:593
DEBU             Skipping login 43fbc691-63b0-4af5-b0c6-1607ddc4a7a8, not a valid Unix login. services/role.go:412
DEBU             Skipping login 43fbc691-63b0-4af5-b0c6-1607ddc4a7a8, not a valid Unix login. services/role.go:412
DEBU [APP:SERVI] Using async streamer for session b05e12ea-6b28-49df-9f66-d34921c0096b. app/session.go:173
ERRO             Error forwarding to /api/2.0/status, err: x509: certificate signed by unknown authority forward/fwd.go:179
WARN [APP:SERVI] Re-init the cache on error: watcher is closed. cache/cache.go:675
DEBU [APP:SERVI] Reloading Linear(attempt=9, duration=9s). cache/cache.go:681
INFO [PROC:1]    Got signal "interrupt", exiting immediately. service/signals.go:86
DEBU [APP:SERVI] Cache is closing, returning from update loop. cache/cache.go:659
DEBU [PROC:1]    Broadcasting event. event:TeleportExit service/supervisor.go:333
WARN [PROC:1]    Sync rotation state cycle failed: watcher has disconnected, going to retry after 10s. service/connect.go:444
DEBU [PROC:1]    Service is completed and removed. service:common.rotate service/supervisor.go:222

teleport config

teleport:
  auth_token: "REDACTED"
  auth_servers:
    - REDACTED:443
auth_service:
  enabled: false
ssh_service:
  enabled: false
proxy_service:
  enabled: false
app_service:
  enabled: true
  apps:
    - name: "ny-nvr-old"
      uri: "https://192.168.40.200:7443"

While I'm here: is there a way to allow an insecure certificate for an app without making the auth server connection insecure too? (e.g. appliances with self-signed certs)
@webvictim
Copy link
Contributor

While I'm here: is there a way to allow an insecure certificate for an app without making the auth server connection insecure too? (e.g. appliances with self-signed certs)

@ThisGuyCodes Set insecure_skip_verify: true in your app config - see https://goteleport.com/docs/application-access/reference/#configuration for more information.

This was referenced Apr 13, 2021
Merged
Merged
@Joerger
Copy link
Contributor

Joerger commented Apr 14, 2021

The error you're seeing here is just teleport trying to connect directly and failing, which is expected, before successfully connecting over proxy. I made PRs to reduce this to a small debug log to reduce confusion, thanks for pointing this out.

@ThisGuyCodes
Copy link
Author

All awesome, thanks!

@Joerger Joerger mentioned this issue Apr 15, 2021
Merged
@awly awly added this to the 6.2 "Buffalo" milestone Apr 19, 2021
@webvictim
Copy link
Contributor

Possibly related:
#6496
#6448

@benarent benarent mentioned this issue Oct 7, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Joerger Joerger

Labels
bug
Projects
None yet
Milestone
6.2 "Buffalo"
Development

Successfully merging a pull request may close this issue.

Improve process connection error handling and logging gravitational/teleport
4 participants
@ThisGuyCodes @webvictim @awly @Joerger