garden dev --hot='*' loses track of Kubernetes authorization

[jmeickle]

Bug

Current Behavior

Initially, hot reloading works fine. After some time it starts reliably failing on save, with k8s auth errors:

⏳ Processing...
ℹ web                       → Hot reloading...
✔ web                       → Hot reloading... → Done (took 822 ms)

🕑 Waiting for code changes...

⏳ Processing...
ℹ web                       → Hot reloading...
✔ web                       → Hot reloading... → Done (took 948 ms)

🕑 Waiting for code changes...

⏳ Processing...
ℹ web                       → Hot reloading...
✖ web                       → Hot reloading...

Failed hot-reloading service web. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
StatusCodeError from Kubernetes API - 401 - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


🕑 Waiting for code changes...

⏳ Processing...
ℹ web                       → Hot reloading...
✖ web                       → Hot reloading...

Failed hot-reloading service web. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
StatusCodeError from Kubernetes API - 401 - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


🕑 Waiting for code changes...

⏳ Processing...
ℹ web                       → Hot reloading...
✖ web                       → Hot reloading...

Failed hot-reloading service web. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
StatusCodeError from Kubernetes API - 401 - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

ctrl-c'ing the process and restarting it fixes it, so I believe this is a token auth issue.

Expected behavior

No auth errors on hot reloads, and retries if there are any errors.

Additional context

We're using a kubectl exec plugin, specifically heptio-authenticator-aws, to fetch time-limited authentication tokens. This is likely what's breaking Garden (we've seen it before with other applications, since many don't implement logic to re-fetch creds after a period of time or on error). Here's what our config looks like:

- name: stg-admin
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - CLUSTER_URL
      - -r
      - arn:aws:iam::ACCT_ID:role/IAM_ROLE_NAME
      command: heptio-authenticator-aws
      env: null

Your environment

[eronarn@ip-192-168-10-243 qf]$ garden version && kubectl version && docker version
0.10.2
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.7", GitCommit:"4683545293d792934a7a7e12f2cc47d20b2dd01b", GitTreeState:"clean", BuildDate:"2019-06-06T01:39:30Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Client: Docker Engine - Community
 Version:           19.03.0-rc2
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        f97efcc
 Built:             Wed Jun  5 01:37:53 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.0-rc2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       f97efcc
  Built:            Wed Jun  5 01:42:10 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[edvald]

This turns out to be an issue in the Kubernetes API library we use, but we should be able to catch and work around it. I'll take a look.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.