DevLoop unexpected EOF error

[ArthurName]

Expected behavior

No EOF error.

Actual behavior

I get this error when running Skaffold:

[foo-backend] error: unexpected EOF
WARN[0659] exit status 1                                 subtask=-1 task=DevLoop

It happens exactly when the last pod reaches AGE 600 seconds, as reported by K9s, i.e. exactly 600 seconds after the environment is 100 % up and running.

This is regardless of Skaffold mode dev or run (which is a bit confusing, since I thought DevLoop was related to dev mode only; correct me if I'm wrong).

Furthermore, also in both cases, after the error, the output tail stops presenting.

The same error message is mentioned by user Mrhoho in issue 6072 but I'm not too sure it's about open files or file watchers; again this error is reported even in run mode, where as I understand file watching is not enabled.

1. Why EOF?
2. Why after 600 seconds?
3. Which file has reached EOF? What can I do to reveal which file it is?
4. Which process exits with status 1 here?
5. How is DevLoop involved when executing skaffold run?

Information

• Skaffold version: v1.39.0
• Operating system: Ubuntu 22.04.1 LTS
• Installed via: https://storage.googleapis.com/skaffold/releases/v1.39.0/skaffold-linux-amd64
• Contents of skaffold.yaml:

apiVersion: skaffold/v2beta21
kind: Config
metadata:
  name: foo-namespaces
profiles:
- name: dev
  deploy:
    kubeContext: foo-dev
    statusCheckDeadlineSeconds: 600
    kubectl:
      manifests:
        - kubernetes/base/foo/foo-namespaces.yaml

---

apiVersion: skaffold/v2beta21
kind: Config
requires:
  - configs: 
      - "foo-namespaces"
    activeProfiles: 
      - name: dev
        activatedBy:
          - "dev"
  - configs:
      - multus-cni 
    path: skaffold-multus-cni.yaml
    activeProfiles: 
    - name: dev-all-services
      activatedBy:
      - "dev"
  - configs:
      - argo
    path: skaffold-argo.yaml
    activeProfiles: 
    - name: dev-all-services
      activatedBy:
      - "dev"
  - configs:
      - metallb 
    path: skaffold-metallb.yaml
    activeProfiles: 
    - name: dev-all-services
      activatedBy:
      - "dev"
  - configs:
      - foo
    path: skaffold-foo.yaml
metadata:
  name: foo-api-dev

Steps to reproduce the behavior

1. Either of these commands:
   a) skaffold dev -p dev -m foo-api-dev --trigger=manual --cleanup=true
   b) skaffold run -p dev -m foo-api-dev --tail
2. Wait 600 seconds.

[rstoermer]

Same error here!

[ArthurName]

Same error here!

@rstoermer Are you using k3d?

[aaron-prindle]

@rstoermer are you also using k3d here?

I haven't had a chance to investigate this more deeply (setup the repro) but the 600s is likely related to Skaffold's statusCheckDeadlineSeconds, see the docs on this value here:
https://skaffold.dev/docs/pipeline-stages/status-check/#configuring-timeout-for-status-check

Related code where the default value is set:
https://github.com/GoogleContainerTools/skaffold/blob/main/pkg/skaffold/kubernetes/status/status_check.go#L54-L55
https://github.com/GoogleContainerTools/skaffold/blob/main/pkg/skaffold/kubernetes/status/status_check.go#L66

[dregin]

I see the same behavior using Skaffold with EKS.