Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try to get more info on TestDevGracefulCancel failures #6662

Closed

Conversation

briandealwis
Copy link
Member

No description provided.

@codecov
Copy link

codecov bot commented Oct 1, 2021

Codecov Report

Merging #6662 (e4aece9) into main (290280e) will decrease coverage by 0.48%.
The diff coverage is 51.72%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #6662      +/-   ##
==========================================
- Coverage   70.48%   69.99%   -0.49%     
==========================================
  Files         515      522       +7     
  Lines       23150    23739     +589     
==========================================
+ Hits        16317    16616     +299     
- Misses       5776     6038     +262     
- Partials     1057     1085      +28     
Impacted Files Coverage Δ
cmd/skaffold/app/cmd/flags.go 89.00% <0.00%> (-1.82%) ⬇️
cmd/skaffold/skaffold.go 0.00% <ø> (ø)
pkg/diag/recommender/container_errors.go 0.00% <0.00%> (ø)
pkg/diag/validator/pod.go 1.32% <0.00%> (ø)
pkg/skaffold/build/buildpacks/logger.go 0.00% <ø> (ø)
pkg/skaffold/build/cluster/logs.go 0.00% <ø> (-16.67%) ⬇️
cmd/skaffold/app/cmd/cmd.go 70.32% <66.66%> (-0.73%) ⬇️
pkg/diag/validator/resource.go 47.05% <66.66%> (ø)
pkg/skaffold/build/docker/docker.go 86.44% <66.66%> (-2.85%) ⬇️
pkg/skaffold/build/docker/errors.go 63.33% <80.00%> (-2.23%) ⬇️
... and 77 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 26dc1b8...e4aece9. Read the comment docs.

@briandealwis briandealwis added the kokoro:force-run forces a kokoro re-run on a PR label Oct 1, 2021
@kokoro-team kokoro-team removed the kokoro:force-run forces a kokoro re-run on a PR label Oct 1, 2021
@briandealwis briandealwis added the kokoro:force-run forces a kokoro re-run on a PR label Oct 1, 2021
@kokoro-team kokoro-team removed the kokoro:force-run forces a kokoro re-run on a PR label Oct 1, 2021
@tejal29
Copy link
Contributor

tejal29 commented Oct 1, 2021

umm. The build passed this time.

@briandealwis briandealwis added the kokoro:force-run forces a kokoro re-run on a PR label Oct 2, 2021
@kokoro-team kokoro-team removed the kokoro:force-run forces a kokoro re-run on a PR label Oct 2, 2021
@briandealwis briandealwis added the kokoro:force-run forces a kokoro re-run on a PR label Oct 4, 2021
@kokoro-team kokoro-team removed the kokoro:force-run forces a kokoro re-run on a PR label Oct 4, 2021
@briandealwis
Copy link
Member Author

time="2021-10-04T14:03:59Z" level=info msg="Pods and deployments should be stable"
time="2021-10-04T14:03:59Z" level=info msg="Signalling SIGINT"
time="2021-10-04T14:03:59Z" level=info msg="Signalled SIGINT"
time="2021-10-04T14:03:59Z" level=debug msg="terminated deployment-leeroy-web-skaffold47jcn-8080 due to context cancellation" subtask=deployment/leeroy-web task=PortForward
time="2021-10-04T14:03:59Z" level=debug msg="terminated deployment-leeroy-app-skaffold47jcn-http due to context cancellation" subtask=deployment/leeroy-app task=PortForward

So it looks like the graceful shutdown isn't actually shutting down.

@briandealwis
Copy link
Member Author

This panic isn't useful since it's actually the testing instance and not the Skaffold under test.

@tejal29
Copy link
Contributor

tejal29 commented Oct 5, 2021

time="2021-10-04T14:03:59Z" level=info msg="Pods and deployments should be stable"
time="2021-10-04T14:03:59Z" level=info msg="Signalling SIGINT"
time="2021-10-04T14:03:59Z" level=info msg="Signalled SIGINT"
time="2021-10-04T14:03:59Z" level=debug msg="terminated deployment-leeroy-web-skaffold47jcn-8080 due to context cancellation" subtask=deployment/leeroy-web task=PortForward
time="2021-10-04T14:03:59Z" level=debug msg="terminated deployment-leeroy-app-skaffold47jcn-http due to context cancellation" subtask=deployment/leeroy-app task=PortForward

So it looks like the graceful shutdown isn't actually shutting down.

Sometimes, I have had to press "Cntrl-C" twice to exit devloop. I always thought it was because I am impatient.

halvards added a commit to halvards/skaffold that referenced this pull request Oct 6, 2021
Adding log statements to better understand what's happening in kokoro.
Unfortunately I haven't been able to reproduce the problem locally (on
Linux) so far.

Related: GoogleContainerTools#6424, GoogleContainerTools#6643, GoogleContainerTools#6662
halvards added a commit to halvards/skaffold that referenced this pull request Oct 7, 2021
Debugging the flaky `TestDevGracefulCancel/multi-config-microservices`
integration test showed that the kubectl port forwarder was stuck with
goroutines waiting on a channels(one per resource).

Search for `goroutine 235` and `goroutine 234` in this Kokoro log:
https://source.cloud.google.com/results/invocations/a9749ab5-8762-4319-a2be-f67c7440f7a2/targets/skaffold%2Fpresubmit/log

This change means that the forwarder also listens for context canceled.

**Related**: GoogleContainerTools#6424, GoogleContainerTools#6643, GoogleContainerTools#6662, GoogleContainerTools#6685
briandealwis pushed a commit that referenced this pull request Oct 7, 2021
Debugging the flaky `TestDevGracefulCancel/multi-config-microservices`
integration test showed that the kubectl port forwarder was stuck with
goroutines waiting on a channels(one per resource).

Search for `goroutine 235` and `goroutine 234` in this Kokoro log:
https://source.cloud.google.com/results/invocations/a9749ab5-8762-4319-a2be-f67c7440f7a2/targets/skaffold%2Fpresubmit/log

This change means that the forwarder also listens for context canceled.

**Related**: #6424, #6643, #6662, #6685
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants