Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP - speculative revert of 1599 #1647

Closed
wants to merge 1 commit into from
Closed

WIP - speculative revert of 1599 #1647

wants to merge 1 commit into from

Conversation

liggitt
Copy link
Contributor

@liggitt liggitt commented Jun 2, 2020 •
edited
Loading

Just to exercise CI... seeing close to 100% failure rates on kubernetes/kubernetes#91579 on master

the containerd diff brought in fifo/logging/process killing changes, which might be related to kubernetes/kubernetes#91579

cc @BenTheElder

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 2, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: liggitt
To complete the pull request process, please assign munnerz
You can assign the PR to them by writing /assign @munnerz in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot requested review from krzyzacy and munnerz June 2, 2020 22:05
@k8s-ci-robot k8s-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Jun 2, 2020
@liggitt liggitt changed the title TMP - speculative revert of 1599 WIP - speculative revert of 1599 Jun 2, 2020
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 2, 2020
@liggitt
Copy link
Contributor Author

liggitt commented Jun 2, 2020

/hold

not really intending to merge this unless it magically cleans up all our CI signal

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 2, 2020
@liggitt
Copy link
Contributor Author

liggitt commented Jun 2, 2020

/test all

@liggitt
Copy link
Contributor Author

liggitt commented Jun 2, 2020 •
edited
Loading

failed, but with timeout startup errors (resource exhaustion?), not the predominant pod kill errors?

/retest

@liggitt
Copy link
Contributor Author

liggitt commented Jun 3, 2020

/test pull-kind-e2e-kubernetes

@liggitt
Copy link
Contributor Author

liggitt commented Jun 3, 2020 •
edited
Loading

Results:

  • 1 run with an unrelated failure
  • 3 green runs

/test pull-kind-e2e-kubernetes

@BenTheElder
Copy link
Member

FYI @dims this mostly seems to be a containerd rollback. The kind base image tag has Sha in it,

https://github.com/kubernetes-sigs/kind/compare/21e528f5..fd490400#diff-065f169938b73c43248f266b4fc6291bL26

https://github.com/containerd/containerd/compare/1e902b2d..449e9269

:/

@liggitt
Copy link
Contributor Author

liggitt commented Jun 3, 2020

https://github.com/kubernetes-sigs/kind/compare/21e528f5..fd490400#diff-424162915c2ab4ab7f1b3f6dff73946e looks interesting, especially with the error we're seeing in on the failed pods:

kind-worker2 containerd[128]: time="2020-06-02T00:55:36.149486402Z" level=error msg="add cg to OOM monitor" error="open /sys/fs/cgroup/memory/kubepods/burstable/pod553b0f48-8dbd-4ff6-8e4c-289a0a9b7f5c/583c17764ed521bc6e30482e10faff6f0cda6d5351cfdb9b942b905df0eec592/memory.oom_control: no such file or directory"

@liggitt
Copy link
Contributor Author

liggitt commented Jun 3, 2020

oh, that just got merged two days ago in #1639 ... it isn't the culprit

@BenTheElder
Copy link
Member

/test all

@liggitt
Copy link
Contributor Author

liggitt commented Jun 3, 2020

/test pull-kind-e2e-kubernetes

@liggitt
Copy link
Contributor Author

liggitt commented Jun 3, 2020

6 e2e runs without pod kill issues ...

@BenTheElder BenTheElder mentioned this pull request Jun 3, 2020
Merged
@BenTheElder
Copy link
Member

more runs coming in #1649 w/ new images

@liggitt
Copy link
Contributor Author

liggitt commented Jun 3, 2020

Closing in favor of #1649

@liggitt liggitt closed this Jun 3, 2020
@liggitt liggitt deleted the containerd branch June 11, 2020 02:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@krzyzacy krzyzacy Awaiting requested review from krzyzacy

@munnerz munnerz Awaiting requested review from munnerz

Assignees
No one assigned
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@liggitt @k8s-ci-robot @BenTheElder