Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubectl wait for prepulling GCE test containers takes too long to timeout #158

Closed
pjh opened this issue Mar 31, 2020 · 1 comment · Fixed by #159
Closed

kubectl wait for prepulling GCE test containers takes too long to timeout #158

pjh opened this issue Mar 31, 2020 · 1 comment · Fixed by #159

Comments

@pjh
Copy link
Contributor

pjh commented Mar 31, 2020

The timeout behavior for kubectl wait is unintuitive - see kubernetes/kubectl#754 (comment). The result is that when one of our prepulled test containers fails to run, run-e2e.sh for GCE tests waits 90 minutes instead of 30 minutes before starting to run the e2e tests. The test run then times out at 120 minutes before most of the tests have been run.

An alternative approach might be something like we do in the GCE smoke-test, although surely that can be made a bit cleaner.

cc @YangLu1031 @yliaog @benmoss
@benmoss
Copy link
Contributor

benmoss commented Mar 31, 2020

We can wrap it with coreutils timeout, so we still get the optimistic case as faster than going back to sleep but the pessimistic case will be sane and only wait for as long as needed. I'll submit a PR.

benmoss pushed a commit to benmoss/windows-testing that referenced this issue Mar 31, 2020
Wrap kubectl wait with coreutils timeout
279b5e7 
We discovered that kubectl wait will actually timeout only after
timeout*number of objects being waited on. So if you wait on 3 pods
specifying a timeout of 10m, it will actually wait 30m.

`kubectl sleep --timeout -1s` will wait a week, so plausibly "forever".

kubernetes-sigs#158
kubernetes/kubectl#754
@benmoss benmoss mentioned this issue Mar 31, 2020
Merged
@jsturtevant jsturtevant mentioned this issue Aug 21, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Wrap kubectl wait with coreutils timeout benmoss/windows-testing
2 participants
@benmoss @pjh