Skip to content

Commit

Permalink
Wrap kubectl wait with coreutils timeout
Browse files Browse the repository at this point in the history
We discovered that kubectl wait will actually timeout only after
timeout*number of objects being waited on. So if you wait on 3 pods
specifying a timeout of 10m, it will actually wait 30m.

`kubectl sleep --timeout -1s` will wait a week, so plausibly "forever".

kubernetes-sigs#158
kubernetes/kubectl#754
  • Loading branch information
Ben Moss committed Mar 31, 2020
1 parent 2af337d commit 279b5e7
Show file tree
Hide file tree
Showing 3 changed files with 6 additions and 6 deletions.
4 changes: 2 additions & 2 deletions gce/hack-run-e2e.sh
Original file line number Diff line number Diff line change
Expand Up @@ -37,14 +37,14 @@ kubectl create -f ${SCRIPT_ROOT}/${PREPULL_FILE}
# Wait a while for the test images to be pulled onto the nodes. In empirical
# testing it could take up to 30 minutes to finish pulling all the test
# containers on a node.
kubectl wait --for=condition=ready pod -l prepull-test-images=e2e --timeout ${PREPULL_TIMEOUT:-30m}
timeout ${PREPULL_TIMEOUT:-30m} kubectl wait --for=condition=ready pod -l prepull-test-images=e2e --timeout -1s
# Check the status of the pods.
kubectl get pods -o wide
kubectl describe pods
# Delete the pods anyway since pre-pulling is best-effort
kubectl delete -f ${SCRIPT_ROOT}/${PREPULL_FILE}
# Wait a few more minutes for the pod to be cleaned up.
kubectl wait --for=delete pod -l prepull-test-images=e2e --timeout 3m
timeout 3m kubectl wait --for=delete pod -l prepull-test-images=e2e --timeout -1s

# Download and set the list of test image repositories to use.
curl \
Expand Down
4 changes: 2 additions & 2 deletions gce/load-test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@ set -o xtrace
SCRIPT_ROOT=$(cd `dirname $0` && pwd)
kubectl create -f ${SCRIPT_ROOT}/loadtest-prepull.yaml
# Wait a while for the test images to be pulled onto the nodes.
kubectl wait --for=condition=ready pod -l prepull-test-images=loadtest --timeout ${PREPULL_TIMEOUT:-10m}
timeout ${PREPULL_TIMEOUT:-10m} kubectl wait --for=condition=ready pod -l prepull-test-images=loadtest --timeout -1s
# Check the status of the pods.
kubectl get pods -o wide
kubectl describe pods
# Delete the pods anyway since pre-pulling is best-effort
kubectl delete -f ${SCRIPT_ROOT}/loadtest-prepull.yaml
# Wait a few more minutes for the pod to be cleaned up.
kubectl wait --for=delete pod -l prepull-test-images=loadtest --timeout 3m
timeout 3m kubectl wait --for=delete pod -l prepull-test-images=loadtest --timeout -1s

$GOPATH/src/k8s.io/perf-tests/run-e2e.sh $@
4 changes: 2 additions & 2 deletions gce/run-e2e.sh
Original file line number Diff line number Diff line change
Expand Up @@ -37,14 +37,14 @@ kubectl create -f ${SCRIPT_ROOT}/${PREPULL_FILE}
# Wait a while for the test images to be pulled onto the nodes. In empirical
# testing it could take up to 30 minutes to finish pulling all the test
# containers on a node.
kubectl wait --for=condition=ready pod -l prepull-test-images=e2e --timeout ${PREPULL_TIMEOUT:-30m}
timeout ${PREPULL_TIMEOUT:-30m} kubectl wait --for=condition=ready pod -l prepull-test-images=e2e --timeout -1s
# Check the status of the pods.
kubectl get pods -o wide
kubectl describe pods
# Delete the pods anyway since pre-pulling is best-effort
kubectl delete -f ${SCRIPT_ROOT}/${PREPULL_FILE}
# Wait a few more minutes for the pod to be cleaned up.
kubectl wait --for=delete pod -l prepull-test-images=e2e --timeout 3m
timeout 3m kubectl wait --for=delete pod -l prepull-test-images=e2e --timeout -1s

# Download and set the list of test image repositories to use.
curl \
Expand Down

0 comments on commit 279b5e7

Please sign in to comment.