Skip to content

Commit

Permalink
test: Bump timeout of service plumbing check
Browse files Browse the repository at this point in the history
When restarting Cilium, we check a number of things to ensure it's
ready, including that the kube-dns service is correctly plumbed (in the
agent and in the datapath's maps).

This check is executed in a loop with a 5s timeout. All of the kube-dns
checks, including that one, are executed in a loop with a 4min timeout.

To check the service plumbing, we shell out twice, to retrieve the
retrieve the agent state and to dump the BPF map contents. These
shelling out can take up to a few seconds, especially when running
locally where we typically execute a kubectl exec inside an SSH command.

As a result of those commands taking a few seconds to execute, the inner
loop regularly times out at 5s. That means we retry until we get a
runtime below 5s. What could have taken 7s now sometimes takes several
10s of seconds because we have to retry. Locally, this can get even
worse and we sometimes hit the 4min timeout of the outer loop because
the inner loop never succeeds in less than 5s.

To avoid this whole mess, we can simply bump the inner loop's timeout to
10s. As per the above, this should (counterintuitively) reduce the total
runtime of the restart checks.

Signed-off-by: Paul Chaignon <[email protected]>
  • Loading branch information
pchaigno authored and qmonnet committed Jan 31, 2023
1 parent 0685982 commit a63eb25
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion test/helpers/kubectl.go
Original file line number Diff line number Diff line change
Expand Up @@ -1973,7 +1973,7 @@ func (kub *Kubectl) ValidateServicePlumbing(namespace, service string) error {
ginkgoext.By("Checking service %s plumbing in cilium pod %s: %s", fullName, ciliumPod, err)
}
return err == nil
}, &TimeoutConfig{Timeout: 5 * time.Second, Ticker: 1 * time.Second})
}, &TimeoutConfig{Timeout: 10 * time.Second, Ticker: 1 * time.Second})
if err != nil {
return err
} else if timeoutErr != nil {
Expand Down

0 comments on commit a63eb25

Please sign in to comment.