Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

e2e tests: synchronize test results #9124

Merged
merged 1 commit into from
Jan 27, 2021

Conversation

vrothberg
Copy link
Member

Use a mutex to synchronize the slice for storing tests results.
Running the e2e tests in parallel is otherwise subject to race
conditions surfacing in lost entries.

Fixes: #8358
Signed-off-by: Valentin Rothberg [email protected]

Use a mutex to synchronize the slice for storing tests results.
Running the e2e tests in parallel is otherwise subject to race
conditions surfacing in lost entries.

Fixes: containers#8358
Signed-off-by: Valentin Rothberg <[email protected]>
@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 27, 2021
@vrothberg
Copy link
Member Author

@containers/podman-maintainers PTAL

Copy link
Member

@saschagrunert saschagrunert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@openshift-ci-robot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: saschagrunert, vrothberg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [saschagrunert,vrothberg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@rhatdan
Copy link
Member

rhatdan commented Jan 27, 2021

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 27, 2021
@openshift-merge-robot openshift-merge-robot merged commit 2ff4da9 into containers:master Jan 27, 2021
@edsantiago
Copy link
Member

Followup: the new code works as expected, thank you! FWIW I'm not seeing any major timing differences between the ubuntus. Here's a quick table listing what I think are the most interesting results (at least one test took longer than 5 seconds, and at least one test had a 50% or greater difference in duration between hosts):

fedora
32
fedora
33
ubuntu
2004
ubuntu
2010
podman create pod with DNS option set and no infra should fail 0.53 0.56 1.01 5.49
podman create pod with IP address and no infra should fail 0.53 0.57 1.00 5.57
podman create pod with --add-host and no infra should fail 0.53 0.57 1.02 5.19
podman create pod with --no-hosts and no infra should fail 0.54 0.56 0.96 5.20
podman create pod with DNS search domain set and no infra should fail 0.54 0.59 1.00 5.28
podman pod start bogus pod 0.54 0.56 0.99 7.59
podman create pod with DNS server set and no infra should fail 0.54 0.58 0.98 5.06
podman create pod with no infra but portbindings should fail 0.56 0.59 0.98 5.46
podman images 0.56 0.58 6.94 0.96
podman create pod 0.61 0.71 1.16 5.56
podman create pod with name 0.63 0.68 1.19 5.85
podman create pod with doubled name 0.69 0.75 1.28 5.98
podman create pod and print id to external file 0.71 0.73 1.25 6.37
podman create pod with same name as ctr 0.81 0.90 1.43 6.40
podman create pod with defaults 0.81 0.90 1.47 6.41
podman create pod with --infra-command 0.84 0.86 1.46 6.58
podman run user capabilities test 1.18 1.22 11.10 9.65
podman init single container by name 1.21 11.46 1.87 11.75
podman pod create --replace 1.27 1.30 2.11 8.21
podman pod prune removes a pod with a stopped container 1.32 1.35 1.98 7.79
podman pod prune doesn't remove a pod with a running container 1.33 1.45 2.13 7.27
podman init single container by ID 1.33 11.38 1.89 11.72
podman create pod with DNS option set 1.44 1.52 2.22 7.37
podman create pod with IP address 1.47 1.34 2.13 7.50
podman create pod with DNS server set 1.47 1.48 2.20 7.46
podman create pod with MAC address 1.49 1.60 2.27 7.22
podman create pod with --add-host 1.54 1.38 2.16 7.14
podman container in pod with IP address shares IP address 1.58 1.47 2.29 7.83
podman exec simple command using latest 1.59 5.07 1.95 1.68
podman create pod with network portbindings 1.60 2.02 2.29 5.35
podman attach to multiple containers 1.71 6.09 2.56 2.20
podman create pod without network portbindings 1.73 1.75 2.28 7.51
podman create pod with --infra-command --infra-image 1.98 1.91 3.04 9.11
podman create pod with --no-hosts 2.06 2.10 2.92 8.40
podman pause a running container by id and another by name 2.15 2.33 5.61 2.76
podman run --seccomp-policy image (block all syscalls) 2.17 3.06 2.96 7.43
podman create pod with --infra-image 2.19 2.00 2.52 6.48
podman exec environment test 2.48 5.34 2.18 1.91
podman init all three containers, one running 2.59 22.71 3.62 22.97
podman images --all flag 2.98 3.66 5.56 4.82
podman pod restart all pods 3.01 5.03 4.33 3.63
podman pull + inspect from unqualified-search registry 3.04 3.26 5.14 5.11
podman pull check all tags 3.76 3.83 5.86 5.70
podman remote test context dir contains empty dirs and symlinks 4.03 4.08 7.01 7.09
podman run check dnsname plugin 4.39 11.76 17.01 15.08
podman pod restart multiple pods 5.20 7.92 6.83 6.01
podman search with wildcards 6.22 4.00 6.20 5.60
podman exec cannot be invoked 6.53 5.68 1.86 6.86
podman exec command not found 6.58 3.40 1.83 6.48
podman exec preserve fds sanity check 6.71 3.31 1.89 4.09
podman exec exit code 6.74 2.96 1.92 1.63
podman exec simple command with user 6.76 5.72 1.88 1.65
podman run selinux 7.02 1.13 0.87 0.80
podman exec --detach 7.04 6.33 2.43 7.81
podman exec os.Setenv env 7.09 5.10 1.96 1.69
podman exec with user from run 7.10 5.89 2.22 2.42
podman exec simple working directory test 7.17 5.97 2.23 3.53
podman exec missing working directory test 7.41 5.79 2.12 4.23
podman exec --privileged with user 9.92 7.43 2.97 2.55
podman exec --privileged 12.20 8.21 3.71 3.25
podman exec --privileged container not running as root 11.27 4.89 3.72 3.10
podman exec with user with cap-add 12.09 9.24 4.02 3.55
podman exec with user with and cap-drop cap-add 14.70 10.19 4.43 3.88
podman save --multi-image-archive (untagged images) 17.51 28.16 24.76 21.32
podman exec preserves --group-add groups 33.50 29.16 16.82 28.72
podman exec preserves container groups with --user and --group-add 49.40 3.98 13.73 41.67

Some of those are pretty big differences (4s vs 49s, 3 vs 22)

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CI: integration timing results are randomly incomplete
6 participants