Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

network upgrade tests: error adding pod: failed to allocate: duplicate allocation is not allowed #11558

Closed
edsantiago opened this issue Sep 13, 2021 · 5 comments · Fixed by #12260
Closed

network upgrade tests: error adding pod: failed to allocate: duplicate allocation is not allowed #11558

edsantiago opened this issue Sep 13, 2021 · 5 comments · Fixed by #12260
Assignees
@Luap99
Labels
flakes Flakes from Continuous Integration locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@edsantiago
Copy link
Member

New flake in the upgrade tests:

[+0033s] not ok 9 network - restart
         # (from function `die' in file test/upgrade/../system/helpers.bash, line 448,
         #  from function `run_podman' in file test/upgrade/../system/helpers.bash, line 221,
         #  in test file test/upgrade/test-upgrade.bats, line 231)
         #   `run_podman start myrunningcontainer' failed with status 125
         # # podman stop -t0 myrunningcontainer
         # myrunningcontainer
         # # podman start myrunningcontainer
         # Error: unable to start container "6d3f37a7d0c1b22a707db47d3360d9c748082dd86d3f49d201e58b9818a5fa67": error configuring network namespace for container 6d3f37a7d0c1b22a707db47d3360d9c748082dd86d3f49d201e58b9818a5fa67: error adding pod myrunningcontainer_myrunningcontainer to CNI network "mynetwork": failed to allocate for range 0: 10.89.0.2 has been allocated to 6d3f37a7d0c1b22a707db47d3360d9c748082dd86d3f49d201e58b9818a5fa67, duplicate allocation is not allowed
         # [ rc=125 (** EXPECTED 0 **) ]
         # #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
         # #| FAIL: exit code is 125; expected 0
         # #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@edsantiago edsantiago added the flakes Flakes from Continuous Integration label Sep 13, 2021
@Luap99
Copy link
Member

Luap99 commented Sep 14, 2021

I will wait for #11322 to merge. Given that this changes how we call cni it could fix this.
If this still happens after #11322 I will investigate further.

@Luap99
Copy link
Member

Luap99 commented Sep 23, 2021

Do you have any new failures after #11322 was merged?

@rhatdan
Copy link
Member

rhatdan commented Sep 23, 2021

If you do, please reopen issue.

@rhatdan rhatdan closed this as completed Sep 23, 2021
@edsantiago
Copy link
Member Author

It's back:

Weird: a few flakes up until 09-28 (which I will charitably assume were non-rebased PRs), then all of a sudden it starts happening again on 10-25.

@edsantiago edsantiago reopened this Oct 27, 2021
@Luap99 Luap99 mentioned this issue Nov 10, 2021
Merged
@Luap99
Copy link
Member

Luap99 commented Nov 10, 2021

PR #12260 should fix this

Luap99 added a commit to Luap99/libpod that referenced this issue Nov 11, 2021
@Luap99
Fix flake in upgrade tests
c43b81f 
The cni plugins need access to /run/cni and the dnsname plugin needs
access to /run/containers.

The race condition was basically that a `podman stop` could either do the
cleanup itself or the spawned cleanup process would do the cleanup if it
was fast enough. The `podman stop` is executed on the host while the
podman cleanup process is executed in the "parent container". The parent
container contains older plugins than on the host. The dnsname plugin
before version 1.3 could error and this would prevent CNI from
doing a proper cleanup. The plugin errors because it could not find its
files in /run/containers. On my system the test always failed because
the cleanup process was always faster than the stop process. However in
the CI VMs the stop process was usually faster and so it failed only
sometimes.

Fixes containers#11558

Signed-off-by: Paul Holzinger <[email protected]>
mheon pushed a commit to mheon/libpod that referenced this issue Nov 12, 2021
@Luap99 @mheon
Fix flake in upgrade tests
c2fb170 
The cni plugins need access to /run/cni and the dnsname plugin needs
access to /run/containers.

The race condition was basically that a `podman stop` could either do the
cleanup itself or the spawned cleanup process would do the cleanup if it
was fast enough. The `podman stop` is executed on the host while the
podman cleanup process is executed in the "parent container". The parent
container contains older plugins than on the host. The dnsname plugin
before version 1.3 could error and this would prevent CNI from
doing a proper cleanup. The plugin errors because it could not find its
files in /run/containers. On my system the test always failed because
the cleanup process was always faster than the stop process. However in
the CI VMs the stop process was usually faster and so it failed only
sometimes.

Fixes containers#11558

Signed-off-by: Paul Holzinger <[email protected]>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 21, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@Luap99 Luap99

Labels
flakes Flakes from Continuous Integration locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix flake in upgrade tests Luap99/libpod
3 participants
@edsantiago @rhatdan @Luap99