Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement pod-network-reload #8571

Merged

Conversation

Luap99
Copy link
Member

@Luap99 Luap99 commented Dec 3, 2020

This adds a new command, 'podman network reload', to reload the
networks of existing containers, forcing recreation of firewall
rules after e.g. firewall-cmd --reload wipes them out.

Under the hood, this works by calling CNI to tear down the
existing network, then recreate it using identical settings. We
request that CNI preserve the old IP and MAC address in most
cases (where the container only had 1 IP/MAC), but there will be
some downtime inherent to the teardown/bring-up approach. The
architecture of CNI doesn't really make doing this without
downtime easy (or maybe even possible...).

At present, this only works for root Podman, and only locally.
I don't think there is much of a point to adding remote support
(this is very much a local debugging command), but I think adding
rootless support (to kill/recreate slirp4netns) could be
valuable.

@openshift-ci-robot openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 3, 2020
@Luap99 Luap99 force-pushed the podman-network-reload branch from 34990a4 to d471398 Compare December 3, 2020 12:01
Comment on lines 884 to 886
// FIXME: There two bad logrus.Error in the ocicni lib we should remove them
// https://github.com/cri-o/ocicni/blob/5243a9f23f8a9c1ab706cf1632760cb49eb2277f/pkg/ocicni/ocicni.go#L640
// https://github.com/cri-o/ocicni/blob/5243a9f23f8a9c1ab706cf1632760cb49eb2277f/pkg/ocicni/ocicni.go#L774
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$ sudo bin/podman network reload --all
ERRO[0000] Error deleting network: running [/usr/sbin/iptables -t nat -D POSTROUTING -s 10.88.0.48 -j CNI-2210851933fa8e49e3ef4f38 -m comment --comment name: "podman" id: "b1b538e8bc4078fc3ee1c95b666ebc7449b9a97bacd15bcbe464a29e1be59c1c" --wait]: exit status 2: iptables v1.8.4 (legacy): Couldn't load target `CNI-2210851933fa8e49e3ef4f38':No such file or directory

Try `iptables -h' or 'iptables --help' for more information. 
ERRO[0000] Error while removing pod from CNI network "podman": running [/usr/sbin/iptables -t nat -D POSTROUTING -s 10.88.0.48 -j CNI-2210851933fa8e49e3ef4f38 -m comment --comment name: "podman" id: "b1b538e8bc4078fc3ee1c95b666ebc7449b9a97bacd15bcbe464a29e1be59c1c" --wait]: exit status 2: iptables v1.8.4 (legacy): Couldn't load target `CNI-2210851933fa8e49e3ef4f38':No such file or directory

Try `iptables -h' or 'iptables --help' for more information. 
ERRO[0000] Error deleting network: running [/usr/sbin/iptables -t nat -D POSTROUTING -s 10.88.0.47 -j CNI-520223cf89670efdff5d88d8 -m comment --comment name: "podman" id: "fe7e8eca56f844ec33af10f0aa3b31b44a172776e3277b9550a623ed5d96e72b" --wait]: exit status 2: iptables v1.8.4 (legacy): Couldn't load target `CNI-520223cf89670efdff5d88d8':No such file or directory

Try `iptables -h' or 'iptables --help' for more information. 
ERRO[0000] Error while removing pod from CNI network "podman": running [/usr/sbin/iptables -t nat -D POSTROUTING -s 10.88.0.47 -j CNI-520223cf89670efdff5d88d8 -m comment --comment name: "podman" id: "fe7e8eca56f844ec33af10f0aa3b31b44a172776e3277b9550a623ed5d96e72b" --wait]: exit status 2: iptables v1.8.4 (legacy): Couldn't load target `CNI-520223cf89670efdff5d88d8':No such file or directory

Try `iptables -h' or 'iptables --help' for more information. 
b1b538e8bc4078fc3ee1c95b666ebc7449b9a97bacd15bcbe464a29e1be59c1c
fe7e8eca56f844ec33af10f0aa3b31b44a172776e3277b9550a623ed5d96e72b

These errors are terrible the end user doesn't care and the command still works

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opened cri-o/ocicni#83 to remove these errors

@Luap99 Luap99 force-pushed the podman-network-reload branch from d471398 to de988cc Compare December 3, 2020 12:18
@Luap99
Copy link
Member Author

Luap99 commented Dec 3, 2020

@mheon PTAL

@Luap99 Luap99 force-pushed the podman-network-reload branch from de988cc to a081180 Compare December 3, 2020 12:34
@Luap99 Luap99 force-pushed the podman-network-reload branch from eef9267 to de0c479 Compare December 5, 2020 17:19
@openshift-ci-robot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Luap99

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 5, 2020
@rhatdan
Copy link
Member

rhatdan commented Dec 7, 2020

@mheon @baude PTAL

// teardownCNI will error if the iptables rules do not exists and this is the case after
// a firewall reload. The purpose of network reload is to recreate the rules if they do
// not exists so we should ignore possible errors.
_ = r.teardownCNI(ctr)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be ignoring the error here? Feels like we should at least logrus it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see #8571 (comment) I don't want to display this for users by default. Maybe logrus.Info is good enough?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should string-parse the error and discard any No such file or directory errors, and treat others as real errors?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, PTAL

@mheon
Copy link
Member

mheon commented Dec 7, 2020

One nit, otherwise LGTM

@mheon
Copy link
Member

mheon commented Dec 7, 2020

Thanks for taking this one over @Luap99

This adds a new command, 'podman network reload', to reload the
networks of existing containers, forcing recreation of firewall
rules after e.g. `firewall-cmd --reload` wipes them out.

Under the hood, this works by calling CNI to tear down the
existing network, then recreate it using identical settings. We
request that CNI preserve the old IP and MAC address in most
cases (where the container only had 1 IP/MAC), but there will be
some downtime inherent to the teardown/bring-up approach. The
architecture of CNI doesn't really make doing this without
downtime easy (or maybe even possible...).

At present, this only works for root Podman, and only locally.
I don't think there is much of a point to adding remote support
(this is very much a local debugging command), but I think adding
rootless support (to kill/recreate slirp4netns) could be
valuable.

Signed-off-by: Matthew Heon <[email protected]>
Signed-off-by: Paul Holzinger <[email protected]>
@Luap99 Luap99 force-pushed the podman-network-reload branch from de0c479 to b0286d6 Compare December 7, 2020 18:27
@mheon
Copy link
Member

mheon commented Dec 7, 2020

LGTM!

@Luap99 Luap99 changed the title WIP: Implement pod-network-reload Implement pod-network-reload Dec 7, 2020
@openshift-ci-robot openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 7, 2020
@rhatdan
Copy link
Member

rhatdan commented Dec 8, 2020

/lgtm

Nice work @Luap99

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 8, 2020
@openshift-merge-robot openshift-merge-robot merged commit 9b3a81a into containers:master Dec 8, 2020
@Luap99 Luap99 deleted the podman-network-reload branch January 2, 2021 20:18
Luap99 pushed a commit to Luap99/libpod that referenced this pull request Jan 14, 2021
The changes from cri-o/ocicni#83 are needed
to improve the user experience when using the new network reload command.

see: containers#8571 (comment)

Signed-off-by: Paul Holzinger <[email protected]>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants