Skip to content

Commit

Permalink
cni: use tmpfs location for ipam plugin
Browse files Browse the repository at this point in the history
When a Nomad host reboots, the network namespace files in the tmpfs in
`/var/run` are wiped out. So when we restore allocations after a host reboot, we
need to be able to restore both the network namespace and the network
configuration. But because the netns is newly created and we need to run the CNI
plugins again, this create potential conflicts with the IPAM plugin which has
written state to persistent disk at `/var/lib/cni`. These IPs aren't the ones
advertised to Consul, so there's no particular reason to keep them around after
a host reboot because all virtual interfaces need to be recreated too.

Reconfigure the CNI bridge configuration to use `/var/run/cni` as its state
directory. We already expect this location to be created by CNI because the
netns files are hard-coded to be created there too in `libcni`.

Note this does not fix the problem described for Docker in #24292 because that
appears to be related to the netns itself being restored unexpectedly from
Docker's state.

Ref: #24292 (comment)
Ref: https://www.cni.dev/plugins/current/ipam/host-local/#files
  • Loading branch information
tgross committed Dec 11, 2024
1 parent 3bc2c07 commit 6e8bf84
Show file tree
Hide file tree
Showing 3 changed files with 11 additions and 6 deletions.
3 changes: 3 additions & 0 deletions .changelog/24650.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
```release-note:bug
networking: use a tmpfs location for the state of CNI IPAM plugin used by bridge mode, to fix a bug where allocations would fail to restore after host reboot
```
7 changes: 4 additions & 3 deletions client/allocrunner/cni/bridge.go
Original file line number Diff line number Diff line change
Expand Up @@ -58,9 +58,10 @@ func NewNomadBridgeConflist(conf NomadBridgeConfig) Conflist {
ForceAddress: true,
HairpinMode: conf.HairpinMode,
Ipam: IPAM{
Type: "host-local",
Ranges: ipRanges,
Routes: ipRoutes,
Type: "host-local",
Ranges: ipRanges,
Routes: ipRoutes,
DataDir: "/var/run/cni",
},
},
Firewall{
Expand Down
7 changes: 4 additions & 3 deletions client/allocrunner/cni/plugins.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,10 @@ type Bridge struct {
Ipam IPAM `json:"ipam"`
}
type IPAM struct {
Type string `json:"type"`
Ranges [][]Range `json:"ranges"`
Routes []Route `json:"routes"`
Type string `json:"type"`
Ranges [][]Range `json:"ranges"`
Routes []Route `json:"routes"`
DataDir string `json:"dataDir"`
}
type Range struct {
Subnet string `json:"subnet"`
Expand Down

0 comments on commit 6e8bf84

Please sign in to comment.