DHCP ignores containers mac-address when sending DHCPREQUEST to server

[leier69]

Hi!

I'm opening this because I'm trying to make a Podman container to join a PFsense LAN network on a bridge using a static DHCP lease. I thought that when podman released 1.7 (supporting setting static mac-addresses) this would be an easy task. I was wrong.

So currently im using the following network config list:

[root@leierpc leier]# cat /etc/cni/net.d/podnet.conflist
{
   "cniVersion": "0.4.0",
   "name": "podnet",
   "plugins": [
      {
         "type": "bridge",
         "bridge": "br0",
         "ipam": {
            "type": "dhcp"
         }
      },
      {
        "type": "tuning",
        "capabilities": {
          "mac": true
      }
      }
   ]
}

And this works from a Podman perspective. It asks for a IP through the dhcp.sock and the mac-address sat is indeed static. From Podmans side of things.

[root@leierpc leier]# podman run -d --net podnet --name="testMac"  --mac-address=3e:84:ea:ab:7d:52 docker.io/library/debian tail -f /dev/null
2020/02/04 19:58:25 5a0841b3f0347378d14ce3f7673da0380a720cf1873bc9a768f98aacc42c9d39/podnet/eth0: acquiring lease
2020/02/04 19:58:26 5a0841b3f0347378d14ce3f7673da0380a720cf1873bc9a768f98aacc42c9d39/podnet/eth0: lease acquired, expiration is 2020-02-04 21:58:26.330024245 +0100 CET m=+13799.752185010
5a0841b3f0347378d14ce3f7673da0380a720cf1873bc9a768f98aacc42c9d39

[root@leierpc leier]# podman exec testMac ip link show 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth0@if38: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 3e:84:ea:ab:7d:52 brd ff:ff:ff:ff:ff:ff link-netnsid 0

But the DHCP plugin (/usr/libexec/cni/dhcp daemon &) does not respect the containers mac-address and presents the DHCP server with a seemingly random mac-address.

[root@leierpc leier]# podman start testMac 
2020/02/04 20:15:24 5a0841b3f0347378d14ce3f7673da0380a720cf1873bc9a768f98aacc42c9d39/podnet/eth0: acquiring lease
2020/02/04 20:15:25 5a0841b3f0347378d14ce3f7673da0380a720cf1873bc9a768f98aacc42c9d39/podnet/eth0: lease acquired, expiration is 2020-02-04 22:15:25.249862644 +0100 CET m=+14818.672023419
testMac

The mac-address on container testMac is currently 3e:84:ea:ab:7d:52 and nowhere to be seen

[root@leierpc leier]# tcpdump -n -i br0
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
20:15:24.216490 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 12:9f:61:68:d7:a7, length 326
20:15:24.392415 IP6 :: > ff02::1:ff68:d7a7: ICMP6, neighbor solicitation, who has fe80::109f:61ff:fe68:d7a7, length 32

All help would be appriciated, thanku

[mccv1r0]

It looks like the tuning plugin doesn't set the mac address supplied by the runtime until after the ipam plugin has finished. So in the case of the dhcp plugin, the mac address used for e.g. dhcp discover, request etc will be the random mac was assigned by Linux.

[leier69]

@mars1024 @squeed any thoughts?

[mars1024]

@mars1024 @squeed any thoughts?

Just as the comments by @mccv1r0 , the set-mac-address action happens in chained plugin tuning after the interface plugin bridge where the DHCP request is sent from. So the DHCP is not requesting with your assigned MAC address but a random one generated by kernel.

For now, I haven't thought of a good solution for this.

cc @dcbw @bboreham for more help!

[leier69]

Something worth noting is that when I change the config to acually specify the mac-address like so:

{
   "cniVersion": "0.4.0",
   "name": "podnet",
   "plugins": [
      {
         "type": "bridge",
         "bridge": "br0",
         "ipam": {
            "type": "dhcp"
         }
      },
      {
        "type": "tuning",
        "capabilities": {
          "mac": 3e:84:ea:ab:7d:52
      }
      }
   ]
}

It works in the same way as setting it throught the "--mac-address" argument. This means that the DHCP is still presenting the server with a random mac : (

[bboreham]

One idea: we could create an "ip assign" plugin which could appear later in the chain, change the bridge plugin so it just creates the veth at layer 2 if no ipam is supplied.

{
   "cniVersion": "0.4.0",
   "name": "podnet",
   "plugins": [
      {
         "type": "bridge",
         "bridge": "br0"
      },
      {
        "type": "tuning",
        "capabilities": {
          "mac": true
      },
      {
         "type": "ip-assign",
         "ipam": {
            "type": "dhcp"
         }
      }
   ]
}

[leier69]

That would solve the Issue but I think it would be a better idea to integrate it as an optional option in "tuning".
I was imagining something like this:

{
   "cniVersion": "0.4.0",
   "name": "podnet",
   "plugins": [
      {
         "type": "bridge",
         "bridge": "br0",
         "interface-share-mac": true,
         "ipam": {
            "type": "dhcp",
         }
      },
      {
        "type": "tuning",
        "capabilities": {
          "mac": true
          <or some option here>
      }
      }
   ]
}

interface-share-mac is a really bad name but I think it would be nicer to integrate it. It would be nice if you could just ask the mac plugin to use the containers mac instead of the interface's. or maybe an option under "type": "bridge" where you specify that the interface and container should have the same mac.

I would love to hear your thoughts on this @mccv1r0 @bboreham

[bboreham]

I suspect we are not understanding each other. I think the problem you have identified is that the tuning plugin runs after the bridge and dhcp plugins have finished, hence they cannot use the mac that it sets.

[leier69]

Hmm yea I see. Then ur solution is better, but still kind of feels like an extra step. Should not the default behavior of CNI be to look over the conf before setting an interface mac at random?

[leier69]

so where do we take this now @bboreham ? Do you know anybody that could help?

[bboreham]

Under CNI chaining, each plugin runs independently and with zero knowledge of what any other plugin does. The whole idea of CNI is that plugins can come from anywhere.
There are some videos where we explain the design of CNI and how the project works, e.g. https://www.youtube.com/watch?v=YjjrQiJOyME

Another option would be to replace bridge with a plugin that is otherwise similar (perhaps it works by wrapping bridge) but also knows you want to set the MAC before calling the IPAM plugin. I don't know if anyone has coded that already.

[leier69]

what do you mean by "wrapping bridge"?

[leier69]

And also, would it not be possible to add an option to the bridge plugin that sets the mac address of the interface it creates? That would solve the issue instantly. @bboreham

[bboreham]

Wrapping bridge meaning a plugin that calls the code in the current bridge plugin.

And we won’t add the option into bridge because we’re trying to have a composable design. There would be a similar need in macvlan, flannel, etc., etc.

[xenorites]

Is there any roadmap for this? I am wanting to roll a new deployment that relies on a podman backend, but I need some services to be set to specific IPs; I currently use DHCP for IPAM.

Would it be possible to control the UID generated by the DHCP plugin? I think that would allow for more granular control, or at least provide reliable results.

[mccv1r0]

Is there any roadmap for this?

Not yet.

I need some services to be set to specific IPs

As a workaround to get you going, podman allows you to set the IPv4 address explicitly, see ip=

You need to use host-local IPAM plugin for the range used for services needing static addresses. The value supplied to podman run must be in the host-local range. dhcp could be used for the rest? Or just use host-local for everything, letting host-local range replace the dhcp server range.

[carroarmato0]

Is there any roadmap for this?

Not yet.

I need some services to be set to specific IPs

As a workaround to get you going, podman allows you to set the IPv4 address explicitly, see ip=

You need to use host-local IPAM plugin for the range used for services needing static addresses. The value supplied to podman run must be in the host-local range. dhcp could be used for the rest? Or just use host-local for everything, letting host-local range replace the dhcp server range.

Would this still fully expose the container to the local network? (context: I'm running Home-Assistant in a container exposed on my network through macvlan, but every time I recreate the container, the IP address changes due to a new mac address being generated)

[mccv1r0]

Would this still fully expose the container to the local network?

Yes. The only difference would be your container would get the IPv4 address specified via podman run --ip=. If you want the mac-address to stay the same, add --mac-address to podman run (never version of podman support this) It uses the tuning plugin as described above so it will not solve your dhcp issue, but will keep your service from dealing with ARP timeouts due to mac/IP changes.

[carroarmato0]

Would this still fully expose the container to the local network?

Yes. The only difference would be your container would get the IPv4 address specified via podman run --ip=. If you want the mac-address to stay the same, add --mac-address to podman run (never version of podman support this) It uses the tuning plugin as described above so it will not solve your dhcp issue, but will keep your service from dealing with ARP timeouts due to mac/IP changes.

You just overly simplified my life. Got it to work. Thanks! :)

[zeroepoch]

Any update on this issue? The ability to set the MAC address with the macvlan plugin would be very useful. Some control over the MAC address being generated (not completely random) would also solve this problem for the most part because for static DHCP leases you just need it to be consistent.

[Domini]

As @zeroepoch I'm trying to use the prepopulated MAC address with MACVLAN networking to assign specific IP from DHCP.

One idea: we could create an "ip assign" plugin which could appear later in the chain, change the bridge plugin so it just creates the veth at layer 2 if no ipam is supplied.

{
   "cniVersion": "0.4.0",
   "name": "podnet",
   "plugins": [
      {
         "type": "bridge",
         "bridge": "br0"
      },
      {
        "type": "tuning",
        "capabilities": {
          "mac": true
      },
      {
         "type": "ip-assign",
         "ipam": {
            "type": "dhcp"
         }
      }
   ]
}

I like this idea as it allows support in all drivers at once and not just in one driver as the idea about the new feature in the bridge driver. Also, it detaches DHCP leasing into a separate entity which is an improvement per KISS principle.

Is there any progress on this issue?

[squeed]

My thought would be to implement the mac convention / capability arg directly in the interface plugins.

[bboreham]

we could create an "ip assign" plugin

Sadly this doesn't really help, as there are other steps performed by bridge et al besides assigning the IP - setting up routes for instance. So this design ends up with the same downside that you have to modify every bridge-like plugin, and you may as well do the conceptually simpler thing of having them support the mac capability.

[bolind]

I'm also affected by this issue. I'm guessing it mainly stems from the use case of wanting a predictable IP for one's long-lived container, such that it can be reached via a hostname rather than an IP-address.

While this is being worked on, does anyone have a workaround?

[Drakulix]

I'm also affected by this issue. I'm guessing it mainly stems from the use case of wanting a predictable IP for one's long-lived container, such that it can be reached via a hostname rather than an IP-address.

While this is being worked on, does anyone have a workaround?

I fear that this could needlessly spam the issue, so such discussions should likely be kept down to a minimum. But I have a somewhat working and a little creative work-around deployed.

My containers are attached via macvlan to a separate vlan network (to contain the mac-address spamming in my routers UI) and I am also running the dnsname plugin in my containers cni network.

However the containers are unable to communicate with the host via macvlan, so the dnsmasq instance of dnsname would be inaccessible by the containers. To work-around that, I do not actually have dnsmasq installed, but instead I use a small script as a replacement, that spawn dnsmasq with the given arguments/config files in another container with a static-ip-address:

#!/bin/bash

NAME=$(echo $3 | cut -c 13-)
FOLDER=$(dirname $NAME)
sed -i.bak '/pid-file/d' $NAME
rm $FOLDER/pidfile
podman run --name dnsmasq --hostname dns --network=extern-static-dnsmasq -v $FOLDER:$FOLDER:Z --rm -d --cap-add=NET_ADMIN andyshinn/dnsmasq --log-facility=- $@
sleep 5;
pgrep dnsmasq | tail -1 > $FOLDER/pidfile

Then I set the dns server of my other containers via --dns to the static dnsmasq container IP.

This is all super hacky, but works quite stable so far. But I would absolutely discourage anyone to deploy this anywhere, but their own home network or testing environment. Debugging this requires knowledge into how cni works and building this was a whole day of reading source code and trying other rather crazy ideas, until I finally settled upon this "solution". So be careful.

[Domini]

While we wait for the release, here is a small workaround that might fit in certain circumstances.

IPAM DHCP already sends Client ID in the format full-container-id/CNI-network-name/container-iface-name e. g. af09e70c2313b355811ecf4376e47093ad83f037f90d91fa70f4c8dd63157d78/eth5_macvlan/eth0.

If your container is long-lived and you control DHCP server and it has ability to issue addresses based on Client ID, you could assign the address based on this Client ID e. g. for dnsmasq it would be --dhcp-host=id:af09e70c2313b355811ecf4376e47093ad83f037f90d91fa70f4c8dd63157d78/eth5_macvlan/eth0,192.168.0.160.

BTW with dnsmasq it's quite easily scriptable if you also use dhcp-hostsdir.

I hope it helps.

[alexanderniebuhr]

the solution #480 by @clinta just works fine with --mac-address flag ...

just need to run go build for macvlan plugin file and replace file on server node, very easy until official release. I dont think we need any workarounds with client id. its just simple to build file