Declare and use networks in yaml with podman play kube

[MartinX3]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

I miss the ability to create and use networks in yaml files like PVCs.

Steps to reproduce the issue:

No issue

Describe the results you received:

Missing support.

Describe the results you expected:

Support of the feature.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.17.4
Git Commit:   f6526ada1025c2e3f88745ba83b8b461ca659933
Built:        Thu Dec  9 19:30:40 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon ist in conmon 1:2.0.32-2 enthalten
    path: /usr/bin/conmon
    version: 'conmon version 2.0.32, commit: 436b460d1586c2e4ab4e845448449ddd9136767a'
  cpus: 8
  distribution:
    distribution: endeavouros
    version: unknown
  eventLogger: journald
  hostname: deus-ex-machina
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.16.1-zen1-1-zen
  linkmode: dynamic
  logDriver: journald
  memFree: 857567232
  memTotal: 33377427456
  ociRuntime:
    name: crun
    package: /usr/bin/crun ist in crun 1.4.1-1 enthalten
    path: /usr/bin/crun
    version: |-
      crun version 1.4.1
      commit: 802613580a3f25a88105ce4b78126202fef51dfb
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns ist in slirp4netns 1.1.12-1 enthalten
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 36722176000
  swapTotal: 36722176000
  uptime: 20h 18m 41.31s (Approximately 0.83 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/martin/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /home/martin/.local/share/containers/storage
  graphStatus:
    Build Version: 'Btrfs v5.15.1 '
    Library Version: "102"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /home/martin/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 1639074640
  BuiltTime: Thu Dec  9 19:30:40 2021
  GitCommit: f6526ada1025c2e3f88745ba83b8b461ca659933
  GoVersion: go1.17.4
  OsArch: linux/amd64
  Version: 3.4.4

Package info (e.g. output of rpm -q podman or apt list podman):

https://archlinux.org/packages/community/x86_64/podman/

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Arch Linux

[mheon]

Original discussion in #12852

I believe this is related to podman play kube.

[fpoirotte]

For comparison with kubernetes: k8s automatically adds a network interface to each pod. That interface is attached to k8s's network. It is not possible to attach multiple network interfaces to the pod, unless a plugin such as Multus is used.

Multus uses an annotation (k8s.v1.cni.cncf.io/networks) inside the YAML file to attach & configure additional network interfaces to the pod. See https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/how-to-use.md#run-pod-with-network-annotation.

It would be great if podman supported those annotations too, but this may prove difficult:

• Multus allows the same annotation to be used multiple times (resulting in multiple interfaces on the same network). podman seems to treat the values given to --network as a set and can only attach a single interface per network.
• Multus allows each interface name to be customized. podman does too for bridges through the "interface_name" option (see https://docs.podman.io/en/latest/markdown/podman-run.1.html#network-mode-net), but the option is not supported yet for slirp4netns networks
• Multus allows a default route to be set for the interface. AFAICT, there is no equivalent option in podman.

[MartinX3]

Or we need support for kind: NetworkPolicy
https://kubernetes.io/docs/concepts/services-networking/network-policies/

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[MartinX3]

@mheon please remove the stale status

[fpoirotte]

Regarding NetworkPolicy objects, they are akin to firewalls, allowing only select traffic to pass around between specified pods (even if the pods are connected to the same network).

I don't think supporting them would solve this issue, but it could be a nice addition in the future (although it may be hard to implement in podman due to its dependency on labels and such).

[mheon]

I think that's a separate feature request, but it's definitely an interesting one. Mind making a separate issue for it? I think this has applications even outside of podman play kube.

[MartinX3]

@mheon done
#13538

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[MartinX3]

/remove Stale

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[MartinX3]

/remove Stale

[rhatdan]

@Luap99 PTAL

[Luap99]

k8s.v1.cni.cncf.io/networks seems to only specify particular network names.
If I understand the ask here correctly the goal is to also create networks. I have no experience with the k8s yaml format but I don't think it is a good idea to hack this in via annotations.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[MartinX3]

/remove Stale

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[MartinX3]

/remove Stale

[rhatdan]

@umohnani8 PTAL

[rhatdan]

@mheon PTAL

[andrew-kennedy]

I'm really missing this currently, as now we have official docs and blog posts explaining how to go from docker-compose services -> kubernetes yaml files -> systemd running via podman-kube@$(systemd-escape some.yml).service and yet there is no way to get those running pods to communicate easily if the original scenario was one app/service per docker-compose.yml file (and thus one pod/service per kubernetes.yml file).

[fpoirotte]

k8s.v1.cni.cncf.io/networks seems to only specify particular network names. If I understand the ask here correctly the goal is to also create networks. I have no experience with the k8s yaml format but I don't think it is a good idea to hack this in via annotations.

FWIU, new networks can be declared using the NetworkAttachmentDefinition resource kind (syntax / example), then referenced (and further configured) from a Pod manifest using k8s.v1.cni.cncf.io/networks (example)

I think it would make sense to add support for NetworkAttachmentDefinition. Users could then embed both the NetworkAttachmentDefinition and Pod/Deployment definition in the same YAML manifest (similar to how ConfigMap objects work ref)

[rhatdan]

@umohnani8 @haircommander @saschagrunert WDYT? ^^

[ancosma]

In order to mimic K8s behavior in regards to network, we have to create one common network to which all pods created by play-kube will be added.
Make play-kube do following by default:

podman network create play-kube-network
podman play kube --network play-kube-network KUBEFILE

This would allow pods to discover and communicate with other pods - that's the basic of K8s network model.
Note that I'm not a go developer, but since I need the default K8s behavior in regards to pods sharing same network space, I would give it a try to add creation of default network (if it doesn't exists) and attaching pods to it by default (maybe if play-kube is not called with --network option).

[umohnani8]

@MartinX3 did #16029 resolve the original ask of this issue? Or is something more needed?

[MartinX3]

I think the basic is solved.
Just this network has ipv6 disabled.
I don't know if it is a rootless limitation?
And I can't edit an existing network to enable it.

(But also IPv6 is broken, see #15850)

And I think the following things are still missing:

• https://kubernetes.io/docs/concepts/services-networking/network-policies/
• https://kubernetes.io/docs/concepts/services-networking/service/ (network isolation by namespaces)

[fpoirotte]

@MartinX3 did #16029 resolve the original ask of this issue? Or is something more needed?

While #16029 makes sure that pods share a common network by default, it does not address the case where you have defined your own networks and want to use them by referencing them in the YAML manifest. The current way to use those networks requires passing additional options (namely, --network) when calling podman play kube (i.e. outside the manifest).

[ancosma]

NetworkPolicy and multiple networks (as with Multus) are provided in K8s by CNI (and plugins). To some extent there is a conflict with podman network stack direction (see https://www.redhat.com/sysadmin/podman-new-network-stack) and podman-kube - which mimics K8s as a single node deployment.
Some things are possible with less effort, some require effort which makes development not be feasible and then the question is if podman is supposed to support/implement full k8s in podman-kube.
There are some questions which I'm asking myself and I would like to know your answer if possible:

• How much k8s should be in podman-kube?
• Make sense to use it in production because it's a lightweight k8s implementation (if no HA is required)?
• To which extent should I use podman-kube and when is the time to continue with k8s single node deployment (kind, minikube, etc.) for local development (if podman-kube is meant only for development)?

While I definitely like to have podman-kube 100% compatibility with k8s it seems that we're implementing k8s in podman - which only make sense if podman-kube is meant to be used in production and it's aim is to have k8s "API" compatibility - but then maybe podman-kube should use CNI and not netavark/aardark?

Edit:
P.S.: If podman-kube is supposed to be used in production I guess I'll give it a try to implement NetworkPolicy, Service and Namespace in the future, if not already implemented by the time I'll need them.

[MartinX3]

In my case I run rootless podman as a single node container solution in my private servers.

In our company we run a kubernetes cluster, but I can't simply deploy podman kube yaml files there, because of the missing compatibility.
Otherwise this ticket would resolved together with the other stuff missing.

[drpuur]

I also miss this basic feature, to add the network name in the yaml. For the time being, I just copied the systemd service file and made the links point to /etc/systemd/user...

[root@onyx ~]# cp /usr/lib/systemd/user/[email protected] /etc/systemd/user
[root@onyx ~]# diff /usr/lib/systemd/user/podman-kube\@.service /etc/systemd/user/podman-kube\@.service
12c12,13
< ExecStart=/usr/bin/podman play kube --replace --service-container=true %I
---
> ExecStart=/usr/bin/podman play kube --replace --network web --service-container=true %I

Please let me know if there is a better way :-)

[alexanderniebuhr]

I really don't care if the option is available as an cli option or an yaml annotation, however using an automatic GitOps pipeline (containers/fetchit#280), currently there is no way to set the network, because the cli options are not available in that context. Means all pods are created on the default network, which is unexpected. It would be better if a simple yaml annotation can be added, where we can set existing network names.

[rhatdan]

@Luap99 Thoughts?
@alexanderniebuhr Care to open a PR?

[Steve973]

I would like this feature, too. The less extra scripting (or other steps) that we have to do would be preferred.

[rhatdan]

@ygalblum something you could tackle?

[dmotte]

This would be extremely useful to me too, because I'm using [email protected] with rootless Podman and I need to set the --net slirp4netns:port_handler=slirp4netns option to be able to see the correct source IPs of the requests in the containers.

My "temporary fix" for now is the following:

sudo sed -i 's|^\(ExecStart=/usr/bin/podman play kube\)|\1 --net slirp4netns:port_handler=slirp4netns|' \
/usr/lib/systemd/*/[email protected]

but honestly I don't like it a lot because it alters Podman's systemd files

[vrothberg]

@ygalblum do you have ideas? I think Quadlet can help out here if not already supported.

[ygalblum]

Yes. Quadlet already has everything in place to support what @dmotte is describing. Assuming the K8S YAML is mykub.yaml the .kube Quadlet file should be:

[Kube]
Yaml=mykub.yaml
Network=slirp4netns:port_handler=slirp4netns

You can see more info here: https://docs.podman.io/en/stable/markdown/podman-systemd.unit.5.html#kube-units-kube

[dmotte]

Thank you so much for the answers guys! Didn't know about Quadlet. I did some research and found that "It has been merged into Podman 4.4" (source). Unfortunately, on the server:

$ podman --version
podman version 4.3.1

Because I'm using Podman from the Debian 12 repos :(

I guess I'll have to install Podman in some other way, or keep my current solution for now. But thanks anyway for letting me know!