Podman support beta #674

LimeHat · 2021-10-29T22:03:58Z

As discussed. Added build tags to avoid breaking things

hellt · 2021-11-17T09:49:01Z

in #679 we have added support for CPU quota and mem limits

LimeHat · 2022-01-05T17:16:57Z

All right, finally managed to get back to this.

For build process to work without cgo and extra dependencies need to add a few build tags

podman exclude_graphdriver_btrfs btrfs_noversion exclude_graphdriver_devicemapper exclude_graphdriver_overlay containers_image_openpgp
E.g.:
sergey@clab-1:~/clab-podman/containerlab$ CGO_ENABLED=0 go build -trimpath -tags "podman exclude_graphdriver_btrfs btrfs_noversion exclude_graphdriver_devicemapper exclude_graphdriver_overlay containers_image_openpgp"

Will add #679 soon and do more checks to see how it behaves with the latest main.

LimeHat · 2022-01-06T17:30:06Z

Looks decent for a beta. Still need to add ipv6/dual-stack for mgmt addressing (prob. will do this separately); update docs & the build pipeline.
@hellt take a look when you're back. thx.

hellt · 2022-01-13T10:02:51Z

runtime/all/all_with_podman.go

+//go:build linux && podman
+// +build linux,podman
+
+package all
+
+import (
+	_ "github.com/srl-labs/containerlab/runtime/containerd"
+	_ "github.com/srl-labs/containerlab/runtime/docker"
+	_ "github.com/srl-labs/containerlab/runtime/ignite"
+	_ "github.com/srl-labs/containerlab/runtime/podman"
+)


do we still need build tags to have a build with podman, considering that you managed to exclude cgo dependency?

In principle - we don't; but the build process would require the presence of the other tags i mentioned earlier:
exclude_graphdriver_btrfs btrfs_noversion exclude_graphdriver_devicemapper exclude_graphdriver_overlay containers_image_openpgp

In the current form this is a safety first approach: in case of no tags go build will succeed without podman (based on runtime/all/all.go); otherwise we expect to see all the tags (I even contemplated to include them all).

It's a little bit ugly either way. 😞
I suspect this approach might be more straightforward for external contributors..

yes, that is what I thought, but once this PR is done, I think it would be best to include podman support to the GA build of containerlab (ofc we will include those tags to a regular build pipeline delivered by goreleaser)

That way it would be possible to let users with podman to play with it, without fetching separate binary. At this point, the binary increase that podman will incur won't matter much I guess.

we can keep the all and all_w_podman files for the time being

I think it would be best to include podman support to the GA build of containerlab

I mean, that is the plan;
the question is whether we want to keep an option to build clab without podman support for other contributors?

hellt · 2022-01-14T16:42:34Z

Do you know which build tag is missing in my trial to build the tip of that branch with the following cmd on ubuntu 20.04

CGO_ENABLED=0 go build -o $(pwd)/bin/containerlab -ldflags="-s -w -X 'github.com/srl-labs/containerlab/cmd.version=0.0.0' -X 'github.com/srl-labs/containerlab/cmd.commit=$(git rev-parse --short HEAD)' -X 'github.com/srl-labs/containerlab/cmd.date=$(date)'" -trimpath -tags "podman exclude_graphdriver_btrfs btrfs_noversion exclude_graphdriver_devicemapper exclude_graphdriver_overlay containers_image_openpgp" main.go
# github.com/containers/image/v5/manifest
../go/pkg/mod/github.com/containers/image/[email protected]/manifest/oci.go:44:234: undefined: v1.MediaTypeImageLayerNonDistributableZstd
../go/pkg/mod/github.com/containers/image/[email protected]/manifest/oci.go:44: undefined: v1.MediaTypeImageLayerZstd
../go/pkg/mod/github.com/containers/image/[email protected]/manifest/oci.go:105:39: undefined: v1.MediaTypeImageLayerNonDistributableZstd
../go/pkg/mod/github.com/containers/image/[email protected]/manifest/oci.go:110:39: undefined: v1.MediaTypeImageLayerZstd
make: *** [Makefile:13: build-with-podman] Error 2

I've asked the same q here - containers/podman#6866 (comment)

LimeHat · 2022-01-14T16:58:00Z

Do you know which build tag is missing in my trial to build the tip of that branch with the following cmd on ubuntu 20.04

Nothing to do with the tags; just a regular dependency management pain :)
I did this commit for a reason, but the last merge undid it

hellt · 2022-01-14T17:48:16Z

ouch, I wonder how did you find it out? My google foo led to me to similar reports without a solution

LimeHat · 2022-01-14T17:50:12Z

Do you know which build tag is missing in my trial to build the tip of that branch with the following cmd on ubuntu 20.04

Nothing to do with the tags; just a regular dependency management pain :) I did this commit for a reason, but the last merge undid it

Looks like the new containerd version introduced dependency on the regular v1.0.2 of image-spec; so poor go mod cannot resolve this anymore. Duh.

LimeHat · 2022-01-14T18:03:58Z

@hellt can you try this one. bumped to non-released v1.0.3, that should do it.

… podman-support

runtime/podman/portmaps.go

runtime/podman/util.go

hellt · 2022-01-17T09:58:31Z

I have a weird issue when I try this branch with podman installed on ubuntu 20.04
it looks like the issue with the missing podman.socket file

❯ podman info
host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 8
  distribution:
    codename: focal
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: devbox-u20
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.4.60-uksm
  linkmode: dynamic
  logDriver: journald
  memFree: 13552488448
  memTotal: 25221267456
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: ea1fe3938eefa14eb707f1d22adff4db670645d6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 0
  swapTotal: 0
  uptime: 641h 0m 46.64s (Approximately 26.71 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.4.2

podman cli works:

❯ podman ps -a
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES

which makes me think that it talks to a socket?

But the clab fails:

❯ dclab dep -t srl.clab.yml --runtime podman
INFO[0000] Containerlab v0.0.0 started                  
INFO[0000] Parsing & checking topology file: srl.clab.yml 
Error: could not list containers: unable to connect to Podman socket: Get "http://d/v3.4.4/libpod/_ping": dial unix /run//podman/podman.sock: connect: no such file or directory

and there is no socket file by that path

❯ ls -la /run/podman/podman.socket
ls: cannot access '/run/podman/podman.socket': No such file or directory

LimeHat · 2022-01-17T12:18:43Z

Right;
podman API depends on the systemd service so you need to enable or start it first (sudo systemctl start podman). Sorry I haven't documented this yet.

… podman-support

htmltest launched in CI will have github.com links excluded to remove 429 responses

hellt · 2022-01-17T21:03:03Z

Thanks @LimeHat, that is a solid chunk of work!

I have tuned the test pipeline a bit to accommodate for podman smoke tests

Currently I excluded the following tests, please check if my understanding is correct as to why these tests can not be currently executed with the podman runtime

1 Matching IPv6 address in `clab ins -a` output

This test is excluded as podman network is created with ipv4 address only.
Is IPv6 just not implemented in this PR or is there any blocker that prevents creating a backing linux bridge with ipv6 address?

2 Verify static ipv6 mgmt assignment

This test is skipped for the reason (1)

3 Host entries verification

Skipped for IPv6, since no IPv6 entries are added because of (1)

4 verification that management network can use an existing bridge

In this test the failure seems to be related to the fact that with podman runtime we can't use the custom name for the podman bridge?
Here is the error that I got for this test case

Verify management network is using user-specified bridge              | FAIL |
'278: veth46c36a53@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cni-podman1 state UP mode DEFAULT group default ' does not contain 'master 01-03-mgmt state UP'

the expectation of this test case is that the management veth pairs will have one end in the user defined bridge (named 01-03-mgmt in that test case) to ensure that the existing bridge is used to connect management interfaces.

5 custom bridge name test suite

This suite is skipped, as it tests the same as (4) it seems :D

6 check if --keep-mgmt-name works

this suite is skipped because destroy -t ${topo} --keep-mgmt-net have no effect, and the reason is that setting bridge name for podman to a certain name is not respected by the runtime.

LimeHat · 2022-01-18T10:44:27Z

Hi Roman, thanks for the thorough review & test efforts! 👍

So for 1-3), AFAIR, I encountered a limitation of the current API design where we cannot supply 2 distinct subnets to the network create call. So v6 itself it not an issue (I could've implemented it as either v4 or v6), but dual-stack is challenging. There are potential ways to workaround this (e.g. switch to a lower-level API or just implement custom cni cfg file logic); I just haven't spend enough time on this and planned to implement this later.
Now, there's a podman 4.0 release on the horizon (rc1 is out), which introduces some compatibility-breaking API changes (😩), but I believe they reworked this part so a slice of subnets will be accepted. I need to look at it in more detail, but that's another option.

For 4-6 - right.. I see I left a TODO item in the code on this 😄 let me check if i can accommodate custom bridge naming without too much hassle, I think I may be able to.
Since podman uses CNI-based networking suite, there's a bit of a distinction between a "network name" and an actual network interface naming in the underlying OS (and in the case of bridges, they don't inherit the former by default).
As a matter of fact, p.6 was kind of implemented (but based on the network name instead of bridge name).

Anyway, all the good points, and thanks for jogging my memory on the bridge naming thing. I'll look into it and get back to you soon.

hellt · 2022-01-24T13:46:53Z

Thanks @LimeHat
I am merging this, the future improvements will follow in subsequent PRs to keep them short and sweet

LimeHat added 6 commits January 5, 2022 08:29

Add initial podman support

5a6b55e

Add a few debug outputs

1c1f457

Update go.mod/go.sum

500efba

Add build tags for podman

e0710e6

Remove cgo build tags/depedency

6cfbe58

Update go.mod after a rebase

98df40b

LimeHat force-pushed the podman-support branch from 0910749 to 98df40b Compare January 5, 2022 17:06

LimeHat added 2 commits January 6, 2022 15:43

Fix go.mod version of image-spec dep

509ff34

Add cpu/mem limits to podman

fa8c422

LimeHat force-pushed the podman-support branch from 2dfbebf to fa8c422 Compare January 6, 2022 16:44

LimeHat marked this pull request as ready for review January 6, 2022 17:27

LimeHat changed the title ~~Podman support alpha~~ Podman support beta Jan 6, 2022

LimeHat assigned hellt and LimeHat Jan 6, 2022

LimeHat added 2 commits January 6, 2022 12:10

Add support for a custom gw address (v4) with podman

66cc26b

Add support for custom MTU on a podman mgmt net

0e34630

LimeHat linked an issue Jan 6, 2022 that may be closed by this pull request

add podman support #643

Closed

hellt reviewed Jan 13, 2022

View reviewed changes

hellt added 4 commits January 14, 2022 14:04

Merge branch 'main' into podman-support

627e06b

added info about podman runtime option

b9670d4

removed unused runtime consts

b6f15cf

applied gofmt for imports

dd5e411

added make target

4e122cf

Bump image-spec version to v1.0.3 to resolve conflicts

626b8cd

Merge branch 'podman-support' of github.com:LimeHat/containerlab into…

f7b43a5

… podman-support

hellt reviewed Jan 17, 2022

View reviewed changes

runtime/podman/portmaps.go Outdated Show resolved Hide resolved

hellt reviewed Jan 17, 2022

View reviewed changes

runtime/podman/portmaps.go Outdated Show resolved Hide resolved

hellt reviewed Jan 17, 2022

View reviewed changes

runtime/podman/util.go Outdated Show resolved Hide resolved

LimeHat and others added 13 commits January 17, 2022 05:46

Remove unused code artifacts

f2530a5

Merge branch 'podman-support' of github.com:LimeHat/containerlab into…

06e1733

… podman-support

fix typos

37cbb82

enable smoke tests for podman

beef04a

fix build cmd

9fd8b0e

set binary name

bb17d8c

added podman service start

02b800b

check podman bind mounts

aba2d1e

fix tests for podman

4154728

added sudo

b235b7f

split htmltest to two configs

8372fcb

htmltest launched in CI will have github.com links excluded to remove 429 responses

added url wildcard

90b2684

scope out a single test instead of a suite

6148f2b

hellt added 3 commits January 24, 2022 13:34

ignore backup lab files

ff1df8c

added podman flags to build config

a584f34

build tags rework

d594889

hellt merged commit db5847d into srl-labs:main Jan 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Podman support beta #674

Podman support beta #674

LimeHat commented Oct 29, 2021

hellt commented Nov 17, 2021

LimeHat commented Jan 5, 2022

LimeHat commented Jan 6, 2022

hellt Jan 13, 2022

LimeHat Jan 13, 2022 •

edited

Loading

hellt Jan 14, 2022 •

edited

Loading

LimeHat Jan 14, 2022

hellt commented Jan 14, 2022

LimeHat commented Jan 14, 2022 •

edited

Loading

hellt commented Jan 14, 2022

LimeHat commented Jan 14, 2022

LimeHat commented Jan 14, 2022

hellt commented Jan 17, 2022

LimeHat commented Jan 17, 2022

hellt commented Jan 17, 2022

LimeHat commented Jan 18, 2022 •

edited

Loading

hellt commented Jan 24, 2022

Podman support beta #674

Podman support beta #674

Conversation

LimeHat commented Oct 29, 2021

hellt commented Nov 17, 2021

LimeHat commented Jan 5, 2022

LimeHat commented Jan 6, 2022

hellt Jan 13, 2022

Choose a reason for hiding this comment

LimeHat Jan 13, 2022 • edited Loading

Choose a reason for hiding this comment

hellt Jan 14, 2022 • edited Loading

Choose a reason for hiding this comment

LimeHat Jan 14, 2022

Choose a reason for hiding this comment

hellt commented Jan 14, 2022

LimeHat commented Jan 14, 2022 • edited Loading

hellt commented Jan 14, 2022

LimeHat commented Jan 14, 2022

LimeHat commented Jan 14, 2022

hellt commented Jan 17, 2022

LimeHat commented Jan 17, 2022

hellt commented Jan 17, 2022

1 Matching IPv6 address in clab ins -a output

2 Verify static ipv6 mgmt assignment

3 Host entries verification

4 verification that management network can use an existing bridge

5 custom bridge name test suite

6 check if --keep-mgmt-name works

LimeHat commented Jan 18, 2022 • edited Loading

hellt commented Jan 24, 2022

LimeHat Jan 13, 2022 •

edited

Loading

hellt Jan 14, 2022 •

edited

Loading

LimeHat commented Jan 14, 2022 •

edited

Loading

1 Matching IPv6 address in `clab ins -a` output

LimeHat commented Jan 18, 2022 •

edited

Loading