Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rootless networking error on container startup since v5.0.0 #22168

Closed
flyingfishflash opened this issue Mar 26, 2024 · 71 comments
Closed

rootless networking error on container startup since v5.0.0 #22168

flyingfishflash opened this issue Mar 26, 2024 · 71 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. network Networking related issue or feature pasta pasta(1) bugs or features

Comments

@flyingfishflash
Copy link

Issue Description

Since updating to podman 5, I am receiving the following error upon container startup.

Error: rootless netns: mount resolv.conf to "/run/user/10001/containers/networks/rootless-netns/run/systemd/resolve/stub-resolv.conf": no such file or directory

These are rootless containers running as systemd services via quadlet

Steps to reproduce the issue

Steps to reproduce the issue

  1. create hello-world.container
[Unit]
Description=Hello World

[Container]
AutoUpdate=registry
ContainerName=hello-world
Image=quay.io/podman/hello:latest
LogDriver=journald
Network=hello-world.network

[Install]
WantedBy=default.target

  1. create hello-world.network
[Network]

  1. systemctl --user daemon-reload
  2. systemctl --user start hello-world

Describe the results you received

Mar 25 20:58:40 nas systemd[711]: Starting Hello World...
Mar 25 20:58:40 nas podman[135892]: 2024-03-25 20:58:40.545778613 -0400 EDT m=+0.014761399 container create bfde8a8e86e9ffd86e5de627a10f4b3baf5a70bccc5f71b09666f628e46ad29b (image=quay.io/podman/hello:latest, name=hello-world, org.opencontainers.image.revision=76b262056eae09851d0a952d0f42b5bbeedde471, org.opencontainers.image.source=https://raw.githubusercontent.com/containers/PodmanHello/76b262056eae09851d0a952d0f42b5bbeedde471/Containerfile, io.containers.autoupdate=registry, maintainer=Podman Maintainers, org.opencontainers.image.description=Hello world image with ascii art, org.opencontainers.image.documentation=https://github.com/containers/PodmanHello/blob/76b262056eae09851d0a952d0f42b5bbeedde471/README.md, io.buildah.version=1.23.1, org.opencontainers.image.title=hello image, artist=Máirín Ní Ḋuḃṫaiġ, X/Twitter:@mairin, PODMAN_SYSTEMD_UNIT=hello-world.service, io.containers.capabilities=sys_chroot, org.opencontainers.image.url=https://github.com/containers/PodmanHello/actions/runs/8406198111)
Mar 25 20:58:40 nas podman[135892]: 2024-03-25 20:58:40.551110562 -0400 EDT m=+0.020093348 container remove bfde8a8e86e9ffd86e5de627a10f4b3baf5a70bccc5f71b09666f628e46ad29b (image=quay.io/podman/hello:latest, name=hello-world, org.opencontainers.image.url=https://github.com/containers/PodmanHello/actions/runs/8406198111, PODMAN_SYSTEMD_UNIT=hello-world.service, artist=Máirín Ní Ḋuḃṫaiġ, X/Twitter:@mairin, org.opencontainers.image.source=https://raw.githubusercontent.com/containers/PodmanHello/76b262056eae09851d0a952d0f42b5bbeedde471/Containerfile, org.opencontainers.image.title=hello image, io.buildah.version=1.23.1, io.containers.capabilities=sys_chroot, maintainer=Podman Maintainers, org.opencontainers.image.documentation=https://github.com/containers/PodmanHello/blob/76b262056eae09851d0a952d0f42b5bbeedde471/README.md, io.containers.autoupdate=registry, org.opencontainers.image.description=Hello world image with ascii art, org.opencontainers.image.revision=76b262056eae09851d0a952d0f42b5bbeedde471)
Mar 25 20:58:40 nas podman[135892]: 2024-03-25 20:58:40.541326914 -0400 EDT m=+0.010309705 image pull 338f8d8caa62e120293bac50496b88d73298047bfe4789a5a0621fe5ceb09860 quay.io/podman/hello:latest
Mar 25 20:58:40 nas hello-world[135892]: Error: rootless netns: mount resolv.conf to "/run/user/4005/containers/networks/rootless-netns/run/systemd/resolve/stub-resolv.conf": no such file or directory
Mar 25 20:58:40 nas systemd[711]: hello-world.service: Main process exited, code=exited, status=127/n/a
Mar 25 20:58:40 nas systemd[711]: hello-world.service: Failed with result 'exit-code'.
Mar 25 20:58:40 nas systemd[711]: Failed to start Hello World.

Describe the results you expected

successful container creation

podman info output

host:
  arch: amd64
  buildahVersion: 1.35.1
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.10-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: 2dcd736e46ded79a53339462bc251694b150f870'
  cpuUtilization:
    idlePercent: 99.51
    systemPercent: 0.14
    userPercent: 0.34
  cpus: 24
  databaseBackend: boltdb
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 1815
  hostname: nas
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 4005
      size: 1
    - container_id: 1
      host_id: 427680
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 4005
      size: 1
    - container_id: 1
      host_id: 427680
      size: 65536
  kernel: 6.8.1-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 20794699776
  memTotal: 33447477248
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: /usr/lib/podman/aardvark-dns is owned by aardvark-dns 1.10.0-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: /usr/lib/podman/netavark is owned by netavark 1.10.3-1
    path: /usr/lib/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.14.4-1
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/4005/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: /usr/bin/pasta is owned by passt 2024_03_20.71dd405-1
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/4005/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.3-1
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 3h 39m 55.00s (Approximately 0.12 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/container-runner/.config/containers/storage.conf
  containerStore:
    number: 26
    paused: 0
    running: 18
    stopped: 8
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/container-runner/.local/share/containers/storage
  graphRootAllocated: 490304405504
  graphRootUsed: 180877135872
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 45
  runRoot: /run/user/4005/containers
  transientStore: false
  volumePath: /home/container-runner/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.0
  Built: 1711060217
  BuiltTime: Thu Mar 21 18:30:17 2024
  GitCommit: e71ec6f1d94d2d97fb3afe08aae0d8adaf8bddf0-dirty
  GoVersion: go1.22.1
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.0


### Podman in a container

No

### Privileged Or Rootless

Rootless

### Upstream Latest Release

Yes

### Additional environment details

- archlinux
- netplan to manage systemd networking

### Additional information

_No response_
@flyingfishflash flyingfishflash added the kind/bug Categorizes issue or PR as related to a bug. label Mar 26, 2024
@sbrivio-rh sbrivio-rh added the network Networking related issue or feature label Mar 26, 2024
@thyeestes
Copy link

Same issue here. However I have a other VM with Fedora installed, where this issue doen't occur. Apart from the linux version I'm not able to find the differentiating factors in relation to this issue.

@rhatdan
Copy link
Member

rhatdan commented Mar 26, 2024

@Luap99 PTAL

@rhatdan
Copy link
Member

rhatdan commented Mar 26, 2024

@mheon PTAL

@Luap99
Copy link
Member

Luap99 commented Mar 26, 2024

Does it only fail in quadlet or does a normal podman run --network bridge quay.io/podman/hello:latest fail in the same way?
If so please run the command with --log-level debug and provide the output.

@flyingfishflash
Copy link
Author

flyingfishflash commented Mar 26, 2024

@Luap99 Same error:

[container-runner@redacted ~]$ podman run --network bridge quay.io/podman/hello:latest
Trying to pull quay.io/podman/hello:latest...
Getting image source signatures
Copying blob c6b9c3bd2ed6 done   | 
Copying config 338f8d8caa done   | 
Writing manifest to image destination
Error: rootless netns: mount resolv.conf to "/run/user/4005/containers/networks/rootless-netns/run/systemd/resolve/stub-resolv.conf": no such file or directory
[container-runner@redacted ~]$ 

This file does exist though:

-rw-r--r-- 1 systemd-resolve systemd-resolve 943 Mar 24 18:39 /run/systemd/resolve/stub-resolv.conf

with debug:

[container-runner@redacted ~]$ podman run --network bridge quay.io/podman/hello:latest --log-level=debug
Error: rootless netns: mount resolv.conf to "/run/user/4005/containers/networks/rootless-netns/run/systemd/resolve/stub-resolv.conf": no such file or directory
[container-runner@redacted ~]$ 

@Luap99
Copy link
Member

Luap99 commented Mar 26, 2024

you need to add --log-level=debug first as this a podman arg not a container one.
podman --log-level=debug run --network bridge quay.io/podman/hello:latest

@Xinkai
Copy link

Xinkai commented Mar 26, 2024

Same problem here, and it happens without quadlet. I think it happens when a network other than the default is specified.

Command: podman --log-level=debug run --network bridge quay.io/podman/hello:latest

INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman --log-level=debug run --network bridge quay.io/podman/hello:latest)
DEBU[0000] Using conmon: "/usr/bin/conmon"
INFO[0000] Using boltdb as database backend
DEBU[0000] Initializing boltdb state at /home/xinkai/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver
DEBU[0000] Using graph root /home/xinkai/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000/containers
DEBU[0000] Using static dir /home/xinkai/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/xinkai/.local/share/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
INFO[0000] [graphdriver] using prior storage driver: overlay
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 97
DEBU[0000] Pulling image quay.io/podman/hello:latest (policy: missing)
DEBU[0000] Looking up image "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "quay.io/podman/hello:latest" ...
DEBU[0000] parsed reference into "[overlay@/home/xinkai/.local/share/containers/storage+/run/user/1000/containers]@338f8d8caa62e120293bac50496b88d73298047bfe4789a5a0621fe5ceb09860"
DEBU[0000] Found image "quay.io/podman/hello:latest" as "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Found image "quay.io/podman/hello:latest" as "quay.io/podman/hello:latest" in local containers storage ([overlay@/home/xinkai/.local/share/containers/storage+/run/user/1000/containers]@338f8d8caa62e120293bac50496b88d73298047bfe4789a5a0621fe5ceb09860)
DEBU[0000] exporting opaque data as blob "sha256:338f8d8caa62e120293bac50496b88d73298047bfe4789a5a0621fe5ceb09860"
DEBU[0000] Looking up image "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "quay.io/podman/hello:latest" ...
DEBU[0000] parsed reference into "[overlay@/home/xinkai/.local/share/containers/storage+/run/user/1000/containers]@338f8d8caa62e120293bac50496b88d73298047bfe4789a5a0621fe5ceb09860"
DEBU[0000] Found image "quay.io/podman/hello:latest" as "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Found image "quay.io/podman/hello:latest" as "quay.io/podman/hello:latest" in local containers storage ([overlay@/home/xinkai/.local/share/containers/storage+/run/user/1000/containers]@338f8d8caa62e120293bac50496b88d73298047bfe4789a5a0621fe5ceb09860)
DEBU[0000] exporting opaque data as blob "sha256:338f8d8caa62e120293bac50496b88d73298047bfe4789a5a0621fe5ceb09860"
DEBU[0000] Looking up image "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "quay.io/podman/hello:latest" ...
DEBU[0000] parsed reference into "[overlay@/home/xinkai/.local/share/containers/storage+/run/user/1000/containers]@338f8d8caa62e120293bac50496b88d73298047bfe4789a5a0621fe5ceb09860"
DEBU[0000] Found image "quay.io/podman/hello:latest" as "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Found image "quay.io/podman/hello:latest" as "quay.io/podman/hello:latest" in local containers storage ([overlay@/home/xinkai/.local/share/containers/storage+/run/user/1000/containers]@338f8d8caa62e120293bac50496b88d73298047bfe4789a5a0621fe5ceb09860)
DEBU[0000] exporting opaque data as blob "sha256:338f8d8caa62e120293bac50496b88d73298047bfe4789a5a0621fe5ceb09860"
DEBU[0000] Inspecting image 338f8d8caa62e120293bac50496b88d73298047bfe4789a5a0621fe5ceb09860
DEBU[0000] exporting opaque data as blob "sha256:338f8d8caa62e120293bac50496b88d73298047bfe4789a5a0621fe5ceb09860"
DEBU[0000] exporting opaque data as blob "sha256:338f8d8caa62e120293bac50496b88d73298047bfe4789a5a0621fe5ceb09860"
DEBU[0000] Inspecting image 338f8d8caa62e120293bac50496b88d73298047bfe4789a5a0621fe5ceb09860
DEBU[0000] Inspecting image 338f8d8caa62e120293bac50496b88d73298047bfe4789a5a0621fe5ceb09860
DEBU[0000] Inspecting image 338f8d8caa62e120293bac50496b88d73298047bfe4789a5a0621fe5ceb09860
DEBU[0000] using systemd mode: false
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Loading seccomp profile from "/etc/containers/seccomp.json"
DEBU[0000] Successfully loaded network local-llm: &{local-llm 9121c0354f8edcec58998ae192917a9ae88aa82f750e5624645fd58453c5140c bridge podman1 2024-03-27 00:29:31.994458753 +0800 CST [{{{10.89.0.0 ffffff00}} 10.89.0.1 <nil>} {{{fd69:fef8:d38d:6084:: ffffffffffffffff0000000000000000}} fd69:fef8:d38d:6084::1 <nil>}] [] true false true [] map[] map[] map[driver:host-local]}
DEBU[0000] Successfully loaded 2 networks
DEBU[0000] Allocated lock 38 for container 8b3de70ca2143d0898c1589c18405a19b1028b3bcc9f537279dd9022fbc914d4
DEBU[0000] exporting opaque data as blob "sha256:338f8d8caa62e120293bac50496b88d73298047bfe4789a5a0621fe5ceb09860"
DEBU[0000] Cached value indicated that idmapped mounts for overlay are not supported
DEBU[0000] Check for idmapped mounts support
DEBU[0000] Created container "8b3de70ca2143d0898c1589c18405a19b1028b3bcc9f537279dd9022fbc914d4"
DEBU[0000] Container "8b3de70ca2143d0898c1589c18405a19b1028b3bcc9f537279dd9022fbc914d4" has work directory "/home/xinkai/.local/share/containers/storage/overlay-containers/8b3de70ca2143d0898c1589c18405a19b1028b3bcc9f537279dd9022fbc914d4/userdata"
DEBU[0000] Container "8b3de70ca2143d0898c1589c18405a19b1028b3bcc9f537279dd9022fbc914d4" has run directory "/run/user/1000/containers/overlay-containers/8b3de70ca2143d0898c1589c18405a19b1028b3bcc9f537279dd9022fbc914d4/userdata"
DEBU[0000] Not attaching to stdin
INFO[0000] Received shutdown.Stop(), terminating!        PID=27448
DEBU[0000] Enabling signal proxying
DEBU[0000] overlay: mount_data=lowerdir=/home/xinkai/.local/share/containers/storage/overlay/l/MYYBPVA3KGAUVPC6FML4R3KKZI,upperdir=/home/xinkai/.local/share/containers/storage/overlay/3a2e6139f482f0f2a968e4ba82b771730c0e382690762e4fe7960b60cfa74871/diff,workdir=/home/xinkai/.local/share/containers/storage/overlay/3a2e6139f482f0f2a968e4ba82b771730c0e382690762e4fe7960b60cfa74871/work,userxattr
DEBU[0000] Made network namespace at /run/user/1000/netns/netns-796b8aa3-54ee-0d00-1633-67f35919bb79 for container 8b3de70ca2143d0898c1589c18405a19b1028b3bcc9f537279dd9022fbc914d4
DEBU[0000] Mounted container "8b3de70ca2143d0898c1589c18405a19b1028b3bcc9f537279dd9022fbc914d4" at "/home/xinkai/.local/share/containers/storage/overlay/3a2e6139f482f0f2a968e4ba82b771730c0e382690762e4fe7960b60cfa74871/merged"
DEBU[0000] Created root filesystem for container 8b3de70ca2143d0898c1589c18405a19b1028b3bcc9f537279dd9022fbc914d4 at /home/xinkai/.local/share/containers/storage/overlay/3a2e6139f482f0f2a968e4ba82b771730c0e382690762e4fe7960b60cfa74871/merged
DEBU[0000] The path of /etc/resolv.conf in the mount ns is "/run/systemd/resolve/stub-resolv.conf"
DEBU[0000] Unmounted container "8b3de70ca2143d0898c1589c18405a19b1028b3bcc9f537279dd9022fbc914d4"
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Cleaning up container 8b3de70ca2143d0898c1589c18405a19b1028b3bcc9f537279dd9022fbc914d4
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Container 8b3de70ca2143d0898c1589c18405a19b1028b3bcc9f537279dd9022fbc914d4 storage is already unmounted, skipping...
DEBU[0000] ExitCode msg: "rootless netns: mount resolv.conf to \"/run/user/1000/containers/networks/rootless-netns/run/systemd/resolve/stub-resolv.conf\": no such file or directory"
Error: rootless netns: mount resolv.conf to "/run/user/1000/containers/networks/rootless-netns/run/systemd/resolve/stub-resolv.conf": no such file or directory
DEBU[0000] Shutting down engines

My setup: Arch Linux, podman (5.0.0-1), netavark (1.10.3-1), aardvark-dns (1.10.0-1), cni-plugins (1.4.1-1), slirp4netns (1.2.3-1)

I also ran tree /run/user/1000/netns/, and tree /run/user/1000/containers/networks/rootless-netns

/run/user/1000/netns/
├── netns-14f9c69e-4fcf-84de-012b-0acb855bb923
├── netns-606c6b59-a81b-2c12-5e7d-911418b9d438
├── netns-bb0f1796-98ac-cd8e-39e1-bfcfe7783d5c
└── netns-db507a44-b311-b901-8f50-3482d62c6580

1 directory, 4 files
/run/user/1000/containers/networks/rootless-netns
├── rootless-netns
└── run
    ├── systemd
    └── user
        └── 1000

5 directories, 1 file

@Luap99
Copy link
Member

Luap99 commented Mar 27, 2024

This is a podman error so it is not related to pasta, slirp4netns or netavark in any way. It is not clear to me why that would suddenly fail. We use this code for several years now without running into this (I did refactor it for 5.0 but the functionality should be the same). I also use the default systemd-resolvd setup and it works on my machine with 5.0 so the question would be what is different between my system and yours? The only thing I can think of would be the kernel.

Looking at the directory structure from the host is kinda pointless because this setup does a rather complex mount namesapce setup in order to overwrite resolv.conf in the network namespace. So what you see there isn't what the mount namespace sees.

@Luap99
Copy link
Member

Luap99 commented Mar 27, 2024

Can you reboot to clear out any temporary leftover state, then after the boot run

podman unshare strace -f -o log.txt podman --log-level debug unshare --rootless-netns true

and then upload the created log.txt file here.

Also can you show me the output of findmnt -o TARGET,PROPAGATION on your systems?

@thyeestes
Copy link

log.txt

@thyeestes
Copy link

TARGET PROPAGATION
/ shared
├─/sys shared
│ ├─/sys/kernel/security shared
│ ├─/sys/fs/cgroup shared
│ ├─/sys/fs/pstore shared
│ ├─/sys/firmware/efi/efivars shared
│ ├─/sys/fs/bpf shared
│ ├─/sys/kernel/debug shared
│ ├─/sys/kernel/tracing shared
│ ├─/sys/kernel/config shared
│ └─/sys/fs/fuse/connections shared
├─/proc shared
│ └─/proc/sys/fs/binfmt_misc shared
│ └─/proc/sys/fs/binfmt_misc shared
├─/dev shared
│ ├─/dev/pts shared
│ ├─/dev/shm shared
│ ├─/dev/mqueue shared
│ └─/dev/hugepages shared
├─/run shared
│ ├─/run/lock shared
│ ├─/run/credentials/systemd-sysusers.service shared
│ ├─/run/credentials/systemd-sysctl.service shared
│ ├─/run/credentials/systemd-tmpfiles-setup-dev.service shared
│ ├─/run/user/1000 shared
│ │ ├─/run/user/1000/gvfs shared
│ │ └─/run/user/1000/doc shared
│ ├─/run/credentials/systemd-tmpfiles-setup.service shared
│ ├─/run/credentials/systemd-resolved.service shared
│ └─/run/snapd/ns private
│ └─/run/snapd/ns/snapd-desktop-integration.mnt private
├─/boot/efi shared
├─/snap/bare/5 shared
├─/snap/core20/2105 shared
├─/snap/core20/2182 shared
├─/snap/core22/1033 shared
├─/snap/core22/1122 shared
├─/snap/firefox/3972 shared
├─/snap/firefox/4033 shared
├─/snap/firmware-updater/121 shared
├─/snap/firmware-updater/109 shared
├─/snap/gnome-42-2204/141 shared
├─/snap/gnome-42-2204/172 shared
├─/snap/gtk-common-themes/1535 shared
├─/snap/snap-store/1106 shared
├─/snap/snap-store/1110 shared
├─/snap/snapd/20671 shared
├─/snap/snapd/21184 shared
├─/var/snap/firefox/common/host-hunspell shared
├─/snap/snapd-desktop-integration/83 shared
├─/media/movies shared
└─/media/series shared

@thyeestes
Copy link

I hope this helps

@Luap99
Copy link
Member

Luap99 commented Mar 27, 2024

@thyeestes Did you reboot before running this command? From your strace it looks like something already setup the netns before. Please make sure to remove /run/user/1000/containers/networks/rootless-netns before running this command.

@Luap99
Copy link
Member

Luap99 commented Mar 27, 2024

/run and /run/user/1000 are shared so this is good

@thyeestes
Copy link

I rebooted my machine, could the setup of netns be caused by the .kube and .container files I have on my system?

Anyway I removed the folder /run/user/1000/containers/networks/rootless-netns and its contents

New log file attached
log.txt

@Luap99
Copy link
Member

Luap99 commented Mar 27, 2024

I rebooted my machine, could the setup of netns be caused by the .kube and .container files I have on my system?

yes

Thanks the new log seems to have the info I need and that command did not log any error? I can see in the log that the pasta setup failed so that explains the root cause. If you do not see any errors related to that then something in our error reporting is definitely wrong.

pasta is failing because it cannot open the netns, that can happen if the selinux setup isn't right.

@thyeestes
Copy link

This is the output I get on screen, there is a permission denied error:
INFO[0000] podman filtering at log level debug
DEBU[0000] Called unshare.PersistentPreRunE(podman --log-level debug unshare --rootless-netns true)
DEBU[0000] Using conmon: "/usr/libexec/podman/conmon"
INFO[0000] Using boltdb as database backend
DEBU[0000] Initializing boltdb state at /home/thyestes/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/thyestes/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000/containers
DEBU[0000] Using static dir /home/thyestes/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/thyestes/.local/share/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 13
DEBU[0000] Creating rootless network namespace at "/run/user/1000/containers/networks/rootless-netns/rootless-netns"
DEBU[0000] pasta arguments: --config-net --pid /run/user/1000/containers/networks/rootless-netns/rootless-netns-conn.pid -t none -u none -T none -U none --no-map-gw --dns none --quiet --netns /run/user/1000/containers/networks/rootless-netns/rootless-netns
DEBU[0000] Cleaning up rootless network namespace
ERRO[0000] Failed to cleanup rootless netns: rootless netns: cleanup: 1 error occurred:
* rootless netns: kill network process: open /run/user/1000/containers/networks/rootless-netns/rootless-netns-conn.pid: no such file or directory

Error: setting up Pasta: pasta failed with exit code 1:
Can't run AVX2 build, using non-AVX2 version: No such file or directory
No external routable interface for IPv6
Couldn't open network namespace /run/user/1000/containers/networks/rootless-netns/rootless-netns: Permission denied

DEBU[0000] Shutting down engines

I am running on Ubuntu so no selinux installed.

@Luap99
Copy link
Member

Luap99 commented Mar 27, 2024

How did you install passt (pasta)? What version is this?

@Luap99
Copy link
Member

Luap99 commented Mar 27, 2024

Also on ubuntu apparmor might blocking the access instead of selinux.

@thyeestes
Copy link

thyeestes commented Mar 27, 2024

Yes I installed passt manually, when podman was upgraded to version 5.

Package: passt
Status: install ok installed
Priority: optional
Section: net
Installed-Size: 224
Maintainer: Ubuntu Developers [email protected]
Architecture: amd64
Multi-Arch: foreign
Version: 0.0~git20230627.289301b-1
Depends: libc6 (>= 2.34)
Suggests: apparmor
Conffiles:
/etc/apparmor.d/abstractions/passt 3056c2e078fe4b310a67c4a12416acb8
/etc/apparmor.d/abstractions/pasta fb43832bc4728ae7b57c86ad806e26f1
/etc/apparmor.d/usr.bin.passt 6a73e8a45041fc0d732bccc30d69d57b
Description: user-mode networking daemons for virtual machines and namespaces
passt implements a translation layer between a Layer-2 network interface and
native Layer-4 sockets (TCP, UDP, ICMP/ICMPv6 echo) on a host. It doesn't
require any capabilities or privileges, and it can be used as a simple
replacement for Slirp.
.
pasta (same binary as passt, different command) offers equivalent
functionality, for network namespaces: traffic is forwarded using a tap
interface inside the namespace, without the need to create further interfaces
on the host, hence not requiring any capabilities or privileges.
Original-Maintainer: Stefano Brivio [email protected]
Homepage: https://passt.top/

@sbrivio-rh
Copy link
Collaborator

Yes I installed passt manually, when podman was upgraded to version 5.

[...]

Version: 0.0~git20230627.289301b-1

Uh oh, that's ancient. Could you please try installing 0.0~git20240220.1e6f92b-1 (current version for Ubuntu 24.04) or 0.0~git20240326.4988e2b-1 (current version for Debian unstable)?

The AppArmor profile (/etc/apparmor.d/abstractions/pasta) has:

  @{run}/user/@{uid}/netns/*            r,      # pasta_open_ns(), pasta.c

so I don't think AppArmor is giving you any issue here, but the lack of pasta commit 594dce66d3bb ("isolation: keep CAP_SYS_PTRACE when required") probably is.

@sbrivio-rh
Copy link
Collaborator

so I don't think AppArmor is giving you any issue here, but the lack of pasta commit 594dce66d3bb ("isolation: keep CAP_SYS_PTRACE when required") probably is.

Ah, no, that commit is actually included in 0.0~git20230627.289301b-1. But still, that version of the passt package is not really tested/used with Podman 5.0, so I think we can expect issues with that combination.

@Luap99
Copy link
Member

Luap99 commented Mar 27, 2024

The AppArmor profile (/etc/apparmor.d/abstractions/pasta) has:

  @{run}/user/@{uid}/netns/*            r,      # pasta_open_ns(), pasta.c

Actually we use /run/user/1000/containers/networks/rootless-netns/rootless-netns for the rootless netns case so that doesn't match your glob and thus likely causes the issues. And well the path depends on our runroot so it can be basically anything as this one is user configurable. I don't think it is sane for pasta to limit what netns paths it can use (same for its selinux policy)

@sbrivio-rh
Copy link
Collaborator

The AppArmor profile (/etc/apparmor.d/abstractions/pasta) has:

  @{run}/user/@{uid}/netns/*            r,      # pasta_open_ns(), pasta.c

Actually we use /run/user/1000/containers/networks/rootless-netns/rootless-netns for the rootless netns case so that doesn't match your glob and thus likely causes the issues.

Oops, I misread, right.

And well the path depends on our runroot so it can be basically anything as this one is user configurable. I don't think it is sane for pasta to limit what netns paths it can use (same for its selinux policy)

That should depend on Podman's profile, which should in turn include that abstraction and add specific paths on top, like we do for libvirt, which has in its profile:

  # support for passt network back-end
  audit /usr/bin/passt Cx -> passt,
  profile passt {
    audit /usr/bin/passt r,

    owner @{run}/user/[0-9]*/libvirt/qemu/run/passt/* rw,
    signal (receive) set=("term") peer=/usr/sbin/libvirtd,
    signal (receive) set=("term") peer=libvirtd,

    include if exists <abstractions/passt>
  }

but indeed we still need to implement proper inclusions (and domain transitions for SELinux).

Meanwhile I can fix this up by changing the glob to @{run}/user/@{uid}/** or suchlike, I'm just puzzled as to why I don't see this on Debian, though.

@thyeestes
Copy link

@sbrivio-rh I am not too experienced, so I don't know how to install a newer version of passt.

@flyingfishflash
Copy link
Author

FWIW today I updated (in Archlinux) to pasta 2024_03_26.4988e2b, and I am still getting the same error

@Luap99
Copy link
Member

Luap99 commented Mar 27, 2024

@flyingfishflash do you get the same permission denied error for the pasta setup?

@sbrivio-rh
Copy link
Collaborator

@sbrivio-rh I am not too experienced, so I don't know how to install a newer version of passt.

There are a number of ways, such as adding a Ubuntu version to apt's configuration, or a number of tricks with aptitude. Two simpler ways are:

...in any case, if the issue is actually the one @Luap99 mentioned at #22168 (comment), you would need to edit passt's AppArmor policy at /etc/apparmor.d/abstractions/pasta and change the @{run}/user/@{uid}/netns/* glob to something like @{run}/user/@{uid}/**.

I wonder, though, if this is actually the same issue as originally reported in this ticket. The setup description at #22168 (comment) mentions slirp4netns, so I'm not sure if that has anything to do with this.

Luap99 added a commit to Luap99/common that referenced this issue Apr 2, 2024
When the netns program fails to configure the netns or we fail for any
other reason during the setup we must make sure to remove the netns
mount again. Without it the next command sees the existing mount and
thinks the netns was setup correctly but than later fails during the
custom resolv.conf mount because the resolv.conf source file was never
created.

For future we should consider adding checks due ensure pasta/slirp4netns
is still running when we access the netns to make it more fault
tolerant.

The reason this is a common problem is that on boot pasta can likely
fail if it was started before the networking was fully configured (i.e.
no default route).

Fixes containers/podman#22168

Signed-off-by: Paul Holzinger <[email protected]>
Luap99 added a commit to Luap99/libpod that referenced this issue Apr 3, 2024
@andreaippo
Copy link

andreaippo commented Apr 9, 2024

Hello,
I'm experiencing the same issue on opensuse tumbleweed:

podman compose up -d

Error response from daemon: rootless netns: mount resolv.conf to "/etc/resolv.conf": no such file or directory
Error: executing /usr/local/bin/docker-compose up -d: exit status 1

It is unclear to me if a fix/workaround is available.

I did:

sudo rm -r /run/user/$UID/containers/networks/rootless-netns

and confirmed that the folder is gone with:

sudo ls /run/user/1000/containers/networks/

but the problem is still there

podman info

host:
  arch: amd64
  buildahVersion: 1.35.3
  cgroupControllers:
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.3.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: unknown'
  cpuUtilization:
    idlePercent: 97.64
    systemPercent: 0.75
    userPercent: 1.61
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: opensuse-tumbleweed
    version: "20240407"
  eventLogger: journald
  freeLocks: 1683
  hostname: andromeda
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.8.4-rc1-1-default
  linkmode: dynamic
  logDriver: journald
  memFree: 19067904000
  memTotal: 32794767360
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.3.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.2.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.2.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-20240220.1e6f92b-1.2.x86_64
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1.2.x86_64
    version: |-
      slirp4netns version 1.2.3
      commit: unknown
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 69789020160
  swapTotal: 69789020160
  uptime: 1h 50m 26.00s (Approximately 0.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.opensuse.org
  - registry.suse.com
  - docker.io
store:
  configFile: /home/andrea/.config/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 0
    stopped: 7
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/andrea/.local/share/containers/storage
  graphRootAllocated: 707194257408
  graphRootUsed: 205737566208
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 16
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/andrea/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.1
  Built: 1712166221
  BuiltTime: Wed Apr  3 19:43:41 2024
  GitCommit: ""
  GoVersion: go1.21.9
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.1

Thanks

@sbrivio-rh
Copy link
Collaborator

Hello, I'm experiencing the same issue on opensuse tumbleweed:

Can you install a package version including this change and try again? I'm not sure it's the same issue, but the issue fixed by those changes would otherwise prevent pasta(1) from starting.

@andreaippo
Copy link

Hello, I'm experiencing the same issue on opensuse tumbleweed:

Can you install a package version including this change and try again? I'm not sure it's the same issue, but the issue fixed by those changes would otherwise prevent pasta(1) from starting.

Hi, I think that the hard linking is already in place on my system:

-rwxr-xr-x 1 root root      169736 Mar 14 10:40 passt
-rwxr-xr-x 1 root root      194320 Mar 14 10:40 passt.avx2
lrwxrwxrwx 1 root root           5 Mar 14 10:40 pasta -> passt
lrwxrwxrwx 1 root root          10 Mar 14 10:40 pasta.avx2 -> passt.avx2

Is there anything else I have to do?

I've read about apparmor, do I need to take some action there, or will this hard link make the executable run just because of the renaming?

Thanks

@sbrivio-rh
Copy link
Collaborator

Hi, I think that the hard linking is already in place on my system:

-rwxr-xr-x 1 root root      169736 Mar 14 10:40 passt
-rwxr-xr-x 1 root root      194320 Mar 14 10:40 passt.avx2
lrwxrwxrwx 1 root root           5 Mar 14 10:40 pasta -> passt
lrwxrwxrwx 1 root root          10 Mar 14 10:40 pasta.avx2 -> passt.avx2

Those are symbolic ("soft") links (note the 1 in -rwxr-xr-x 1 root root), look:

$ touch x
$ ln -s x s
$ ln x h
$ ls -l
total 0
-rw-r--r-- 2 sbrivio sbrivio 0 Apr  9 22:28 h
lrwxrwxrwx 1 sbrivio sbrivio 1 Apr  9 22:28 s -> x
-rw-r--r-- 2 sbrivio sbrivio 0 Apr  9 22:28 x

Is there anything else I have to do?

...yes, I think you should make sure that version of the package is installed.

I've read about apparmor, do I need to take some action there, or will this hard link make the executable run just because of the renaming?

The hard link is needed so that the right profile (usr.bin.pasta) is associated to the pasta command (instead of the usr.bin.passt profile), but the issue I pointed to includes more than that: the AppArmor profile also had missing rules that have been added now.

@danishprakash
Copy link
Contributor

danishprakash commented Apr 10, 2024

Although merged, the updated TW package hasn't yet been released. Just to confirm, @andreaippo, you can confirm you're not running the latest version by:

$ rpm -q passt
passt-20240220.1e6f92b-1.2.x86_64

The package with the fixes should be available for install alongside today's TW snapshot release.

@realSConway
Copy link

I confirm same issue on openSUSE MicroOS VERSION="20240407".

Containers/pods not able to start, error: Error: unable to start container "xxx": rootless netns: mount resolv.conf to "/run/user/1000/containers/networks/rootless-netns/run/netconfig/resolv.conf": no such file or directory

host:
  arch: amd64
  buildahVersion: 1.35.3
  cgroupControllers:
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.3.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: unknown'
  cpuUtilization:
    idlePercent: 99.85
    systemPercent: 0.11
    userPercent: 0.04
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: opensuse-microos
    version: "20240407"
  eventLogger: journald
  freeLocks: 2001
  hostname: srv04
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    - container_id: 65537
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    - container_id: 65537
      host_id: 165536
      size: 65536
  kernel: 6.8.4-rc1-1-default
  linkmode: dynamic
  logDriver: journald
  memFree: 12159459328
  memTotal: 17597767680
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.3.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.2.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.2.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-20240220.1e6f92b-1.2.x86_64
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
   path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1.2.x86_64
    version: |-
      slirp4netns version 1.2.3
      commit: unknown
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 28h 26m 20.00s (Approximately 1.17 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.opensuse.org
  - registry.suse.com
  - docker.io
  - quay.io
store:
  configFile: /home/support/.config/containers/storage.conf
  containerStore:
    number: 16
    paused: 0
    running: 2
    stopped: 14
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /home/support/.local/share/containers/storage
  graphRootAllocated: 21474836480
  graphRootUsed: 18801340416
  graphStatus:
    Build Version: Btrfs v6.8
    Library Version: "102"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 47
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/support/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.1
  Built: 1712166221
  BuiltTime: Wed Apr  3 19:43:41 2024
  GitCommit: ""
  GoVersion: go1.21.9
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.1

$ rpm -q passt returns passt-20240220.1e6f92b-1.2.x86_64

Lets see if tonight's snapshot release fixes it =)

@andreaippo
Copy link

rpm -q passt

Hi,

indeed it looks like I'm not running the latest:

 🐧 andrea 10:03:36 10/04/24  🏠  ✅  rpm -q passt
passt-20240220.1e6f92b-1.2.x86_64

@andreaippo
Copy link

andreaippo commented Apr 10, 2024

Hi, I think that the hard linking is already in place on my system:

-rwxr-xr-x 1 root root      169736 Mar 14 10:40 passt
-rwxr-xr-x 1 root root      194320 Mar 14 10:40 passt.avx2
lrwxrwxrwx 1 root root           5 Mar 14 10:40 pasta -> passt
lrwxrwxrwx 1 root root          10 Mar 14 10:40 pasta.avx2 -> passt.avx2

Those are symbolic ("soft") links (note the 1 in -rwxr-xr-x 1 root root), look:

$ touch x
$ ln -s x s
$ ln x h
$ ls -l
total 0
-rw-r--r-- 2 sbrivio sbrivio 0 Apr  9 22:28 h
lrwxrwxrwx 1 sbrivio sbrivio 1 Apr  9 22:28 s -> x
-rw-r--r-- 2 sbrivio sbrivio 0 Apr  9 22:28 x

Is there anything else I have to do?

...yes, I think you should make sure that version of the package is installed.

I've read about apparmor, do I need to take some action there, or will this hard link make the executable run just because of the renaming?

The hard link is needed so that the right profile (usr.bin.pasta) is associated to the pasta command (instead of the usr.bin.passt profile), but the issue I pointed to includes more than that: the AppArmor profile also had missing rules that have been added now.

Thanks for the explanation :)

Since I don't know exactly how to import the required apparmor profile, I think I will wait for tonight's opensuse Tumbleweed snapshot and let the update take care of putting all the pieces in the right places :)

@SiNONiMiTY
Copy link

Updated to OpenSUSE Snapshot 20240409, and podman works once again

@realSConway
Copy link

hmm, I updated my openSUSE MicroOS server to version 20240409, but still getting error:
Error: starting container xxx: rootless netns: mount resolv.conf to "/run/user/1000/containers/networks/rootless-netns/run/netconfig/resolv.conf": no such file or directory

@sbrivio-rh
Copy link
Collaborator

hmm, I updated my openSUSE MicroOS server to version 20240409, but still getting error: Error: starting container xxx: rootless netns: mount resolv.conf to "/run/user/1000/containers/networks/rootless-netns/run/netconfig/resolv.conf": no such file or directory

Did you have a look at #22168 (comment) too?

@andreaippo
Copy link

I can also confirm that with opensuse Tumbleweed snapshot 20240409 the problem is gone.

I had previously followed instructions mentioned in #22168 (comment)

@realSConway
Copy link

realSConway commented Apr 11, 2024

Yes I rebooted machine, and followed comment: #22168 (comment)
Still error: Error: starting container xxx: rootless netns: mount resolv.conf to "/run/user/1000/containers/networks/rootless-netns/run/netconfig/resolv.conf": no such file or directory

@andreaippo
Copy link

Yes I rebooted machine, and followed comment: #22168 (comment) Still error: Error: starting container xxx: rootless netns: mount resolv.conf to "/run/user/1000/containers/networks/rootless-netns/run/netconfig/resolv.conf": no such file or directory

I'd also suggest reporting this on the opensuse forums, maybe there's some difference between microOS and tumbleweed due to immutability? No idea

@Luap99
Copy link
Member

Luap99 commented Apr 11, 2024

Yes I rebooted machine, and followed comment: #22168 (comment) Still error: Error: starting container xxx: rootless netns: mount resolv.conf to "/run/user/1000/containers/networks/rootless-netns/run/netconfig/resolv.conf": no such file or directory

On second look your path is different from the defaults (or well not sure what the default is on micosos), the issue is the link to /run/netconfig/resolv.conf because /run/netconfig will not exists in our private mount namespace. Podman should change the code to create the parent dirs. If this worked on 4.X this is likely a regression from me when I reworked the code into c/common. I suggest you file a new issue for that.

@Luap99
Copy link
Member

Luap99 commented Apr 11, 2024

Yes I rebooted machine, and followed comment: #22168 (comment) Still error: Error: starting container xxx: rootless netns: mount resolv.conf to "/run/user/1000/containers/networks/rootless-netns/run/netconfig/resolv.conf": no such file or directory

On second look your path is different from the defaults (or well not sure what the default is on micosos), the issue is the link to /run/netconfig/resolv.conf because /run/netconfig will not exists in our private mount namespace. Podman should change the code to create the parent dirs. If this worked on 4.X this is likely a regression from me when I reworked the code into c/common. I suggest you file a new issue for that.

Nevermind I looked at the code and it seems to do the right thing in this case, if you say you rebooted did you actually make sure you do not have anything auto start a container at boot? I suggest you delete the directory and then immediately run podman unshare --rootless-netns true if this fails with the same error please rerun the command with --log-level debug.

@realSConway
Copy link

I ran podman unshare --rootless-netns true which returned:

podman unshare --rootless-netns true
ERRO[0000] Failed to cleanup rootless netns: rootless netns: cleanup: 1 error occurred:
        * rootless netns: kill network process: open /run/user/1000/containers/networks/rootless-netns
/rootless-netns-conn.pid: no such file or directory

Error: rootless netns: mount resolv.conf to "/run/user/1000/containers/networks/rootless-netns/run/net
config/resolv.conf": no such file or directory

I then tried as root, but got warning/error: Error: please use unshare with rootless
After running podman unshare --rootless-netns true a secondtime as rootless exit code 0 and pod was able to start again.

Luap99 added a commit to Luap99/common that referenced this issue Apr 12, 2024
When the netns program fails to configure the netns or we fail for any
other reason during the setup we must make sure to remove the netns
mount again. Without it the next command sees the existing mount and
thinks the netns was setup correctly but than later fails during the
custom resolv.conf mount because the resolv.conf source file was never
created.

For future we should consider adding checks due ensure pasta/slirp4netns
is still running when we access the netns to make it more fault
tolerant.

The reason this is a common problem is that on boot pasta can likely
fail if it was started before the networking was fully configured (i.e.
no default route).

Fixes containers/podman#22168

Signed-off-by: Paul Holzinger <[email protected]>
@HernandoR
Copy link

experiencing same, whats interesting is i don't think i saw any pasta installed on my machine. an minimal example is follows:

 $ ls -la /run/user/$UID/containers/networks
total 32
drwxr-xr-x 2 lz lz    60 May 17 23:00 .
drwx------ 7 lz lz   140 May 17 22:41 ..
-rw------- 1 lz lz 65536 May 17 22:56 ipam.db

$ podman run --network bridge quay.io/podman/hello:latest     
Error: setting up Pasta: could not find pasta, the network namespace can't be configured: exec: "pasta": executable file not found in $PATH

$ podman run --network bridge quay.io/podman/hello:latest
Error: rootless netns: mount resolv.conf to "/run/user/1000/containers/networks/rootless-netns/run/systemd/resolve/resolv.conf": no such file or directory

after installed passt with vision 0.0~git20240426
the error changed at least

$ podman run --network bridge quay.io/podman/hello:latest
Error: rootless netns: mount resolv.conf to "/run/user/1000/containers/networks/rootless-netns/run/systemd/resolve/resolv.conf": no such file or directory

$ pasta --version                              
pasta 0.0~git20240426.d03c4e2-1
Copyright Red Hat
GNU General Public License, version 2 or later
  <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.


$ sudo rm -r /run/user/$UID/containers/networks/rootless-netns

$ podman run --network bridge quay.io/podman/hello:latest     
Error: OCI runtime error: crun: unknown version specified

@sbrivio-rh
Copy link
Collaborator

$ podman run --network bridge quay.io/podman/hello:latest     
Error: OCI runtime error: crun: unknown version specified

That's another issue, you need a newer crun version (at least 1.14.3, https://github.com/containers/crun/releases/tag/1.14.3), including this commit: containers/crun@0860c0f

@HernandoR
Copy link

$ podman run --network bridge quay.io/podman/hello:latest     
Error: OCI runtime error: crun: unknown version specified

That's another issue, you need a newer crun version (at least 1.14.3, https://github.com/containers/crun/releases/tag/1.14.3), including this commit: containers/crun@0860c0f

Yes, i understood, just to conclude this issue, and remind what's may happening after the fix. Also, just to remind the followings. My crun installed by homebrew along with podman is actually good enough with version 1.15 but the podman is using apt installed 1.8. It can be checked out through podman info and which cron as follows:

$  podman info|grep crun
    name: crun
    package: crun_1.8.1-1+deb12u1_amd64
    path: /usr/bin/crun
      crun version 1.8.1
      rundir: /run/user/1000/crun

$ which crun           
/home/linuxbrew/.linuxbrew/bin/crun

$  crun --version     
crun version 1.15
commit: e6eacaf4034e84185fd8780ac9262bbf57082278
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL

In my case, i just apt remove crun, and all goes just fine.

netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Jun 4, 2024
Tested only on netbsd/amd64, probably a regression for others, but newer VMs
don't work with old podman.
NOTE: if you have old config files, you will probably want to delete them.
Newer podman fails to read them.
I personally deleted my ~/.config/containers/podman

# Release Notes

## 5.0.3
### Security
- This release addresses CVE-2024-3727, a vulnerability in the containers/image library which allows attackers to trigger authenticated registry access on behalf of the victim user.

### Bugfixes
- Fixed a bug where `podman machine start` would fail if the machine had a volume with a long target path ([#22226](https://github.com/containers/podman/issues/22226)).
- Fixed a bug where `podman machine start` mounted volumes with paths that included dashes in the wrong location ([#22505](https://github.com/containers/podman/issues/22505)).

### Misc
- Updated Buildah to v1.35.4
- Updated the containers/common library to v0.58.3
- Updated the containers/image library to v5.30.1

## 5.0.2
### Bugfixes
- Fixed a bug that could leak IPAM entries when a network was removed ([#22034](https://github.com/containers/podman/issues/22034)).
- Fixed a bug that could cause the rootless network namespace to not be cleaned up on if an error occurred during setup resulting in errors relating to a missing resolv.conf being displayed ([#22168](https://github.com/containers/podman/issues/22168)).
- Fixed a bug where Podman would use rootless network namespace logic for nested containers ([#22218](https://github.com/containers/podman/issues/22218)).
- Fixed a bug where writing to volumes on a Mac could result in EACCESS failures when using the `:z` or `:Z` volume mount options on a directory with read only files ([#19852](https://github.com/containers/podman/issues/19852))

### API
- Fixed a bug in the Compat List endpoint for Networks which could result in a server crash due to concurrent writes to a map ([#22330](https://github.com/containers/podman/issues/22330)).

## 5.0.1
### Bugfixes
- Fixed a bug where rootless containers using the Pasta network driver did not properly handle localhost DNS resolvers on the host leading to DNS resolution issues ([#22044](https://github.com/containers/podman/issues/22044)).
- Fixed a bug where Podman would warn that cgroups v1 systems were no longer supported on FreeBSD hosts.
- Fixed a bug where HyperV `podman machine` VMs required an SSH client be installed on the system ([#22075](https://github.com/containers/podman/issues/22075)).
- Fixed a bug that prevented the remote Podman client's `podman build` command from working properly when connecting from a rootless client to a rootful server ([#22109](https://github.com/containers/podman/issues/22109)).

### Misc
- The HyperV driver to `podman machine` now fails immediately if admin privileges are not available (previously, it would only fail when it reached operations that required admin privileges).

## 5.0.0
### Security
- Fixed [CVE-2024-1753](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-1753) in Buildah and `podman build` which allowed a user to write files to the `/` directory of the host machine if selinux was not enabled.

### Features
- VMs created by `podman machine` can now use the native Apple hypervisor (`applehv`) when run on MacOS.
- A new command has been added, `podman machine reset`, which will remove all existing `podman machine` VMs and relevant configurations.
- The `podman manifest add` command now supports a new `--artifact` option to add OCI artifacts to a manifest list.
- The `podman create`, `podman run`, and `podman push` commands now support the `--retry` and `--retry-delay` options to configure retries for pushing and pulling images.
- The `podman run` and `podman exec` commands now support a new option, `--preserve-fd`, which allows passing a list of file descriptors into the container (as an alternative to `--preserve-fds`, which passes a specific number of file descriptors).
- Quadlet now supports templated units ([#17744](https://github.com/containers/podman/discussions/17744)).
- The `podman kube play` command can now create image-based volumes using the `volume.podman.io/image` annotation.
- Containers created with `podman kube play` can now include volumes from other containers (similar to the `--volumes-from` option) using a new annotation, `io.podman.annotations.volumes-from` ([#16819](https://github.com/containers/podman/issues/16819)).
- Pods created with `podman kube play` can now set user namespace options through the the `io.podman.annotations.userns` annotation in the pod definition ([#20658](https://github.com/containers/podman/issues/20658)).
- Macvlan and ipvlan networks can adjust the name of the network interface created inside containers via the new `containers.conf` field `interface_name` ([#21313](https://github.com/containers/podman/issues/21313)).
- The `--gpus` option to `podman create` and `podman run` is now compatible with Nvidia GPUs ([#21156](https://github.com/containers/podman/issues/21156)).
- The `--mount` option to `podman create` and `podman run` supports a new mount option, `no-dereference`, to mount a symlink (instead of its dereferenced target) into a container ([#20098](https://github.com/containers/podman/issues/20098)).
- Podman now supports a new global option, `--config`, to point to a Docker configuration where we can source registry login credentials.
- The `podman ps --format` command now supports a new format specifier, `.Label` ([#20957](https://github.com/containers/podman/issues/20957)).
- The `uidmapping` and `gidmapping` options to the `podman run --userns=auto` option can now map to host IDs by prefixing host IDs with the `@` symbol.
- Quadlet now supports systemd-style drop-in directories.
- Quadlet now supports creating pods via new `.pod` unit files ([#17687](https://github.com/containers/podman/discussions/17687)).
- Quadlet now supports two new keys, `Entrypoint` and `StopTimeout`, in `.container` files ([#20585](https://github.com/containers/podman/issues/20585) and [#21134](https://github.com/containers/podman/issues/21134)).
- Quadlet now supports specifying the `Ulimit` key multiple times in `.container` files to set more than one ulimit on a container.
- Quadlet now supports setting the `Notify` key to `healthy` in `.container` files, to only sdnotify that a container has started when its health check begins passing ([#18189](https://github.com/containers/podman/issues/18189)).

### Breaking Changes
- The backend for the `podman machine` commands has seen extensive rewrites. Configuration files have changed format and VMs from Podman 4.x and earlier are no longer usable. `podman machine` VMs must be recreated with Podman 5.
- The `podman machine init` command now pulls images as OCI artifacts, instead of using HTTP. As a result, a valid `policy.json` file is required on the host. Windows and Mac installers have been changed to install this file.
- QEMU is no longer a supported VM provider for `podman machine` on Mac. Instead, the native Apple hypervisor is supported.
- The `ConfigPath` and `Image` fields are no longer provided by the `podman machine inspect` command. Users can also no longer use `{{ .ConfigPath }}` or `{{ .Image }}` as arguments to `podman machine inspect --format`.
- The output of `podman inspect` for containers has seen a number of breaking changes to improve Docker compatibility, including changing `Entrypoint` from a string to an array of strings and StopSignal from an int to a string.
- The `podman inspect` command for containers now returns nil for healthchecks when inspecting containers without healthchecks.
- The `podman pod inspect` command now outputs a JSON array regardless of the number of pods inspected (previously, inspecting a single pod would omit the array).
- It is no longer possible to create new BoltDB databases; attempting to do so will result in an error. All new Podman installations will now use the SQLite database backend. Existing BoltDB databases remain usable.
- Support for CNI networking has been gated by a build tag and will not be enabled by default.
- Podman will now print warnings when used on cgroups v1 systems. Support for cgroups v1 is deprecated and will be removed in a future release. The `PODMAN_CGROUPSV1_WARNING` environment variable can be set to suppress warnings.
- Network statistics sent over the Docker API are now per-interface, and not aggregated, improving Docker compatibility.
- The default tool for rootless networking has been swapped from `slirp4netns` to `pasta` for improved performance. As a result, networks named `pasta` are no longer supported.
- The `--image` option replaces the now deprecated `--image-path` option for `podman machine init`.
- The output of `podman events --format "{{json .}}"` has been changed to improve Docker compatibility, including the `time` and `timeNano` fields ([#14993](https://github.com/containers/podman/issues/14993)).
- The name of `podman machine` VMs and the username used within the VM are now validated and must match this regex: `[a-zA-Z0-9][a-zA-Z0-9_.-]*`.
- Using multiple filters with the List Images REST API now combines the filters with AND instead of OR, improving Docker compatibility ([#18412](https://github.com/containers/podman/issues/18412)).
- The parsing for a number of Podman CLI options which accept arrays has been changed to no longer accept string-delineated lists, and instead to require the option to be passed multiple times. These options are `--annotation` to `podman manifest annotate` and `podman manifest add`, the `--configmap`, `--log-opt`, and `--annotation` options to `podman kube play`, the `--pubkeysfile` option to `podman image trust set`, the `--encryption-key` and `--decryption-key` options to `podman create`, `podman run`, `podman push` and `podman pull`, the `--env-file` option to `podman exec`, the `--bkio-weight-device`, `--device-read-bps`, `--device-write-bps` `--device-read-iops`, `--device-write-iops`, `--device`, `--label-file`, `--chrootdirs`, `--log-opt`, and `--env-file` options to `podman create` and `podman run`, and the `--hooks-dir` and `--module` global options.

### Changes
- The `podman system reset` command no longer waits for running containers to gracefully stop, and instead immediately sends SIGKILL ([#21874](https://github.com/containers/podman/issues/21874)).
- The `podman network inspect` command now includes running containers using the network in its output ([#14126](https://github.com/containers/podman/issues/14126)).
- The `podman compose` command is now supported on non-AMD64/ARM64 architectures.
- VMs created by `podman machine` will now pass HTTP proxy environment variables into the VM for all providers.
- The `--no-trunc` option to the `podman kube play` and `podman kube generate` commands has been deprecated. Podman now complies to the Kubernetes specification for annotation size, removing the need for this option.
- The `DOCKER_HOST` environment variable will be set by default for rootless users when podman-docker is installed.
- Connections from `podman system connection` and farms from `podman farm` are now written to a new configuration file called `podman-connections.conf`. As a result, Podman no longer writes to `containers.conf`. Existing connections from `containers.conf` will still be respected.
- Most `podman farm` subcommands (save for `podman farm build`) no longer need to connect to the machines in the farm to run.
- The `podman create` and `podman run` commands no longer require specifying an entrypoint on the command line when the container image does not define one. In this case, an empty command will be passed to the OCI runtime, and the resulting behavior is runtime-specific.
- The default SELinux label for content mounted from the host in `podman machine` VMs on Mac is now `system_u:object_r:nfs_t:s0` so that it can be shared with all containers without issue.
- Newly-created VMs created by `podman machine` will now share a single SSH key key for access. As a result, `podman machine rm --save-keys` is deprecated as the key will persist by default.

### Bugfixes
- Fixed a bug where the `podman stats` command would not show network statistics when the `pasta` network mode was used.
- Fixed a bug where `podman machine` VMs using the HyperV provider could not mount shares on directories that did not yet exist.
- Fixed a bug where the `podman compose` command did not respect the `--connection` and `--url` options.
- Fixed a bug where the `podman stop -t -1` command would wait for 0 seconds, not infinite seconds, before sending SIGKILL ([#21811](https://github.com/containers/podman/issues/21811)).
- Fixed a bug where Podman could deadlock when cleaning up a container when the `slirp4netns` network mode was used with a restart policy of `always` or `unless-stopped` or `on-failure` and a user namespace ([#21477](https://github.com/containers/podman/issues/21477)).
- Fixed a bug where uninstalling Podman on Mac did not remove the `docker.sock` symlink ([#20650](https://github.com/containers/podman/issues/20650)).
- Fixed a bug where preexisting volumes being mounted into a new container using a path that exists in said container would not be properly chowned ([#21608](https://github.com/containers/podman/issues/21608)).
- Fixed a bug where the `podman image scp` command could fail if there was not sufficient space in the destination machine's `/tmp` for the image ([#21239](https://github.com/containers/podman/issues/21239)).
- Fixed a bug where containers killed by running out of memory (including due to a memory limit) were not properly marked as OOM killed in `podman inspect` ([#13102](https://github.com/containers/podman/issues/13102)).
- Fixed a bug where `podman kube play` did not create memory-backed emptyDir volumes using a tmpfs filesystem.
- Fixed a bug where containers started with `--rm` were sometimes not removed after a reboot ([#21482](https://github.com/containers/podman/issues/21482)).
- Fixed a bug where the `podman events` command using the remote Podman client did not display the network name associated with network events ([#21311](https://github.com/containers/podman/issues/21311)).
- Fixed a bug where the `podman farm build` did not properly handle the `--tls-verify` option and would override server defaults even if the option was not set by the user ([#21352](https://github.com/containers/podman/issues/21352)).
- Fixed a bug where the `podman inspect` command could segfault on FreeBSD ([#21117](https://github.com/containers/podman/issues/21117)).
- Fixed a bug where Quadlet did not properly handle comment lines ending with a backslash ([#21555](https://github.com/containers/podman/issues/21555)).
- Fixed a bug where Quadlet would sometimes not report errors when malformed quadlet files were present.
- Fixed a bug where Quadlet could hang when given a `.container` file with certain types of trailing whitespace ([#21109](https://github.com/containers/podman/issues/21109)).
- Fixed a bug where Quadlet could panic when generating from Kubernetes YAML containing the `bind-mount-options` key ([#21080](https://github.com/containers/podman/issues/21080)).
- Fixed a bug where Quadlet did not properly strip quoting from values in `.container` files ([#20992](https://github.com/containers/podman/issues/20992)).
- Fixed a bug where the `--publish-all` option to `podman kube play` did not function when used with the remote Podman client.
- Fixed a bug where the `podman kube play --build` command could not build images whose Dockerfile specified an image from a private registry with a self-signed certificate in a `FROM` directive ([#20890](https://github.com/containers/podman/discussions/20890)).
- Fixed a bug where container remove events did not have the correct exit code set ([#19124](https://github.com/containers/podman/issues/19124)).

### API
- A new API endpoint, `/libpod/images/$name/resolve`, has been added to resolve a (potential) short name to a list of fully-qualified image references Podman which could be used to pull the image.
- Fixed a bug where the List API for Images did not properly handle filters and would discard all but the last listed filter.
- Fixed a bug in the Docker Create API for Containers where entries from `/etc/hosts` were copied into create containers, resulting in incompatibility with network aliases.
- Fixed a bug in the Libpod and Docker Exec APIs for Containers which caused incorrect header values to be set when upgrading a connection for an interactive exec session.
- The API bindings have been refactored to reduce code size, leading to smaller binaries ([#17167](https://github.com/containers/podman/issues/17167)).

### Misc
- Failed image pulls will now generate an event including the error.
- The gzip compression library used for sending build contexts, improving performance for remote `podman build`.
- Updated Buildah to v1.35.0
- Updated the containers/image library to v5.30.0
- Updated the containers/storage library to v1.53.0
- Updated the containers/common library to v0.58.0
- Updated the libhvee library to v0.7.0

## 4.9.3
### Features
- The `podman container commit` command now features a `--config` option which accepts a filename containing a JSON-encoded container configuration to be merged in to the newly-created image.

## 4.9.2
### Security
- This release addresses a number of Buildkit vulnerabilities including but not limited to: [CVE-2024-23651](https://github.com/advisories/GHSA-m3r6-h7wv-7xxv), [CVE-2024-23652](https://github.com/advisories/GHSA-4v98-7qmw-rqr8), and [CVE-2024-23653](https://github.com/advisories/GHSA-wr6v-9f75-vh2g).

### Misc
- Updated Buildah to v1.33.5
- Updated the containers/common library to v0.57.4

## 4.9.1
### Bugfixes
- Fixed a bug where the `--rootful` option to `podman machine set` would not set the machine to use the root connection ([#21195](https://github.com/containers/podman/issues/21195)).
- Fixed a bug where podman would crash when running in a containerized environment with `euid != 0` and capabilities set ([#20766](https://github.com/containers/podman/issues/20766)).
- Fixed a bug where the `podman info` command would crash on if called multiple times when podman was running as `euid=0` without `CAP_SYS_ADMIN` ([#20908](https://github.com/containers/podman/issues/20908)).
- Fixed a bug where `podman machine` commands were not relayed to the correct machine on AppleHV ([#21115](https://github.com/containers/podman/issues/21115)).
- Fixed a bug where the `podman machine list` and `podman machine inspect` commands would not show the correct `Last Up` time on AppleHV ([#21244](https://github.com/containers/podman/issues/21244)).

### Misc
- Updated the Mac pkginstaller QEMU to v8.2.1
- Updated Buildah to v1.33.4
- Updated the containers/image library to v5.29.2
- Updated the containers/common library to v0.57.3

## 4.9.0
### Features
- The `podman farm` suite of commands for multi-architecture builds is now fully enabled and documented.
- Add a network recovery service to Podman Machine VMs using the QEMU backend to detect and recover from an inoperable host networking issues experienced by Mac users when running for long periods of time.

### Bugfixes
- Fixed a bug where the HyperV provider for `podman machine` did not forward the API socket to the host machine.
- Fixed a bug where improperly formatted annotations passed to `podman kube play` could cause Podman to panic.
- Fixed a bug where `podman system reset` could fail if non-Podman containers (e.g. containers created by Buildah) were present.

### Misc
- Containers run in `podman machine` VMs now default to a PID limit of unlimited, instead of 2048.

## 4.8.3
### Security
- Fixed [GHSA-45x7-px36-x8w8](https://github.com/advisories/GHSA-45x7-px36-x8w8): CVE-2023-48795 by vendoring golang.org/x/crypto v0.17.0.

## 4.8.2
### Bugfixes
- Fixed a bug in the MacOS pkginstaller where Podman machine was using a different QEMU binary than the one installed using the installer, if it existed on the system ([#20808](https://github.com/containers/podman/issues/20808)).
- Fixed a bug on Windows (WSL) with the first-time install of user-mode networking when using the init command, as opposed to set ([#20921](https://github.com/containers/podman/issues/20921)).

### Quadlet
- Fixed a bug where Kube image build failed when starting service with missing image ([#20432](https://github.com/containers/podman/issues/20432)).

## 4.8.1
### Bugfixes
- Fixed a bug on Windows (WSL) where wsl.conf/resolv.conf was not restored when user-mode networking was disabled after being enabled ([#20625](https://github.com/containers/podman/issues/20625)).
- Fixed a bug where currently if user specifies `podman kube play --replace`, the pod is removed on the client side, not the server side ([#20705](https://github.com/containers/podman/discussions/20705)).
- Fixed a bug where `podman machine rm -f` would cause a deadlock when running with WSL.
- Fixed `database is locked` errors with the new sqlite database backend ([#20809](https://github.com/containers/podman/issues/20809)).
- Fixed a bug where `podman-remote exec` would fail if the server API version is older than 4.8.0 ([#20821](https://github.com/containers/podman/issues/20821)).
- Fixed a bug where Podman would not run any command on systems with a symlinked $HOME ([#20872](https://github.com/containers/podman/issues/20872)).

## 4.8.0
### Features
- Podman machine now supports HyperV as a provider on Windows. This option can be set via the `CONTAINERS_MACHINE_PROVIDER` environment variable, or via containers.conf. HyperV requires Powershell to be run as Admin. Note that running WSL and HyperV machines at the same time is not supported.
- The `podman build` command now supports Containerfiles with heredoc syntax.
- The `podman login` and `podman logout` commands now support a new option, `--compat-auth-file`, which allows for editing Docker-compatible config files ([#18617](https://github.com/containers/podman/issues/18617)).
- The `podman machine init` and `podman machine set` commands now support a new option, `--usb`, which sets allows USB passthrough for the QEMU provider ([#16707](https://github.com/containers/podman/issues/16707)).
- The `--ulimit` option now supports setting -1 to indicate the maximum limit allowed for the current process ([#19319](https://github.com/containers/podman/issues/19319)).
- The `podman play kube` command now supports the `BUILDAH_ISOLATION` environment variable to change build isolation when the `--build` option is set ([#20024](https://github.com/containers/podman/issues/20024)).
- The `podman volume create` command now supports `--opt o=size=XYZ` on tmpfs file systems ([#20449](https://github.com/containers/podman/issues/20449)).
- The `podman info` command for remote calls now reports client information even if the remote connection is unreachable
- Added a new field, `privileged`, to containers.conf, which sets the defaults for the `--privileged` flag when creating, running or exec'ing into a container.
- The `podman kube play` command now supports setting DefaultMode for volumes ([#19313](https://github.com/containers/podman/issues/19313)).
- The `--opt` option to the `podman network create` command now accepts a new driver specific option, `vrf`, which assigns a VRF to the bridge interface.
- A new option `--rdt-class=COS` has been added to the `podman create` and `podman run` commands that enables assigning a container to a Class Of Service (COS). The COS has to be pre-configured based on a pseudo-filesystem created by the *resctrl* kernel driver that enables interacting with the Intel RDT CAT feature.
- The `podman kube play` command now supports a new option, `--publish-all`, which exposes all containerPorts on the host.
- The --filter option now supports `label!=`, which filters for containers without the specified label.

### Upcoming Deprecations
- We are beginning development on Podman 5.0, which will include a number of breaking changes and deprecations. We are still finalizing what will be done, but a preliminary list is below. Please note that none of these changes are present in Podman 4.8; this is a preview of upcoming changes.
- Podman 5.0 will deprecate the BoltDB database backend. Exact details on the transition to SQLite are still being decided - expect more news here soon.
- The containers.conf configuration file will be broken up into multiple separate files, ensuring that it will never be rewritten by Podman.
- Support for the CNI network backend and Cgroups V1 are being deprecated and gated by build tags. They will not be enabled in Podman builds by default.
- A variety of small breaking changes to the REST API are planned, both to improve Docker compatibility and to better support `containers.conf` settings when creating and managing containers.

### Changes
- Podman now defaults to sqlite as its database backend. For backwards compatibility, if a boltdb database already exists on the system, Podman will continue using it.
- RHEL Subscriptions from the host now flow through to quay.io/podman/* images.
- The `--help` option to the `podman push` command now shows the compression algorithm used.
- The remote Podman client’s `commit` command now shows progress messages ([#19947](https://github.com/containers/podman/issues/19947)).
- The `podman kube play` command now sets the pod hostname to the node/machine name when hostNetwork=true in k8s yaml ([#19321](https://github.com/containers/podman/issues/19321)).
- The `--tty,-t` option to the `podman exec` command now defines the TERM environment variable even if the container is not running with a terminal ([#20334](https://github.com/containers/podman/issues/20334)).
- Podman now also uses the `helper_binaries_dir` option in containers.conf to lookup the init binary (catatonit).
- Podman healthcheck events are now logged as notices.
- Podman machines no longer automatically update, preventing accidental service interruptions ([#20122](https://github.com/containers/podman/issues/20122)).
- The amount of CPUs a podman machine uses now defaults to available cores/2 ([#17066](https://github.com/containers/podman/issues/17066)).
- Podman machine now prohibits using provider names as machine names. `applehv`, `qemu`, `wsl`, and `hyperv` are no longer valid Podman machine names

### Quadlet
- Quadlet now supports the `UIDMap`, `GIDMap`, `SubUIDMap`, and `SubGIDMap` options in .container files.
- Fixed a bug where symlinks were not resolved in search paths ([#20504](https://github.com/containers/podman/issues/20504)).
- Quadlet now supports the `ReadOnlyTmpfs` option.
- The VolatileTmpfs option is now deprecated.
- Quadlet now supports systemd specifiers in User and Group keys.
- Quadlet now supports `ImageName` for .image files.
- Quadlet now supports a new option, `--force`, to the stop command.
- Quadlet now supports the `oneshot` service type for .kube files, which allows yaml files without containers.
- Quadlet now supports podman level arguments ([#20246](https://github.com/containers/podman/issues/20246)).
- Fixed a bug where Quadlet would crash when specifying non key-value options ([#20104](https://github.com/containers/podman/issues/20104)).
- Quadlet now removes anonymous volumes when removing a container ([#20070](https://github.com/containers/podman/issues/20070)).
- Quadlet now supports a new unit type, `.image`.

### Bugfixes
- Fixed a bug where mounted volumes on Podman machines on MacOS would have a max open files limit ([#16106](https://github.com/containers/podman/issues/16106)).
- Fixed a bug where setting both the `--uts` and `--network` options to `host` did not fill /etc/hostname with the host's name ([#20448](https://github.com/containers/podman/issues/20448)).
- Fixed a bug where the remote Podman client’s `build` command would incorrectly parse https paths ([#20475](https://github.com/containers/podman/issues/20475)).
- Fixed a bug where running Docker Compose against a WSL podman machine would fail ([#20373](https://github.com/containers/podman/issues/20373)).
- Fixed a race condition where parallel tagging and untagging of images would fail ([#17515](https://github.com/containers/podman/issues/17515)).
- Fixed a bug where the `podman exec` command would leak sessions when the specified command does not existFixed a bug where the `podman exec` command would leak sessions when the specified command does not exist ([#20392](https://github.com/containers/podman/issues/20392)).
- Fixed a bug where the `podman history` command did not display the size of certain layers ([#20375](https://github.com/containers/podman/issues/20375)).
- Fixed a bug where a container with a custom user namespace and `--restart always/on-failure` would not correctly cleanup the netnsm on restart, resulting in leaked ips and network namespaces ([#18615](https://github.com/containers/podman/issues/18615)).
- Fixed a bug where remote calls to the `podman top` command would incorrectly parse options ([#19176](https://github.com/containers/podman/issues/19176)).
- Fixed a bug where the `--read-only-tmpfs` option to the `podman run` command was incorrectly handled when the `--read-only` option was set ([#20225](https://github.com/containers/podman/issues/20225)).
- Fixed a bug where creating containers in parallel may cause a deadlock if both containers attempt to use the same named volume ([#20313](https://github.com/containers/podman/issues/20313)).
- Fixed a bug where a container restarted by the Podman service would occasionally not mount its storage ([#17042](https://github.com/containers/podman/issues/17042)).
- Fixed a bug where the `--filter` option to the `podman images` command would not correctly filter ids, digests, or intermediates ([#19966](https://github.com/containers/podman/issues/19966)).
- Fixed a bug where setting the `--replace` option to the `podman run` command would print both the old and new container ID. Now, only the new container ID is printed.
- Fixed a bug where the `podman machine ls` command would show Creation time as LastUp time for machines that have never been booted. Now, new machines show `Never`, with the json value being ZeroTime.
- Fixed a bug in the `podman build` command where the default pull policy was not set to `missing` ([#20125](https://github.com/containers/podman/issues/20125)).
- Fixed a bug where setting the static or volume directory in `containers.conf` would lead to cleanup errors ([#19938](https://github.com/containers/podman/issues/19938)).
- Fixed a bug where the `podman kube play` command exposed all containerPorts on the host ([#17028](https://github.com/containers/podman/issues/17028)).
- Fixed a bug where the `podman farm update` command did not verify farm and connection existence before updating ([#20080](https://github.com/containers/podman/issues/20080)).
- Fixed a bug where remote Podman calls would not honor the `--connection` option while the `CONTAINER_HOST` environment variable was set. The active destination is not resolved with the correct priority, that is, CLI flags, env vars, ActiveService from containers.conf, RemoteURI ([#15588](https://github.com/containers/podman/issues/15588)).
- Fixed a bug where the `--env-host` option was not honoring the default from containers.conf

### API
- Fixed a bug in the Compat Image Prune endpoint where the dangling filter was set twice ([#20469](https://github.com/containers/podman/issues/20469)).
- Fixed a bug in the Compat API where attempting to connect a container to a network while the connection already exists returned a 200 status code. It now correctly returns a 500 error code.
- Fixed a bug in the Compat API where some responses would not have compatible error details if progress data had not been sent yet ([#20013](https://github.com/containers/podman/issues/20013)).
- The Libpod Pull endpoint now supports a new option, compatMode which causes the streamed JSON payload to be identical to the Compat endpoint.
- Fixed a bug in the Libpod Container Create endpoint where it would return an incorrect status code if the image was not found. The endpoint now correctly returns 404.
- The Compat Network List endpoint should see a significant performance improvement ([#20035](https://github.com/containers/podman/issues/20035)).

### Misc
- Updated Buildah to v1.33.2
- Updated the containers/storage library to v1.51.0
- Updated the containers/image library to v5.29.0
- Updated the containers/common library to v0.57.0
- Updated the containers/libhvee library to v0.5.0
- Podman Machine now runs with gvproxy v0.7.1

## 4.7.2
### Security
- Fixed [GHSA-jq35-85cj-fj4p](https://github.com/moby/moby/security/advisories/GHSA-jq35-85cj-fj4p).

### Bugfixes
- WSL: Fixed `podman compose` command.
- Fixed a bug in `podman compose` to try all configured providers before throwing an error ([#20502](https://github.com/containers/podman/issues/20502)).

## 4.7.1
### Bugfixes
- Fixed a bug involving non-English locales of Windows where machine installs using user-mode networking were rejected due to erroneous version detection ([#20209](https://github.com/containers/podman/issues/20209)).
- Fixed a regression in --env-file handling ([#19565](https://github.com/containers/podman/issues/19565)).
- Fixed a bug where podman inspect would fail when stat'ing a device failed.

### API
- The network list compat API endpoint is now much faster ([#20035](https://github.com/containers/podman/issues/20035)).

## 4.7.0
### Security
- Now the io.containers.capabilities LABEL in an image can be an empty string.

### Features
- New command set: `podman farm [create,list,remove,update]` has been created to "farm" out builds to machines running Podman for different architectures.
- New command: `podman compose` as a thin wrapper around an external compose provider such as docker-compose or podman-compose.
- FreeBSD: `podman run --device` is now supported.
- Linux: Add a new `--module` flag for Podman.
- Podmansh: Timeout is now configurable using the `podmansh_timeout` option in containers.conf.
- SELinux: Add support for confined users to create containers but restrict them from creating privileged containers.
- WSL: Registers shared socket bindings on Windows, to allow other WSL distributions easy remote access ([#15190](https://github.com/containers/podman/issues/15190)).
- WSL: Enabling user-mode-networking on older WSL2 generations will now detect an error with upgrade guidance.
- The `podman build` command now supports two new options: `--layer-label` and `--cw`.
- The `podman kube generate` command now supports generation of k8s DaemonSet kind ([#18899](https://github.com/containers/podman/issues/18899)).
- The `podman kube generate` and `podman kube play` commands now support the k8s `TerminationGracePeriodSeconds` field ([RH BZ#2218061](https://bugzilla.redhat.com/show_bug.cgi?id=2218061)).
- The `podman kube generate` and `podman kube play` commands now support `securityContext.procMount: Unmasked` ([#19881](https://github.com/containers/podman/issues/19881)).
- The `podman generate kube` command now supports a `--podman-only` flag to allow podman-only reserved annotations to be used in the generated YAML file. These annotations cannot be used by Kubernetes.
- The `podman kube generate` now supports a `--no-trunc` flag that supports YAML files with annotations longer than 63 characters. Warning: if an annotation is longer than 63 chars, then the generated yaml file is not Kubernetes compatible.
- An infra name annotation `io.podman.annotations.infra.name` is added in the generated yaml when the `pod create` command has `--infra-name` set. This annotation can also be used with `kube play` when wanting to customize the infra container name ([#18312](https://github.com/containers/podman/issues/18312)).
- The syntax of `--uidmap` and `--gidmap` has been extended to lookup the parent user namespace and to extend default mappings ([#18333](https://github.com/containers/podman/issues/18333)).
- The `podman kube` commands now support the `List` kind ([#19052](https://github.com/containers/podman/issues/19052)).
- The `podman kube play` command now supports environment variables in kube.yaml ([#15983](https://github.com/containers/podman/issues/15983)).
- The `podman push` and `podman manifest push` commands now support the `--force-compression` optionto prevent reusing other blobs ([#18860](https://github.com/containers/podman/issues/18660)).
- The `podman manifest push` command now supports `--add-compression` to push with compressed variants.
- The `podman manifest push` command now honors the `add_compression` field from containers.conf if `--add-compression` is not set.
- The `podman run` and `podman create --mount` commands now support the `ramfs` type ([#19659](https://github.com/containers/podman/issues/19659)).
- When running under systemd (e.g., via Quadlet), Podman will extend the start timeout in 30 second steps up to a maximum of 5 minutes when pulling an image.
- The `--add-host` option now accepts the special string `host-gateway` instead of an IP Address, which will be mapped to the host IP address.
- The `podman generate systemd` command is deprecated.  Use Quadlet for running containers and pods under systemd.
- The `podman secret rm` command now supports an `--ignore` option.
- The `--env-file` option now supports multiline variables ([#18724](https://github.com/containers/podman/issues/18724)).
- The `--read-only-tmpfs` flag now affects /dev and /dev/shm as well as /run, /tmp, /var/tmp ([#12937](https://github.com/containers/podman/issues/12937)).
- The Podman `--mount` option now supports bind mounts passed as globs.
- The `--mount` option can now be specified in containers.conf using the `mounts` field.
- The `podman stats` now has an `--all` option to get all containers stats ([#19252](https://github.com/containers/podman/issues/19252)).
- There is now a new `--sdnotify=healthy` policy where Podman sends the READY message once the container turns healthy ([#6160](https://github.com/containers/podman/issues/6160)).
- Temporary files created when dealing with images in `/var/tmp` will automatically be cleaned up on reboot.
- There is now a new filter option `since` for `podman volume ls` and `podman volume prune` ([#19228](https://github.com/containers/podman/issues/19228)).
- The `podman inspect` command now has tab-completion support ([#18672])(https://github.com/containers/podman/issues/18672)).
- The `podman kube play` command now has support for the use of reserved annotations in the generated YAML.
- The progress bar is now displayed when decompressing a Podman machine image ([#19240](https://github.com/containers/podman/issues/19240)).
- The `podman secret inspect` command supports a new option `--showsecret` which will output the actual secret.
- The `podman secret create` now supports a `--replace` option, which allows you to modify secrets without replacing containers.
- The `podman login` command can now read the secret for a registry from its secret database created with `podman secret create` ([#18667]](https://github.com/containers/podman/issues/18667)).
- The remote Podman client’s `podman play kube` command now works with the `--userns` option ([#17392](https://github.com/containers/podman/pull/17392)).

### Changes
- The `/tmp` and `/var/tmp` inside of a `podman kube play` will no longer be `noexec`.
- The limit of inotify instances has been bumped from 128 to 524288 for podman machine ([#19848](https://github.com/containers/podman/issues/19848)).
- The `podman kube play` has been improved to only pull a newer image for the "latest" tag ([#19801](https://github.com/containers/podman/issues/19801)).
- Pulling from an `oci` transport will use the optional name for naming the image.
- The `podman info` command will always display the existence of the Podman socket.
- The echo server example in socket_activation.md has been rewritten to use quadlet instead of `podman generate systemd`.
- Kubernetes support table documentation correctly show volumes support.
- The `podman auto-update` manpage and documentation has been updated and now includes references to Quadlet.

### Quadlet
- Quadlet now supports setting Ulimit values.
- Quadlet now supports setting the PidsLimit option in a container.
- Quadlet unit files allow DNS field in Network group and DNS, DNSSearch, and DNSOption field in Container group ([#19884](https://github.com/containers/podman/issues/19884)).
- Quadlet now supports ShmSize option in unit files.
- Quadlet now recursively calls in user directories for unit files.
- Quadlet now allows the user to set the service working directory relative to the YAML or Unit files ([17177](https://github.com/containers/podman/discussions/17177)).
- Quadlet now allows setting user-defined names for `Volume` and `Network` units via the `VolumeName` and `NetworkName` directives, respectively.
- Kube quadlets can now support autoupdate.

### Bugfixes
- Fixed an issue where containers were being restarted after a `podman kill`.
- Fixed a bug where events could report incorrect healthcheck results ([#19237](https://github.com/containers/podman/issues/19237).
- Fixed a bug where running a container in a pod didn't fail if volumes or mounts were specified in the containers.conf file.
- Fixed a bug where pod cgroup limits were not being honored after a reboot ([#19175](https://github.com/containers/podman/issues/19175)).
- Fixed a bug where `podman rm -af` could fail to remove containers under some circumstances ([#18874](https://github.com/containers/podman/issues/18874)).
- Fixed a bug in rootless to clamp oom_score_adj to current value if it is too low ([#19829](https://github.com/containers/podman/issues/19829)).
- Fixed a bug where `--hostuser` was being parsed in base 8 instead of base 10 ([#19800](https://github.com/containers/podman/issues/19800)).
- Fixed a bug where `kube down` would error when an object did not exist ([#19711](https://github.com/containers/podman/issues/19711)).
- Fixed a bug where containers created via DOCKER API without specifying StopTimeout had StopTimeout defaulting to 0 seconds ([#19139](https://github.com/containers/podman/issues/19139)).
- Fixed a bug in `podman exec` to set umask to match the container it's execing into ([#19713](https://github.com/containers/podman/issues/19713)).
- Fixed a bug where `podman kube play` failed to set a container's Umask to the default `0022`.
- Fixed a bug to automatically reassign Podman's machine ssh port on Windows when it conflicts with in-use system ports ([#19554](https://github.com/containers/podman/issues/19554)).
- Fixed a bug where locales weren't passed to conmon correctly, resulting in a crash if some characters were specified over CLI ([containers/common/#272](https://github.com/containers/conmon/issues/272)).
- Fixed a bug where `podman top` would sometimes not print the full output ([#19504](https://github.com/containers/podman/issues/19504)).
- Fixed a bug were `podman logs --tail` could return incorrect lines when the k8s-file logger is used ([#19545](https://github.com/containers/podman/issues/19545)).
- Fixed a bug where `podman stop` did not ignore cidfile not existing when user specified --ignore flag ([#19546](https://github.com/containers/podman/issues/19546)).
- Fixed a bug where a container with an image volume and an inherited mount from the `--volumes-from` option that used the same path could not be created ([#19529](https://github.com/containers/podman/issues/19529)).
- Fixed a bug where `podman cp` via STDIN did not delete temporary files ([#19496](https://github.com/containers/podman/issues/19496)).
- Fixed a bug where Compatibility API did not accept timeout=-1 for stopping containers ([#17542](https://github.com/containers/podman/issues/17542)).
- Fixed a bug where `podman run --rmi` did not remove the container ([#15640](https://github.com/containers/podman/issues/15640)).
- Fixed a bug to recover from inconsistent podman-machine states with QEMU ([#16054](https://github.com/containers/podman/issues/16054)).
- Fixed a bug where CID Files on remote clients are not removed when container is removed ([#19420](https://github.com/containers/podman/issues/19420)).
- Fixed a bug in `podman inspect` to show a `.NetworkSettings.SandboxKey` path for containers created with --net=none ([#16716](https://github.com/containers/podman/issues/16716)).
- Fixed a concurrency bug in `podman machine start` using the QEMU provider ([#18662](https://github.com/containers/podman/issues/18662)).
- Fixed a bug in `podman run` and `podman create` where the command fails if the user specifies a non-existent authfile path ([#18938](https://github.com/containers/podman/issues/18938)).
- Fixed a bug where some distributions added extra quotes around the distribution name removed from `podman info` output ([#19340](https://github.com/containers/podman/issues/19340)).
- Fixed a crash validating --device argument for create and run ([#19335](https://github.com/containers/podman/issues/19335)).
- Fixed a bug where `.HostConfig.PublishAllPorts` always evaluates to `false` when inspecting a container created with `--publish-all`.
- Fixed a bug in `podman image trust` command to allow using the local policy.json file ([#19073](https://github.com/containers/podman/issues/19073)).
- Fixed a bug where the cgroup file system was not correctly mounted when running without a network namespace in rootless mode ([#20073](https://github.com/containers/podman/issues/20073)).
- Fixed a bug where the `--syslog` flag was not passed to the cleanup process.

### API
- Fixed a bug with parsing of the pull query parameter for the compat /build endpoint ([#17778](https://github.com/containers/podman/issues/17778)).

### Misc
- Updated Buildah to v1.32.0.

## 4.6.2
### Changes
- Fixed a performance issue when calculating diff sizes in overlay. The `podman system df` command should see a significant performance improvement ([#19467](https://github.com/containers/podman/issues/19467)).

### Bugfixes
- Fixed a bug where containers in a pod would use pod the restart policy over the set container restart policy ([#19671](https://github.com/containers/podman/issues/19671)).

### API
- Fixed a bug in the Compat Build endpoint where the pull query parameter did not parse 0/1 as a boolean ([#17778](https://github.com/containers/podman/issues/17778)).

### Misc
- Updated the containers/storage library to v1.48.1

## 4.6.1
### Quadlet
- Quadlet now selects the first Quadlet file found when multiple Quadlets exist with the same name.

### API
- Fixed a bug in the container kill endpoint to correctly return 409 when a container is not running ([#19368](https://github.com/containers/podman/issues/19368)).

### Misc
- Updated Buildah to v1.31.2
- Updated the containers/common library to v0.55.3

## 4.6.0
### Features
- The `podman manifest inspect` command now supports the `--authfile` option, for authentication purposes.
- The `podman wait` command now supports `--condition={healthy,unhealthy}`, allowing waits on successful health checks.
- The `podman push` command now supports a new option, ` --compression-level`, which specifies the compression level to use ([#18939](https://github.com/containers/podman/issues/18939)).
- The `podman machine start` command, when run with `--log-level=debug`, now creates a console window to display the virtual machine while booting.
- Podman now supports a new option, `--imagestore`, which allows images to be stored in a different directory than the graphroot.
- The `--ip-range` option to the `podman network create` command now accepts a new syntax, `<startIP>-<endIP>`, which allows more flexibility when limiting the ip range that Podman assigns.
- [Tech Preview] A new command, `podmansh`, has been added, which executes a user shell within a container when the user logs into the system. The container that the users get added to can be defined via a Podman Quadlet file. This feature is currently a `Tech Preview` which means it's ready for users to try out but changes can be expected in upcoming versions.
- The `podman network create` command supports a new `--option`, `bclim`, for the `macvlan` driver.
- The `podman network create` command now supports adding static routes using the `--route` option.
- The `podman network create` command supports a new `--option`, `no_default_route` for all drivers.
- The `podman info` command now prints network information about the binary path, package version, program version and DNS information ([#18443](https://github.com/containers/podman/issues/18443)).
- The `podman info` command now displays the number of free locks available, helping to debug lock exhaustion scenarios.
- The `podman info` command now outputs information about pasta, if it exists in helper_binaries_dir or $PATH.
- The remote Podman client’s `podman build` command now accepts Containerfiles that are not in the context directory ([#18239](https://github.com/containers/podman/issues/18239)).
- The remote Podman client’s `podman play kube` command now supports the `--configmap` option ([#17513](https://github.com/containers/podman/issues/17513)).
- The `podman kube play` command now supports multi-doc YAML files for configmap arguments. ([#18537](https://github.com/containers/podman/issues/18537)).
- The `podman pod create` command now supports a new flag, `--restart`, which sets the restart policy for all the containers in a pod.
- The `--format={{.Restarts}}` option to the `podman ps` command now shows the number of times a container has been restarted based on its restart policy.
- The `--format={{.Restarts}}` option to the `podman pod ps` command now shows the total number of container restarts in a pod.
- The podman machine provider can now be specified via the `CONTAINERS_MACHINE_PROVIDER` environment variable, as well as via the `provider` field in `containers.conf` ([#17116](https://github.com/containers/podman/issues/17116)).
- A default list of pasta arguments can now be set in `containers.conf` via `pasta_options`.
- The `podman machine init` and `podman machine set` commands now support a new option, `--user-mode-networking`, which improves interops with VPN configs that drop traffic from WSL networking, on Windows.
- The remote Podman client’s `podman push` command now supports the `--digestfile` option ([#18216](https://github.com/containers/podman/issues/18216)).
- Podman now supports a new option, `--out`, that allows redirection or suppression of STDOUT ([#18120](https://github.com/containers/podman/issues/18120)).

### Changes
- When looking up an image by digest, the entire repository of the specified value is now considered. This aligns with Docker's behavior since v20.10.20. Previously, both the repository and the tag was ignored and Podman looked for an image with only a matching digest. Ignoring the name, repository, and tag of the specified value can lead to security issues and is considered harmful.
- The `podman system service` command now emits a warning when binding to a TCP socket. This is not a secure configuration and the Podman team recommends against using it.
- The `podman top` command no longer depends on ps(1) being present in the container image and now uses the one from the host ([#19001](https://github.com/containers/podman/issues/19001)).
- The `--filter id=xxx` option will now treat `xxx` as a CID prefix, and not as a regular expression ([#18471](https://github.com/containers/podman/issues/18471)).
- The `--filter` option now requires multiple `--filter` flags to specify multiple filters. It will no longer support the comma syntax (`--filter label=a,label=b`).
- The `slirp4netns` binary for will now be searched for in paths specified by the `helper_binaries_dir` option in `containers.conf` ([#18239](https://github.com/containers/podman/issues/18568)).
- Podman machine now updates `/run/docker.sock` within the guest to be consistent with its rootless/rootful setting ([#18480](https://github.com/containers/podman/issues/18480)).
- The `podman system df` command now counts files which podman generates for use with specific containers as part of the disk space used by those containers, and which can be reclaimed by removing those containers. It also counts space used by files it associates with specific images and volumes as being used by those images and volumes.
- The `podman build` command now returns a clearer error message when the Containerfile cannot be found. ([#16354](https://github.com/containers/podman/issues/16354)).
- Containers created with `--pid=host` will no longer print errors on podman stop ([#18460](https://github.com/containers/podman/issues/18460)).
- The `podman manifest push` command no longer requires a destination to be specified. If a destination is not provided, the source is used as the destination ([#18360](https://github.com/containers/podman/issues/18360)).
- The `podman system reset` command now warns the user that the graphroot and runroot directories will be deleted ([#18349](https://github.com/containers/podman/issues/18349)), ([#18295](https://github.com/containers/podman/issues/18295)).
- The `package` and `package-install` targets in Makefile have now been fixed and also renamed to `rpm` and `rpm-install` respectively for clarity ([#18817](https://github.com/containers/podman/issues/18817)).

### Quadlet
- Quadlet now exits with a non-zero exit code when errors are found ([#18778](https://github.com/containers/podman/issues/18778)).
- Rootless podman quadlet files can now be installed in `/etc/containers/systemd/users` directory.
- Quadlet now supports the `AutoUpdate` option.
- Quadlet now supports the `Mask` and `Unmask` options.
- Quadlet now supports the `WorkingDir` option, which specifies the default working dir in a container.
- Quadlet now supports the `Sysctl` option, which sets namespaced kernel parameters for containers ([#18727](https://github.com/containers/podman/issues/18727)).
- Quadlet now supports the `SecurityLabelNetsted=true` option, which allows nested SELinux containers.
- Quadlet now supports the `Pull` option in `.container` files ([#18779](https://github.com/containers/podman/issues/18779)).
- Quadlet now supports the `ExitCode` field in `.kube` files, which reflects the exit codes of failed containers.
- Quadlet now supports `PodmanArgs` field.
- Quadlet now supports the `HostName` field, which sets the container's host name, in `.container` files ([#18486](https://github.com/containers/podman/issues/18486)).

### Bugfixes
- Fixed a bug where the `podman machine start` command would fail with a 255 exit code. It now waits for systemd-user sessions to be up, and for SSH to be ready, addressing the flaky machine starts ([#17403](https://github.com/containers/podman/issues/17403)).
- Fixed a bug where the `podman auto update` command did not correctly use authentication files when contacting container registries.
- Fixed a bug where `--label` option to the `podman volume ls` command would return volumes that matched any of the filters, not all of them  ([#19219](https://github.com/containers/podman/issues/19219)).
- Fixed a bug where the `podman kube play` command did not recognize containerPort names inside Kubernetes liveness probes. Now, liveness probes support both containerPort names as well as port numbers ([#18645](https://github.com/containers/podman/issues/18645)).
- Fixed a bug where the `--dns` option to the `podman run` command was ignored for macvlan networks ([#19169](https://github.com/containers/podman/issues/19169)).
- Fixed a bug in the `podman system service` command where setting LISTEN_FDS when listening on TCP would misbehave.
- Fixed a bug where hostnames were not recognized as a network alias. Containers can now resolve other hostnames, in addition to their names ([#17370](https://github.com/containers/podman/issues/17370)).
- Fixed a bug where the `podman pod run` command would error after a reboot on a non-systemd system ([#19175](https://github.com/containers/podman/issues/19175)).
- Fixed a bug where the `--syslog` option returned a fatal error when no syslog server was found ([#19075](https://github.com/containers/podman/issues/19075)).
- Fixed a bug where the `--mount` option would parse the `readonly` option incorrectly ([#18995](https://github.com/containers/podman/issues/18995)).
- Fixed a bug where hook executables invoked by the `podman run` command set an incorrect working directory. It now sets the correct working directory pointing to the container bundle directory ([#18907](https://github.com/containers/podman/issues/18907)).
- Fixed a bug where the `-device-cgroup-rule` option was silently ignored in rootless mode ([#18698](https://github.com/containers/podman/issues/18698)).
- Listing images is now more resilient towards concurrently running image removals.
- Fixed a bug where the `--force` option to the `podman kube down` command would not remove volumes ([#18797](https://github.com/containers/podman/issues/18797)).
- Fixed a bug where setting the `--list-tags` option in the `podman search` command would cause the command to ignore the `--format` option ([#18939](https://github.com/containers/podman/issues/18939)).
- Fixed a bug where the `podman machine start` command did not properly translate the proxy IP.
- Fixed a bug where the `podman auto-update` command would not restart dependent units (specified via `Requires=`) on auto update ([#18926](https://github.com/containers/podman/issues/18926)).
- Fixed a bug where the `podman pull` command would print ids multiple times when using additional stores ([#18647](https://github.com/containers/podman/issues/18647)).
- Fixed a bug where creating a container while setting unmask option to an empty array would cause the create to fail ([#18848](https://github.com/containers/podman/issues/18848)).
- Fixed a bug where the propagation of proxy settings for QEMU VMs was broken.
- Fixed a bug where the `podman rm -fa` command could fail to remove dependency containers such as pod infra containers ([#18180](https://github.com/containers/podman/issues/18180)).
- Fixed a bug  where ` --tz` option to the `podman create ` and `podman run` commands would not create a proper localtime symlink to the zoneinfo file, which was causing some applications (e.g. java) to not read the timezone correctly.
- Fixed a bug where lowering the ulimit after container creation would cause the container to fail ([#18714](https://github.com/containers/podman/issues/18714)).
- Fixed a bug where signals were not forwarded correctly in rootless containers ([#16091](https://github.com/containers/podman/issues/16091)).
- Fixed a bug where the `--filter volume=` option to the `podman events` command would not display the relevant events ([#18618](https://github.com/containers/podman/issues/18618)).
- Fixed a bug in the `podman wait` command where containers created with the `--restart=always` option would result in the container staying in a stopped state.
- Fixed a bug where the `podman stats` command returned an incorrect memory limit after a `container update`. ([#18621](https://github.com/containers/podman/issues/18621)).
- Fixed a bug in the `podman run` command where the `PODMAN_USERNS` environment variable was not ignored when the `--pod` option was set, resulting in a container created in a different user namespace than its pod ([#18580](https://github.com/containers/podman/issues/18580)).
- Fixed a bug where the `podman run` command would not create the `/run/.containerenv` when the tmpfs is mounted on `/run` ([#18531](https://github.com/containers/podman/issues/18531)).
- Fixed a bug where the `$HOME` environment variable would be configured inconsistently between container starts if a new passwd entry had to be created for the container.
- Fixed a bug where the `podman play kube` command would restart initContainers based on the restart policy of the pod. initContainers should never be restarted.
- Fixed a bug in the remote Podman client’s `build` command where an invalid platform would be set.
- Fixed a bug where the `podman history` command did not display tags ([#17763](https://github.com/containers/podman/issues/17763)).
- Fixed a bug where the `podman machine init` command would create invalid machines when run with certain UIDs ([#17893](https://github.com/containers/podman/issues/17893)).
- Fixed a bug in the remote Podman client’s `podman manifest push` command where an error encountered during the push incorrectly claimed that the error occurred while adding an item to the list.
- Fixed a bug where the `podman machine rm` command would remove the machine connection before the user confirms the removal of the machine ([#18330](https://github.com/containers/podman/issues/18330)).
- Fixed a bug in the sqlite database backend where the first read access may fail ([#17859](https://github.com/containers/podman/issues/17859)).
- Fixed a bug where a podman machine could get stuck in the `starting` state ([#16945](https://github.com/containers/podman/issues/16945)).
- Fixed a bug where running a container with the `--network=container:` option would fail when the target container uses the host network mode. The same also now works for the other namespace options (`--pid`, `--uts`, `--cgroupns`, `--ipc`) ([#18027](https://github.com/containers/podman/issues/18027)).
- Fixed a bug where the `--format {{.State}}` option to the `podman ps` command would display the status rather than the state ([#18244](https://github.com/containers/podman/issues/18244)).
- Fixed a bug in the `podman commit` command where setting a `--message` while also specifying `--format=docker` options would incorrectly warn that setting a message is incompatible with OCI image formats ([#17773](https://github.com/containers/podman/issues/17773)).
- Fixed a bug in the `--format` option to the `podman history` command, where the `{{.CreatedAt}}` and `{{.Size}}` fields were inconsistent with Docker’s output ([#17767](https://github.com/containers/podman/issues/17767)), ([#17768](https://github.com/containers/podman/issues/17768)).
- Fixed a bug in the remote Podman client where filtering containers would not return all matching containers ([#18153](https://github.com/containers/podman/issues/18153)).

### API
- Fixed a bug where the Compat and Libpod Top endpoints for Containers did not correctly report errors.
- Fixed a bug in the Compat Pull and Compat Push endpoints where errors were incorrectly handled.
- Fixed a bug in the Compat Wait endpoint to correctly handle the "removed" condition ([#18889](https://github.com/containers/podman/issues/18889)).
- Fixed a bug in the Compat Stats endpoint for Containers where the `online_cpus` field was not set correctly ([#15754](https://github.com/containers/podman/issues/15754)).
- Fixed a bug in the Compat Build endpoint where the pull field accepted a boolean value instead of a string ([#17778](https://github.com/containers/podman/issues/17778)).
- Fixed a bug where the Compat History endpoint for Images did not prefix the image ID with `sha256:` ([#17762](https://github.com/containers/podman/issues/17762)).
- Fixed a bug in the Libpod Export endpoint for Images where exporting to an oci-dir or a docker-dir format would not export to the correct format ([#15897](https://github.com/containers/podman/issues/15897)).
- The Compat Create endpoint for Containers now supports the `platform` parameter ([#18951](https://github.com/containers/podman/issues/18951)).
- The Compat Remove endpoint for Images now supports the `noprune` query parameter, which ensures that dangling parents of the specified image are not removed
- The Compat Info endpoint now reports running rootless and SELinux enabled as security options.
- Fixed a bug in the Auth endpoint where a nil dereference could potentially occur.

### Misc
- The `podman system service` command is now supported on FreeBSD.
- Updated the Mac pkginstaller QEMU to v8.0.0
- Updated Buildah to v1.31.0
- Updated the containers/storage library to v1.48.0
- Updated the containers/image library to v5.26.1
- Updated the containers/common library to v0.55.2

## 4.5.1
### Security
- Do not include image annotations when building spec. These annotations can have security implications - crun, for example, allows rootless containers to preserve the user's groups through an annotation.

### Quadlet
- Fixed a bug in quadlet to recognize the systemd optional prefix '-'.

### Bugfixes
- Fixed a bug where fully resolving symlink paths included the version number, breaking the path to homebrew-installed qemu files ([#18111](https://github.com/containers/podman/issues/18111)).
- Fixed a bug where Podman was splitting the filter map slightly differently compared to Docker ([#18092](https://github.com/containers/podman/issues/18092)).
- Fixed a bug where running `make package` did not work on RHEL 8 environments ([#18421](https://github.com/containers/podman/issues/18421)).
- Fixed a bug to allow comma separated dns server IP addresses in `podman network create --dns` and `podman network update --dns-add/--dns-drop` ([#18663](https://github.com/containers/podman/pull/18663)).
- Fixed a bug to correctly stop containers created with --restart=always in all cases ([#18259](https://github.com/containers/podman/issues/18259)).
- Fixed a bug in podman-remote logs to cor…
@stale-locking-app stale-locking-app bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Aug 16, 2024
@stale-locking-app stale-locking-app bot locked as resolved and limited conversation to collaborators Aug 16, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. network Networking related issue or feature pasta pasta(1) bugs or features
Projects
None yet
Development

No branches or pull requests