[Bug]: After recreating Podman machine containers don't answer to host

[shikigami12]

Issue Description

On Windows 10, after reinitialising, Podman machine containers stop responding on mapped ports. I have a compose file that I use in my system to run services. When I boot the machine and run a docker-compose up -d all works fine, containers answering on their ports, but if remove the Podman machine and init, and start it then run docker-compose up -d containers stop responding on their ports.

Steps to reproduce the issue

Steps to reproduce the issue

1. Boot the Windows PC
2. start Podman machine
3. run docker-compose up -d
4. run podman machine stop
5. run podman machine rm
6. run podman machine init --rootful
7. run podman machine start
8. run docker-compose up -d

Describe the results you received

containers with mapped ports eg. 80:8080,5432:5432 stops responding from the host machine

Describe the results you expected

containers proceed responding on mapped ports

podman info output

host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.5-1.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: '
  cpuUtilization:
    idlePercent: 80.87
    systemPercent: 8.07
    userPercent: 11.06
  cpus: 2
  distribution:
    distribution: fedora
    variant: container
    version: "36"
  eventLogger: journald
  hostname: MP1W66QB
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.10.102.1-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: journald
  memFree: 1076490240
  memTotal: 4114989056
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.7.2-3.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.7.2
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 1h 19m 44.00s (Approximately 0.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 12
    paused: 0
    running: 9
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 269490393088
  graphRootUsed: 2227109888
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 8
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1668180253
  BuiltTime: Fri Nov 11 17:24:13 2022
  GitCommit: ""
  GoVersion: go1.18.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

WSL2

Additional information

No response

[davidvoit]

Are you running podman machine as rootful?

Sine the update to version 4.4.0 I'm getting the following error:

> podman.exe machine start
Starting machine "podman-machine-default"

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility

        podman machine set --rootful

API forwarding listening on: npipe:////./pipe/docker_engine

Docker API clients default to this address. You do not need to set DOCKER_HOST.
Machine "podman-machine-default" started successfully
> podman.exe run -ti --rm ubi8
Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:649e5534d134eac0f942db1db2a7491173c90e40888923ebf261d0277c2ab73e
Copying config sha256:6a2ef33ab97f171b57b06cba98a306d4b78f5a0604f576fa5d7b0c5e76481d73
Writing manifest to image destination
Storing signatures
Resource limits are not supported and ignored on cgroups V1 rootless systems
[root@ffdb5cb15a58 /]# exit
exit
> podman.exe machine stop
Machine "podman-machine-default" stopped successfully
> podman machine set --rootful
> podman.exe machine start
Starting machine "podman-machine-default"
API forwarding listening on: npipe:////./pipe/docker_engine

Docker API clients default to this address. You do not need to set DOCKER_HOST.
Machine "podman-machine-default" started successfully
> podman.exe run -ti --rm ubi8
Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:649e5534d134eac0f942db1db2a7491173c90e40888923ebf261d0277c2ab73e
Copying config sha256:6a2ef33ab97f171b57b06cba98a306d4b78f5a0604f576fa5d7b0c5e76481d73
Writing manifest to image destination
Storing signatures
Error: preparing container d93cc2f67b19b5297a76bf1cb037b1c2517b4657ce37dd426037962b2857c8e8 for attach: crun: writing file `pids.max`: Invalid argument: OCI runtime error

if this is not related than I will create a new issue

[shikigami12]

Are you running podman machine as rootful?

Sine the update to version 4.4.0 I'm getting the following error:

> podman.exe machine start
Starting machine "podman-machine-default"

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility

        podman machine set --rootful

API forwarding listening on: npipe:////./pipe/docker_engine

Docker API clients default to this address. You do not need to set DOCKER_HOST.
Machine "podman-machine-default" started successfully
> podman.exe run -ti --rm ubi8
Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:649e5534d134eac0f942db1db2a7491173c90e40888923ebf261d0277c2ab73e
Copying config sha256:6a2ef33ab97f171b57b06cba98a306d4b78f5a0604f576fa5d7b0c5e76481d73
Writing manifest to image destination
Storing signatures
Resource limits are not supported and ignored on cgroups V1 rootless systems
[root@ffdb5cb15a58 /]# exit
exit
> podman.exe machine stop
Machine "podman-machine-default" stopped successfully
> podman machine set --rootful
> podman.exe machine start
Starting machine "podman-machine-default"
API forwarding listening on: npipe:////./pipe/docker_engine

Docker API clients default to this address. You do not need to set DOCKER_HOST.
Machine "podman-machine-default" started successfully
> podman.exe run -ti --rm ubi8
Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:649e5534d134eac0f942db1db2a7491173c90e40888923ebf261d0277c2ab73e
Copying config sha256:6a2ef33ab97f171b57b06cba98a306d4b78f5a0604f576fa5d7b0c5e76481d73
Writing manifest to image destination
Storing signatures
Error: preparing container d93cc2f67b19b5297a76bf1cb037b1c2517b4657ce37dd426037962b2857c8e8 for attach: crun: writing file `pids.max`: Invalid argument: OCI runtime error

if this is not related than I will create a new issue

Yep, I running it as rootful.

[Luap99]

@shikigami12 Just a guess but I assume this is not related to recreating the machine but rather the fact that you switch to rootful mode? Am I right?

[shikigami12]

@Luap99 It doesn't matter whether run I podman machine in rootful or rootless state, after recreating the machine and running containers they stop responding on the host machine.

Here is a part of my bootstrap script in PowerShell:

Write-Host "Clear podman machine"
podman machine stop
podman machine rm
podman machine init --now
Write-Host "Podman machine reinitialized" -ForegroundColor Green

docker-compose pull
docker-compose up -d

and on the service in docker-compose.yml

  redis-queue:
    profiles:
      - services
      - debezium
    container_name: redis-queue
    image: docker.io/library/redis:7.0.8
    command:
      - redis-server
      - /usr/local/etc/redis/redis.conf
    environment:
      - LOGLEVEL=warning
    ports:
      - 6479:6479

Today I tried to start it with rootful mode and rootless, but no luck. Every time, I need to reboot my PC to receive some response from the container.

[davidvoit]

If you are running the following command

podman.exe run -ti --rm ubi8

crun: writing file `pids.max`: Invalid argument: OCI runtime error

Are you getting the same error as me?

If not these are seperate issues

[shikigami12]

If you are running the following command

podman.exe run -ti --rm ubi8

crun: writing file `pids.max`: Invalid argument: OCI runtime error

Are you getting the same error as me?

If not these are seperate issues

The issue the same for me:

Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:649e5534d134eac0f942db1db2a7491173c90e40888923ebf261d0277c2ab73e
Copying config sha256:6a2ef33ab97f171b57b06cba98a306d4b78f5a0604f576fa5d7b0c5e76481d73
Writing manifest to image destination
Storing signatures
Error: preparing container 2efe819c9a4b2021c87d76659a25f776b29ff9b46e800d4c18b9d7abb7c54d2e for attach: crun: writing file `pids.max`: Invalid argument:
 OCI runtime error

[Luap99]

@n1hility PTAL

[n1hility]

Will take a look. The problem fixed by #17262 might be the culprit (didnt make 4.4.0). (Updated fixed link)

[n1hility]

Ok, sorry about the issue. I can confirm this will be fixed once #17355 is merged and we can cut a new 4.4.1+ release

In the meantime, to workaround, I recommend either downgrading to 4.3.1 or alternatively, you can add --pids-limit -1.

[shikigami12]

thanks

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[n1hility]

This was fix in 4.4