MacOS helper daemon (podman-mac-helper) fails to start and "mount" /var/run/docker.sock

[fogfish]

/kind bug

Description

podman-mac-helper daemon fails to start after machine is started/stopped. It is required to remove podman.sock from $HOME/.local/share/containers/podman/machine/podman-machine-default/podman.sock to make it working.

Steps to reproduce the issue:

1. podman machine init && podman machine start
2. Do something with podman e.g. start some containers
3. podman machine stop
4. podman machine start
5. sudo /usr/local/bin/podman-mac-helper install

Describe the results you received:

The agent fails to start and fails to create /var/run/docker.sock. Trying to start the agent with sudo launchctl start com.github... fails as well. System logs does not provide any inside on the issue.
The helper daemon starts to work only when

1. podman machine stop
2. rm $HOME/.local/share/containers/podman/machine/podman-machine-default/podman.sock
3. podman machine start

Describe the results you expected:

The helper daemon would work out of the box.

Additional information you deem important (e.g. issue happens only occasionally):
n/a

Output of podman version:

Client:       Podman Engine
Version:      4.0.1
API Version:  4.0.1
Go Version:   go1.17.3
Git Commit:   c8b9a2e3ec3630e9172499e15205c11b823c8107-dirty
Built:        Wed Mar  2 11:49:02 2022
OS/Arch:      darwin/amd64

Server:       Podman Engine
Version:      4.0.1
API Version:  4.0.1
Go Version:   go1.16.14

Built:      Fri Feb 25 19:05:11 2022
OS/Arch:    linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    variant: coreos
    version: "35"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.18-200.fc35.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1674285056
  memTotal: 2061381632
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.2-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.2
      commit: f6fbc8f840df1a414f31a60953ae514fa497c748
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 12m 32.98s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 10
    paused: 0
    running: 0
    stopped: 10
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 30
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.1
  Built: 1645808711
  BuiltTime: Fri Feb 25 19:05:11 2022
  GitCommit: ""
  GoVersion: go1.16.14
  OsArch: linux/amd64
  Version: 4.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

n/a

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

No, troubleshooting article exists.

Additional environment details (AWS, VirtualBox, physical, etc.):

MacOS

[mheon]

@baude PTAL

[baude]

@n1hility the helper came from you correct? mind peeking at this issue?

[n1hility]

sure will take a look and follow up with any fix

[n1hility]

/assign

[n1hility]

@fogfish are you sure that the rm is necessary? So far I am having trouble reproducing. It is required to restart the machine after the agent is installed (sorry instructions could be improved there) as the agent only creates it on demand when requested by machine start

[tricktron]

@fogfish Did you try this with the latest version from master?

I am asking because my pr from 2 day ago #13442 now removes the $HOME/.local/share/containers/podman/machine/podman-machine-default/podman.sock file when running podman machine rm.

[fogfish]

@n1hility roger! rm is optional, restart of machine helps.

@tricktron unfortunately, I've tried only 4.0.1 tag but I can try the master as well.

I think the problem is resolved now. It was not obvious about needs to restart machine after installation of agent. Thank you for confirming it!