On macOS Remote-Container (Visual Studio Code) doesn't work

[garam-kim1]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

1. Environment
   MacOS Big Sur 11.5.2

$ uname -a
Darwin ... 20.6.0 Darwin Kernel Version 20.6.0: ...; root:xnu-7195.141.2~5/RELEASE_X86_64 x86_64

2. There was some warning when start machine like podman machine start reports error but seem to run ? #11421, but it works without any problem.

$ podman machine start
INFO[0000] waiting for clients...
INFO[0000] listening tcp://0.0.0.0:7777
INFO[0000] new connection from  to /var/folders/8j/53llg68s7hd2l65nx0x4vvjm0000gp/T/podman/qemu_podman-machine-default.sock
Waiting for VM ...
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]

3. After setting it on vscode (change docker to podman in "docker path"), I tried to run Remote-Container but it keep trying to reconnecting with logs like bellow (SIGKILL repeated).

[5256 ms] Start: Run in container: /bin/sh -c # Watch installed extensions
[5256 ms] Start: Run in container: /bin/sh -c # Watch machine settings
[5343 ms] Port forwarding connection from 49739 > 33369 > 33369 in the container.
[5344 ms] Start: Run in container: /root/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/node -e 
[7866 ms] Port forwarding connection from 49752 > 33369 > 33369 in the container.
[7866 ms] Start: Run in container: /root/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/node -e 
[11316 ms] Port forwarding 49739 > 33369 > 33369: Local close
[11318 ms] Start: Run: podman inspect --type container 8fa306352f5112d6cf5fe2c0001dfeee1e6d16151290505b361030dd908da341
[11343 ms] Port forwarding 49739 > 33369 > 33369 terminated by extension with code null and signal SIGKILL.
[12267 ms] Port forwarding connection from 49760 > 33369 > 33369 in the container.
[12267 ms] Start: Run in container: /root/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/node -e 
[13683 ms] Extensions cache, remote removals: None
[14160 ms] Port forwarding 49760 > 33369 > 33369: Local close
[14173 ms] Port forwarding 49760 > 33369 > 33369 terminated by extension with code null and signal SIGKILL.
[14190 ms] Start: Run: podman inspect --type container 8fa306352f5112d6cf5fe2c0001dfeee1e6d16151290505b361030dd908da341
[15042 ms] Port forwarding connection from 49767 > 33369 > 33369 in the container.
[15043 ms] Start: Run in container: /root/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/node -e 
[17574 ms] Port forwarding 49767 > 33369 > 33369: Local close
[17579 ms] Port forwarding 49767 > 33369 > 33369 terminated by extension with code null and signal SIGKILL.
[17594 ms] Start: Run: podman inspect --type container 8fa306352f5112d6cf5fe2c0001dfeee1e6d16151290505b361030dd908da341
[18509 ms] Port forwarding connection from 49774 > 33369 > 33369 in the container.
[18509 ms] Start: Run in container: /root/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/node -e 
[20703 ms] Port forwarding 49774 > 33369 > 33369: Local close
[20710 ms] Port forwarding 49774 > 33369 > 33369 terminated by extension with code null and signal SIGKILL.
[20728 ms] Start: Run: podman inspect --type container 8fa306352f5112d6cf5fe2c0001dfeee1e6d16151290505b361030dd908da341
[21267 ms] Port forwarding connection from 49782 > 33369 > 33369 in the container.
[21267 ms] Start: Run in container: /root/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/node -e 
[22843 ms] Port forwarding 49782 > 33369 > 33369: Local close
[22850 ms] Port forwarding 49782 > 33369 > 33369 terminated by 

Steps to reproduce the issue:

1. podman machine start

2. Attach to Container in VS Code

3. Repeated message "Reconnecting to Container ...."

Describe the results you received:

Describe the results you expected:
When I using docker, log was like bellow.

Start: Run in container: echo 44951 >'/root/.vscode-server/data/Machine/.devport-3866c3553be8b268c8a7f8c0482c0c0177aa8bfa'
[16038 ms] userEnvProbe taking longer than 2 seconds. Process not found.
[16043 ms] 
[16043 ms] 
[16044 ms] Port forwarding for container port 44951 starts listening on local port.
[16044 ms] Port forwarding local port 44951 to container port 44951
[16047 ms] Start: Run in container: test ! -f '/root/.vscode-server/data/Machine/.onCreateCommandMarker' && set -o noclobber && mkdir -p '/root/.vscode-server/data/Machine' && { > '/root/.vscode-server/data/Machine/.onCreateCommandMarker' ; } 2> /dev/null
[16070 ms] 
[16071 ms] 
[16072 ms] Start: Run in container: test ! -f '/root/.vscode-server/data/Machine/.updateContentCommandMarker' && set -o noclobber && mkdir -p '/root/.vscode-server/data/Machine' && { > '/root/.vscode-server/data/Machine/.updateContentCommandMarker' ; } 2> /dev/null
[16100 ms] 
[16100 ms] 
[16101 ms] Start: Run in container: test ! -f '/root/.vscode-server/data/Machine/.postCreateCommandMarker' && set -o noclobber && mkdir -p '/root/.vscode-server/data/Machine' && { > '/root/.vscode-server/data/Machine/.postCreateCommandMarker' ; } 2> /dev/null
[16102 ms] Start: Run in container: /bin/sh -c # Watch installed extensions
[16103 ms] Start: Run in container: /bin/sh -c # Watch machine settings
[16274 ms] 
[16274 ms] 
[16277 ms] Start: Run in container: # Test for /root/.gitconfig and git
[16295 ms] git not found
[16296 ms] 
[16296 ms] Exit code 1
[16298 ms] Start: Run in container: command -v git >/dev/null 2>&1 && git config --global credential.helper '!f() { /root/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/node /tmp/vscode-remote-containers-3e8a59dc4d30d078284637b9233585f0f4b2a617.js $*; }; f' || true
[16314 ms] 
[16315 ms] 
[16316 ms] Start: Run in container: mkdir -p '/root/.vscode-server/data/Machine' && [ "$(cat '/root/.vscode-server/data/Machine/.postStartCommandMarker' 2>/dev/null)" != '2021-09-03T12:09:34.937612078Z' ] && echo '2021-09-03T12:09:34.937612078Z' > '/root/.vscode-server/data/Machine/.postStartCommandMarker'
[16411 ms] Port forwarding connection from 50334 > 44951 > 44951 in the container.
[16427 ms] Start: Run in container: /root/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/node -e 
[16464 ms] 
[16464 ms] 
[18301 ms] [12:10:06] Extension host agent started.
[20698 ms] Port forwarding connection from 50404 > 44951 > 44951 in the container.
[20699 ms] Start: Run in container: /root/.vscode-server/bin/3866c3553be8b268c8a7f8c0482c0c0177aa8bfa/node -e 
[20738 ms] [12:10:09] [::ffff:127.0.0.1][439eb48f][ManagementConnection] New connection established.
[22962 ms] [12:10:11] [::ffff:127.0.0.1][fb6c5cba][ExtensionHostConnection] New connection established.
[23040 ms] [12:10:11] [::ffff:127.0.0.1][fb6c5cba][ExtensionHostConnection] <287> Launched Extension Host Process.
[23759 ms] Extensions cache, remote removals: None
[80636 ms] Start: Run in container: /bin/sh -c # Watch installed extensions
[83775 ms] Extensions cache, remote removals: None

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Client:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.17
Built:        Tue Aug 31 04:15:26 2021
OS/Arch:      darwin/amd64

Server:
Version:      3.3.0
API Version:  3.3.0
Go Version:   go1.16.6
Built:        Sat Aug 21 04:36:14 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: localhost
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.13.12-200.fc34.x86_64
  linkmode: dynamic
  memFree: 1660219392
  memTotal: 2061852672
  ociRuntime:
    name: crun
    package: crun-0.21-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.21
      commit: c4c3cdf2ce408ed44a9e027c618473e6485c635b
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc34.x86_64
    version: |-
      slirp4netns version 1.1.8+dev
      commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 1m 38.2s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.0
  Built: 1629488174
  BuiltTime: Fri Aug 20 19:36:14 2021
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.3.0

[mheon]

This should be the same issue as #11389 - Podman is operating via SSH, meaning there's no API socket on the Mac, so VS Code isn't able to connect.

@baude @ashley-cui @Luap99 @jwhonce Seems like we need a podman machine command to create the local socket - I don't think we can count on all consumers of the Docker API supporting SSH as a transport.

[Luap99]

I agree we need a local socket.

[tnk4on]

related to issue:

• remote containers - podman over ssh not working microsoft/vscode-remote-release#4643

[tnk4on]

Added a comment to the following Issue.
microsoft/vscode-remote-release#4643

Fixed in this PR(#11503): It is now possible to attach a DevContainer on Podman via SSH remote connection from macOS.
However, this is currently not available on the podman machine, because it requires fixing Podman on the Server side.

[rhatdan]

We hope to update a bug fix release for Podman next week, mainly to fix podman-machine issues. Will work with fedora coreos to get this updated quickly.

[tnk4on]

As a proof of concept, I ran Podman on Fedora running on lima and got Remote-Container working fine.
When the bug fix release comes out, just change the podman system connection settings and it will work for us on podman machine.
https://gist.github.com/tnk4on/b7b26a3970cfc96852f6ac4c592a8a6e

[Luap99]

Since this is fixed on main I close this issue.