vscode interop with podman-machine distro

[worldofgeese]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

VS Code fails opening a project cloned into a rootless or rootful podman machine as a dev container with permission denied

Steps to reproduce the issue:

1. podman machine init

2. podman machine start

3. wsl -d podman-machine-default

4. git clone https://github.com/microsoft/vscode-remote-try-python

5. Open folder for Python devcontainer in VS Code and click the pop-up offering to re-open in a Development Container

Describe the results you received:

[91263 ms] Start: Run: wsl -d podman-machine-default -e /bin/sh -c cd '/home/user/vscode-remote-try-python' && /bin/sh
[91268 ms] Start: Run in host: id -un
[91365 ms] user
[91366 ms] 
[91366 ms] Start: Run in host: cat /etc/passwd
[91368 ms] Start: Run in host: echo ~
[91369 ms] /home/user
[91369 ms] 
[91370 ms] Start: Run in host: test -x '/home/user/.vscode-remote-containers/bin/6d9b74a70ca9c7733b29f0456fd8195364076dda/node'
[91371 ms] 
[91371 ms] 
[91371 ms] Exit code 1
[91372 ms] Start: Run in host: test -x '/home/user/.vscode-server/bin/6d9b74a70ca9c7733b29f0456fd8195364076dda/node'
[91373 ms] 
[91373 ms] 
[91374 ms] Start: Run in host: test -f '/home/user/.vscode-server/bin/6d9b74a70ca9c7733b29f0456fd8195364076dda/node_modules/node-pty/package.json'
[91375 ms] 
[91375 ms] 
[91375 ms] Start: Run in host: test -f '/home/user/.vscode-remote-containers/dist/vscode-remote-containers-server-0.245.0.js'
[91376 ms] 
[91376 ms] 
[91377 ms] userEnvProbe: loginInteractiveShell (default)
[91377 ms] userEnvProbe shell: /bin/bash
[93382 ms] Start: Run in Host: /bin/sh 
[93385 ms] Start: Run in container: for pid in `cd /proc && ls -d [0-9]*`; do { echo $pid ; readlink /proc/$pid/cwd ; readlink /proc/$pid/ns/mnt ; cat /proc/$pid/stat | tr "
[93672 ms] userEnvProbe is taking longer than 2 seconds. Process tree:
  30667: /bin/bash -lic echo -n e4d74c3926a2087884e2e0ba4db4b707; cat /proc/self/environ; echo -n e4d74c3926a2087884e2e0ba4db4b707 
    30681: /bin/bash /usr/local/bin/enterns 
[101391 ms] userEnvProbe is taking longer than 10 seconds. Avoid waiting for user input in your shell's startup scripts. Continuing.
[101424 ms] Start: Run in Host: podman version --format {{.Server.APIVersion}}
[101457 ms] Error: error creating tmpdir: mkdir /run/user/1000/libpod: permission denied

Describe the results you expected:

VS Code should reopen a project containing a dev container as a dev container successfully.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client:       Podman Engine
Version:      4.2.0
API Version:  4.2.0
Go Version:   go1.16.15
Git Commit:   7fe5a419cfd2880df2028ad3d7fd9378a88a04f4
Built:        Thu Aug 11 16:20:57 2022
OS/Arch:      windows/amd64

Server:       Podman Engine
Version:      4.1.1
API Version:  4.1.1
Go Version:   go1.18.4
Built:        Fri Jul 22 21:05:59 2022
OS/Arch:      linux/amd64

Output of podman info:

host:
  arch: amd64
  buildahVersion: 1.26.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.0-2.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpuUtilization:
    idlePercent: 97.58
    systemPercent: 1.06
    userPercent: 1.36
  cpus: 8
  distribution:
    distribution: fedora
    variant: container
    version: "36"
  eventLogger: file
  hostname: DESKTOP-FKPVPQR
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.57.1-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: journald
  memFree: 5120884736
  memTotal: 16659918848
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.5-1.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 4294430720
  swapTotal: 4294967296
  uptime: 2h 25m 12.64s (Approximately 0.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/user/.local/share/containers/storage
  graphRootAllocated: 1081101176832
  graphRootUsed: 804233216
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 4.1.1
  Built: 1658516759
  BuiltTime: Fri Jul 22 21:05:59 2022
  GitCommit: ""
  GoVersion: go1.18.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.1```

Package info (e.g. output of rpm -q podman or apt list podman):

podman-4.1.1-3.fc36.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

This is neither a yes or a no as it appears the Windows binary is 4.2.0 but the podman machine it spawns is 4.1.1

Additional environment details (AWS, VirtualBox, physical, etc.):
Windows 11 with podman machine loaded into WSL2.

In VS Code the following are set in its settings.json:

    "terminal.integrated.defaultProfile.linux": "bash",
	"remote.containers.dockerPath": "podman",
	"remote.containers.dockerComposePath": "podman-compose"

[mheon]

@n1hility PTAL

[n1hility]

This is most likely happening because vscode is running commands directly via wsl which is not entering the nested namespace with systemd. I have not yet gotten to exploring a vscode specific solution, but it might require "code" being launched within the wsl instance (as opposed to externally). Alternatively if they provide a hook to wrap their launcher, commands can be prefixed with enterns which will run them inside the proper namespace.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@n1hility Any chance to look at this?

[worldofgeese]

Just a thought that WSL's new native support for systemd may be useful here.

[n1hility]

@worldofgeese yes it would. Although it’s only Win 11 atm so we can’t rely on it just yet.

[n1hility]

This is still on my radar it’s just been lower priority than other work. I’ll try to find some cycles soon

[TheComputerM]

Devcontainers with podman would be awesome. Are there any workarounds for this, facing this same issue.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[TheComputerM]

This is a very severe issue on windows, please don't let it be swept under the rug as there are no workarounds.

[n1hility]

@TheComputerM i should be able to look at this this week.

[TheComputerM]

Any updates or workarounds that work with podman v4.4.1?

[n1hility]

Unfortunately the current nested namespace approach we need to run systemd prevents it from working reliably. You can make it work if you edit their launch script, and manually launch it, but launching via the IDE fails, and it does update itself so that solution is temporary. One approach that might be workable, is to bind mount the podman socket from the podman machine distro into /mnt/wsl/podman.sock and then use a different distro for vscode integration, which can access that podman.sock file (although you have to reconfigure DOCKER_HOST in your vscode config) We have a feature request I plan to get to soon which will automate setting up the bind for you, but in the meantime you can just do that manually yourself, (for example by installing a custom systemd service to create the mount). Another option is you can still use the ssh integration instead of the WSL integration with vscode, I did successfully get that working.

In the medium term I think the final resolution to this will be the move to WSL's new systemd integration, which should be compatible with the viscose approach. We were waiting on that becoming more widely available, and that has recently occurred, now all versions of WSL will update to the store version after windows updates are applied, and someone runs wsl --update.

[pstovik]

Limited workaround: we have configured SSH server running in the container (as the container is some DEV purpose only) ... and then connect from VsCode via SSH remote

[maica1]

Still getting this kind of error, will be trying the ssh workaround and running code inside distro.
Im facing this issue rn at my personal win11 and company's win10.