Container samples not working using podman

[mjbvz]

Refs #2854

repro

1. Using podman 1.9
2. Run Containers: Try a sample and select Python

Bug
The container fails to start.

Logs:

[0 ms] Start: Resolving remote
[2 ms] Start: Check Docker is running
[2 ms] Start: Run: docker info
[366 ms] Start: Run: docker volume create --label vsch.local.repository=https://github.com/Microsoft/vscode-remote-try-python --label vsch.local.repository.unique=false vsc-remote-sample
[473 ms] Start: Run: docker ps -q -a --filter label=vsch.local.repository=https://github.com/Microsoft/vscode-remote-try-python --filter label=vsch.local.repository.volume=vsc-remote-sample --filter label=vsch.local.repository.folder=vscode-remote-try-python --filter label=vsch.quality=insider
[590 ms] Start: Run: docker build -f /home/parallels/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.113.0/scripts/volumeBootstrap.Dockerfile -t vsc-volume-bootstrap /home/parallels/.vscode-insiders/extensions/ms-vscode-remote.remote-containers-0.113.0/scripts
STEP 1: FROM alpine:3.11.2
STEP 2: RUN apk add --no-cache  nodejs  git     openssh-client  docker-cli      ;
--> Using cache 0faec7dc36e550fd8340d295a87eac456ae140b80a203267ccd6b6f0e0a2c286
STEP 3: COMMIT vsc-volume-bootstrap
--> 0faec7dc36e
0faec7dc36e550fd8340d295a87eac456ae140b80a203267ccd6b6f0e0a2c286
[731 ms] Cloning Github repository: Microsoft/vscode-remote-try-python into /workspaces/vscode-remote-try-python

[731 ms] Start: Run: docker run -d --mount src=vsc-remote-sample,dst=/workspaces,type=volume,volume-driver=local -v /var/run/docker.sock:/var/run/docker.sock vsc-volume-bootstrap sleep infinity```

And the specific error:

$ docker run -d --mount src=vsc-remote-sample,dst=/workspaces,type=volume,volume-driver=local -v /var/run/docker.sock:/var/run/docker.sock vsc-volume-bootstrap sleep infinity
Error: "src=vsc-remote-sample,dst=/workspaces,type=volume,volume-driver=local": incorrect mount format: should be --mount type=<bind|tmpfs|volume>,[src=<host-dir|volume-name>,]target=<ctr-dir>[,options]

[chrmarti]

With Docker we mount the Docker socket into a container to build the image from a volume. We need to investigate if something similar can be done with Podman. (Seeing containers/podman#4056 and containers/podman#4131 discussions.)

[chrmarti]

/cc @egamma

[Chuxel]

@egamma @chrmarti Related question: Given the challenges with the repo approach that people are experiencing yet, should we have the getting started documentation reference the local clone based approach for try repos still while we iron out the kinks?

[chrmarti]

@Chuxel Makes sense, although I wouldn't consider this Podman issue a blocker. It would be good to have a fix for the Docker case where the user gets stuck with a bad config (#2428). I think @egamma mentioned looking into fixing that, so a solution might not be too far out.

[smekkley]

I created this issue in libpod. It's technically possible to control podman from container through ssh varlink but there is a new api server. I'd say we can wait until podman api is complete, then vscode sample container can have podman inside access it through unix socket or tcp socket.
containers/podman#6219

[chrmarti]

Do we need to go through an API server? My understanding of Podman is that it works without a server, so the API server shouldn't add anything beyond giving access to the local libpod. Maybe we can access that from the container directly if we give the container the required privileges.

The reason we need this is that we want to build a Docker image from the sources we clone into a Docker volume. The container for doing this is temporary.

[smekkley]

API is just a way to get out of podman container to do what the current docker solution in vscode is basically doing, it's not actually a container in container. This is already supported by varlink or api(in the future) in podman.

If building container in container is the only requirement, that's also supported but you have to use buildah with chroot isolation mode with vfs inside podman.
This has been supported for a while. It can be done inside rootless podman or docker as long as buildah is present inside the container.

[smekkley]

I wonder if there has been any decision made on this topic.
Mounting docker socket is really bad imo. I think that should be solved first and then support podman properly.

[chrmarti]

Podman in Podman appears to work now: containers/podman#4056 (comment)

[dentudave]

I'm unable to get the sample containers to run using podman, it's failing at the same step, but producing a slightly different error.

repro

1. podman 2.2.1
2. fedora 32 and fedora 33 live usb (checked to make sure it wasn't something I had done to my system)
3. sudo ln -s /usr/bin/podman /usr/bin/docker
4. Run Containers: Try a sample and select Node

[2020-12-18T12:30:51.671Z] [PID 5289] [2 ms] Remote-Containers 0.154.1 in VS Code 1.52.0 (940b5f4bb5fa47866a54529ed759d95d09ee80be).
[2020-12-18T12:30:51.687Z] [PID 5289] [18 ms] Start: Resolving Remote
[2020-12-18T12:30:51.692Z] [PID 5289] [23 ms] Start: Check Docker is running
[2020-12-18T12:30:51.692Z] [PID 5289] [23 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[2020-12-18T12:30:51.833Z] [PID 5289] [164 ms] Server API version: 2.1.0
[2020-12-18T12:30:51.835Z] [PID 5289] [166 ms] Start: Run: docker volume ls -q
[2020-12-18T12:30:51.998Z] [PID 5289] [329 ms] Start: Run: docker ps -q -a --filter label=vsch.local.repository=https://github.com/Microsoft/vscode-remote-try-node --filter label=vsch.local.repository.volume=vsc-remote-sample --filter label=vsch.local.repository.folder=vscode-remote-try-node --filter label=vsch.quality=stable
[2020-12-18T12:30:52.161Z] [PID 5289] [492 ms] Start: Run: docker build -f /home/userName/.vscode/extensions/ms-vscode-remote.remote-containers-0.154.1/scripts/volumeBootstrap.Dockerfile -t vsc-volume-bootstrap /home/userName/.vscode/extensions/ms-vscode-remote.remote-containers-0.154.1/scripts
[2020-12-18T12:30:52.315Z] [PID 5289] STEP 1: FROM alpine:3.11.2
[2020-12-18T12:30:52.355Z] [PID 5289] STEP 2: RUN apk add --no-cache 	nodejs 	git 	openssh-client 	docker-cli 	docker-compose 	;
[2020-12-18T12:30:52.363Z] [PID 5289] --> Using cache d67eafcad1febe41f12f14c01fd80d99bd6ebd09cd23a4535165aeb71e5960a1
STEP 3: COMMIT vsc-volume-bootstrap
[2020-12-18T12:30:52.364Z] [PID 5289] 
[2020-12-18T12:30:52.435Z] [PID 5289] --> d67eafcad1f
[2020-12-18T12:30:52.461Z] [PID 5289] d67eafcad1febe41f12f14c01fd80d99bd6ebd09cd23a4535165aeb71e5960a1
[2020-12-18T12:30:52.487Z] [PID 5289] [818 ms] Cloning Github repository: Microsoft/vscode-remote-try-node into /workspaces/vscode-remote-try-node

[2020-12-18T12:30:52.488Z] [PID 5289] [818 ms] Start: Run: docker run -d --mount type=volume,src=vsc-remote-sample,dst=/workspaces -v /var/run/docker.sock:/var/run/docker.sock vsc-volume-bootstrap sleep infinity
[2020-12-18T12:30:52.678Z] [PID 5289] [1009 ms] Command failed: docker run -d --mount type=volume,src=vsc-remote-sample,dst=/workspaces -v /var/run/docker.sock:/var/run/docker.sock vsc-volume-bootstrap sleep infinity

I've also tried it with the podman-docker package as denoted in #116 it gave the same result.

I'm willing to make a new bug ticket since my problem seems slightly different and this issue is rather broad.

[PavelSosin-320]

@chrmarti Sure, using docker socket is not a good idea. Podman is CLI 1st and doesn't need any server, any client, and any Network security if used locally in one computer or private network! Everything could be done using ssh command or wsl command. Dockerod is redundant too because the object model can be generated from the published API spec. and its connectivity layer is redundant (see above).
Most required for dev-container features can be provided by rootless containers, i.e. without root login.
In the fight of provided features, Podman wins by knockout: 5 sec per extension toolset (node, python,etc.) installation without build.

[matt-froggatt]

I seem to be able to fix the above error using systemctl enable --now podman.socket, which allows for podman to essentially emulate the docker socket. I still seem to get the following error though Command in container failed: mkdir -p '/root/.vscode-server/bin' && ln -s '/vscode/vscode-server/bin/x64/ea3859d4ba2f3e577a159bc91e3074c5d85c0523' '/root/.vscode-server/bin/ea3859d4ba2f3e577a159bc91e3074c5d85c0523'. I believe that error has something to do with the way that Podman handles file permissions in the container differently from Docker, but I don't know enough about containers to give a good answer.

There is a slight workaround for the issue, where if you remove the "remoteUser": "vscode" line in the devcontainer.json file, you can access the container, but it means you are the root user in the container.

[smekkley]

According to @chrmarti , docker socket is mounted in order to build an image alone. With buildah chroot mode, no socket is required. You can just build and export it as tgz and load it from the host container engine.
I'd say docker-in-docker should be fixed first. Giving the host root access to some vscode sample container by default is really bad. Then it'll work automatically with podman without special patch dedicated for podman.

[PavelSosin-320]

Buildah is CLI - it doesn't need any socket. or server. The problem with Podman and Buildah that each one has its own Filesystem tree. The image built by Buildah has to be pushed to somewhere or buildah from is executed to create container in the Podman container storage. Then run by Podman. It made me a headache. Skopeo moves images between repositories and inspects images without a pull. Everything worked for me fine only if there is a V2 OCI container registry in between (their friend JFrog).
To install the latest version of all three the Ubuntu 20.10 is required or the corresponding WSL distro with the genie.

[smekkley]

You can export it as tgz or tar so that you can load from any container engine.
I wanted to suggest this solution because I felt like we are neglecting the issue with the current implementation with docker by fixing solution only for podman.

[chrmarti]

According to @chrmarti , docker socket is mounted in order to build an image alone. With buildah chroot mode, no socket is required. You can just build and export it as tgz and load it from the host container engine.

Exporting and importing a tgz will add to the startup time. Can we instead mount Podman's local image folder and just build the image for that directly?

I'd say docker-in-docker should be fixed first. Giving the host root access to some vscode sample container by default is really bad. Then it'll work automatically with podman without special patch dedicated for podman.

To clarify: Docker's socket is mounted into a temporary container controlled by the extension, this container is not the sample container.

[PavelSosin-320]

Using sockets for Buildah and Podman is meaningless. Both are CLI development tools. Podman socket is used only to use Podman remotely. The real "remote Podman" is Kubernetes. Kubernetes and Docker have divorced. Buildah can run on the "bare" Linux OS or Linux VM. The alternative is Running Buildah in container.
The attempts to mimic Docker using Podman in the development environment adds complexity but no new feature.

[smekkley]

You might be able to bind mount the whole docker/podman image directory, but then it's no different from mounting docker socket for no good reason(the current solution). The current solution might be good enough for environment where docker is running on different filesystem like docker-for-mac, but not for native linux.

I suggest reading the link that @PavelSosin-320 provided. @chrmarti .

[PavelSosin-320]

@matt-froggatt Do you suggest to run Podman as the Docker engine using systemctl enable --now podman.socket ? It will revert all Podman benefits back to the Docker limitations: The Podman socket will restrict access to the Podman to only Rootfull users (root, wheel ) or users that belong to the Docker group. The Podman listener service has to be started during "boot" or externally. Instead of plain Linux security users will have to deal with Proxies, Network security, Ports forwarding over the network. Special efforts will be required to handle unique Docker repository protocol instead of free choice of repository transport.
Everywhere when Linux runs or Linux VM is available Podman, Buildah, Skopeo can natively. without "proxy" software like Dockerod.
I believe that even standalone Buildah can do everything needed for dev-container.
Build the image by Buildah
Create Container by Buildah.
Run the Container by Buildah
If using the Container locally or locally running VM is enough for you.

[leighmcculloch]

I seem to be able to fix the above error using systemctl enable --now podman.socket

@matt-froggatt Could you expand on this work around?

[Sangeppato]

@chrmarti sorry to ping you on this, but according to this discussion it looks like the temporary container should be launched with either --privileged or --security-opt=label=disable in order to access the socket. Is it possible to pass a runtime flag to the temporary container in any way?
I've tried adding those args to the "runArgs" property in the devcontainer.json file, but my understanding is that this would only act on the final container being built and not on the temporary one used to build it (which is where the issue actually lies).

[chrmarti]

There is no way to control this currently. We should probably detect that we are using Podman and then add one of these flags. Thanks!

[DarkBitz]

In my case, I want to use the Alpine image provided by vscode the error with
[2022-10-26T19:10:38.764Z] Error: statfs /Users/darkbitz/git/devcontainertest: no such file or directory
goes away after following the instruction of the podman CLI output

The system helper service is not installed; the default Docker API socket
address can't be used by podman. If you would like to install it run the
following commands:

        sudo /opt/homebrew/Cellar/podman/4.3.0/bin/podman-mac-helper install
        podman machine stop; podman machine start

The leftover bug is

[2022-10-26T19:22:11.806Z] Stop (2110 ms): Installing VS Code Server
[2022-10-26T19:22:11.806Z] Start: Run in container: mkdir -p '/root/.vscode-server/bin' && ln -snf '/vscode/vscode-server/bin/alpine-arm64/d045a5eda657f4d7b676dedbfa7aab8207f8a075' '/root/.vscode-server/bin/d045a5eda657f4d7b676dedbfa7aab8207f8a075'
[2022-10-26T19:22:11.808Z] 
[2022-10-26T19:22:11.808Z] mkdir: cannot create directory ‘/root’: Permission denied
[2022-10-26T19:22:11.808Z] Exit code 1

which can be solved with

ENV HOME /home/vscode

as last statement in the Dockefile
see #174 (comment)

After this the container could started but the modifications in files could not be saved because of permission denied errors :(

[chrmarti]

--userns=keep-id might help, see #6759.

[bam80]

Any progress on this?

[tnk4on]

I opened new issue. see containers/podman#21548