remote containers - podman over ssh not working #4643
Assignees
Labels
bug
Issue identified by VS Code Team member as probable bug
containers
Issue in vscode-remote containers
upstream
Issue identified as 'upstream' component related (exists outside of VS Code Remote)
Comments
|
Are there any additional entries in the dev console? ( |
below is the log output. I've also made a quick screen recording in case more context is needed: https://www.youtube.com/watch?v=w3mT7oM1P40 dev console output:
|
|
and the dev console output with log level |
|
Maybe check if And compare that with |
|
good point, doesn't look like it's reliable: sometimes (rarely) the content of the returned data is |
|
Could you open an issue for Podman for this? We need |
chrmarti
added
bug
|
I'm now working around the stdio issue(s?) and another podman/docker cli incompatibility ( I can now reliably transfer large binary data via stdio: and pipe commands to a container like: In vscode I now get further in the process of connecting to the devcontainer, but still run into an error. When the container doesn't exist yet I get this as part of the initial build/setup: the output of from the dev console log: And when I then close/reopen vscode and try to reconnect to the container at this stage: Any ideas? |
|
|
|
Actually, this is |
|
thanks! ok, my wrapper now looks like this and I'm getting further, but running into another error: some more details: Any more ideas? |
|
some more details: interestingly running the server command manually doesn't lead to an error: from the filesystem: |
|
and when I try to connect to the created container I get: |
|
I think the problem is that |
|
Bit of an aside, but would it be possible to have some documentation on this, ergo, using a combination of a remote host and attaching to a container, and especially for podman in lieu of docker? This is all great stuff and a bit black magic in a way- there's a lot under the hood and, not having used VS Code in a few years, the way it just creates and sorts out a development container with all of .net installed is certainly impressive. I liked it as an IDE before, but a few years ago I was finding some language plugins were a bit immature, and stuff like this is enough to tempt me back for the future. If I can just open an IDE locally and debug an application and dependencies for a language I don't know on a remote server running inside a container as if it were just on my machine, that's seriously impressive stuff, so good work - just unfortunate I still can't quite work out how to do it after ~5 hours. I think there are a lot of variables and like many (and it would seem from the comments, like you too, Christof), I'm new to the podman world and am hitting a tonne of errors at different levels trying to get this to work. Many articles online mention a totally different way of doing things - e.g. loopback of the Docker socket or creating a context, that don't apply for podman, and I believe if I read correctly Code now uses the CLI so these aren't even needed. If someone could consider documenting points like the below, then I'd be really grateful, as it seems other folks will be arriving at this situation increasingly over time as Podman/OCI take hold, and it looks like almost everything is there under the hood but as there are so many ways to go and as the technologies are both evolving quickly it's hard to know what I need to fix to have this working: What is the simplest/best order of steps needed to debug an application running inside a container on a remote host? Do we open a window in SSH to the server and then attach to/create a container if the CLI is being used? Do we need to have podman-remote connected successfully and then go via calling the local binaries? Does the docker.host variable need to be set to point locally or remotely? Does the container need to be created already to appear? Or will Code create this and start remotely? Thanks a lot :) |
|
We have our documentation at https://code.visualstudio.com/docs/remote/containers. This does not include any documentation specific to Podman, but focuses on Docker. |
|
So, I need to connect using the remote-containers features first and foremost, and forget using the remote-SSH beforehand? I've been following this guide this afternoon:
"remote.containers.dockerPath": "podman",
"docker.host":"ssh://user@host" // This is greyed-out and apparently not recognised, but is having a difference somewhere.
I can see near the top of the log, some commands are interacting with the local podman - this may of course be by design, but I know I'm using a newer version locally as my server is still running bionic.: It seems like there are multiple places where this is doing things a bit differently than it's meant to be - unless I'm being a bit thick and am missing something. Am more than happy to contribute documentation (a skeleton at the least) if and when I get there, but it seems I'm still missing something to get this working on podman. Also, whilst a great reference, the article we both linked is quite long, and I think as a common use-case, a briefer tutorial for the basics of creating/attaching to a remote container would be a help to get people up and running; they could then be referred to the existing one for troubleshooting. Thanks |
|
The If Podman-Remote is a Docker-compatible CLI connecting to a remote server, that should work. I imagine you would first setup Podman-Remote to connect to the server you want and then let Remote-Containers use that. Let me know if that's how it works. |
podman supports remote connections over ssh, but there's a bug that lets hang I tried to work around it with a wrapper (see above) and got a bit further than what I posted here, but ran into more issues. (Unfortunately it's been a while and I lost my notes about it. When I find some time I'll try again and update this issue). |
|
That's okay, if I'm doing everything as I should be then I'll take a look into it when I can. Didn't realise the host was just simple environment variables, so can try going forward again with the wrapper. For the time being I've installed docker on a test machine but would be good to have working as I'm deploying with podman eventually so it's lighter. Would be good to have it documented when appears to be working, but in the meantime am happy to share notes etc. I ran into the exec issue with the pipes too. I also had issues connecting different versions of podman, I think it's changed somewhat between 3.0<-->3.1<-->3.2, so some of the issues may have ironed themselves out - but I couldn't verify easily as they weren't talking to each other. Podman-in-Podman didn't quite work for testing purposes, but if I set up a dedicated VM so we have the same version on both sides then I may be able to try with the above (not read and digested it all yet) and see if it's any further forward. |
I use fish these days, so my Bash may be a bit rusty, and SSH adds some complication, but I believe the correct thing would be:
Am just about to try some of this out and see. |
|
So, the below is actually getting quite far for me as a wrapper (Focal and podman 3.2.2 on both sides):
The exec | md5sum loop was stable even if slow at times over SSH, although am using a VM so may be part of it. It would appear to hang but eventually continued, then was fine. And it importantly brought back the same hash. In order for the above to work, I simply set the It would be more sophisticated to parse the SSH url and match against the connections, but given it's a bit of a settings override this was fine and much easier. As my podman is running as rootless, I comment out the username in devcontainer.json so that it runs as root, which translates myself on the remote machine. Otherwise I think it fails to elevate inside the container at the point it creates Now the issue is that it is cycling round and round trying to open some ports. I imagine because podman's pods work differently regarding network space, and/or I know that ports can't be opened on a pod once created, the pod must be recreated, and IIRC there may be workarounds for this in Docker: So I don't know what to do about that right now but that would probably be the immediate next step. VS Code can connect to a running container to start its server and then attach as a debugger, so am I right in thinking that above where it's starting the node process it's expecting some ability to find an unused port and expose it to the host dynamically? |
|
We are using |
|
How is it that the port is exposed using exec? Or is it using some sort of trick like can be achieved with an SSH loopback? I thought generally once a container is up, new ports can't be exposed.
12 Jul 2021 14:59:30 Christof Marti ***@***.***>:
… We are using *docker exec* to access the container and for forwarding the port inside the container. That should all be possible dynamically. Not sure why the connection is closed, but it seems to be closed on the container's side first (*Remote close*) after being successful to some degree (*The client has reconnected.*).
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub[#4643 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/ABGF2AJ23VMVJZ2DO5UYPWLTXLYMFANCNFSM4Y6MF42A].
[data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEgAAABICAYAAABV7bNHAAAAAXNSR0IArs4c6QAAAARzQklUCAgICHwIZIgAAAArSURBVHic7cEBDQAAAMKg909tDjegAAAAAAAAAAAAAAAAAAAAAAAAAAA+DFFIAAEctgHwAAAAAElFTkSuQmCC###24x24:true###][Tracking image][https://github.com/notifications/beacon/ABGF2APY6FPNEO7UUBSMUXDTXLYMFA5CNFSM4Y6MF42KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGRM5OGQ.gif]
|
|
The Remote-Containers extension listens on a local port and when VS Code connects, it uses |
|
Right okay, so the container is the client and the local machine is the server in effect. In that's case I'll have to have a more detailed look.
12 Jul 2021 17:08:15 Christof Marti ***@***.***>:
… The Remote-Containers extension listens on a local port and when VS Code connects, it uses *docker exec* to connect to the port in the container. (VS Code itself does not have any container-related code, only the extension has.)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub[#4643 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/ABGF2AMIYYBWOD7GP3WJW33TXMHO7ANCNFSM4Y6MF42A].
[data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEgAAABICAYAAABV7bNHAAAAAXNSR0IArs4c6QAAAARzQklUCAgICHwIZIgAAAArSURBVHic7cEBDQAAAMKg909tDjegAAAAAAAAAAAAAAAAAAAAAAAAAAA+DFFIAAEctgHwAAAAAElFTkSuQmCC###24x24:true###][Tracking image][https://github.com/notifications/beacon/ABGF2AOBRUUOJQOSB7DPBVDTXMHO7A5CNFSM4Y6MF42KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGRNWPNA.gif]
|
|
Just to clarify: The VS Code UI on your local machine connects to a port on localhost that then uses |
|
Can someone test this with containers/podman#11503? |
|
I'll give it a go when I have a spare minute. |
|
Probably fixed in this PR (containers/podman#11503). VS Code
Server side: macOS side:
|
|
Thanks for the update! Closing as fixed upstream. |
|
Just an FYI, I saw this in the logs for a local container: When I can update, I'll keep an eye out for any similar messages and update if there's anything outstanding or warnings that ought to be taken into account for podman/OCI support going forward. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
I'm trying to use vscode devcontainers with Podman over ssh (Podman has a Mac client that can connect to Linux servers). It works with Docker, but fails with Podman.
I have set up a Podman server (Fedora 33) and configured the Podman client on MacOS to connect to it via ssh:
on MacOS:
In vscode I have a simple
.devcontainer/devcontainer.json. Apart from the usual vanilla exampledevcontainer.jsonfiles I've only changed to use a container volume as the workspace, instead of a bind mount (as to the vscode docs on docker over ssh):In
~/Library/Application Support/Code/User/settings.jsonI've set:When I set
remote.containers.dockerPathtodocker(and make suremyserverhosts a Docker remote server instead of Podman) everything works as expected (vscode creates and connects to a devcontainer on the remote server).When I switch this to a Podman server it doesn't work anymore. Vscode creates a container on the remote host (I can see it with
podman ps), but fails to connect to it. In the startup log output this part loops (until aborted or max time reached):A
podman psshows the container:and I can connect to it fine with
podman exec -ti affectionate_sutherland /bin/bash, etc.This is what it an "execsnoop"-like trace looks like:
(the
<...>is a truncation by the tracing tool)VSCode Version: 1.54.1
Local OS Version: macOS BigSur 11.2.2 x86_64 with Podman 3.0.1 client
Remote OS Version: Fedora 33 x86_64 with Podman 3.0.1 server (prepared for remoting)
Remote Extension/Connection Type: Docker (remote containers / for devcontainer)
Does this issue occur when you try this locally?: No
Does this issue occur when you try this locally and all extensions are disabled?: No
The text was updated successfully, but these errors were encountered: