Cannot attach to container running on remote ssh host

[msalvaris]

When trying to connect to remote container I end up with the following

I am able to connect to remote host with vscode and I am also able to connect to locally running containers

Version: 1.42.0 (user setup)
Commit: ae08d5460b5a45169385ff3fd44208f431992451
Date: 2020-02-06T10:51:34.058Z
Electron: 6.1.6
Chrome: 76.0.3809.146
Node.js: 12.4.0
V8: 7.6.303.31-electron.0
OS: Windows_NT x64 10.0.18363

• Remote OS Version: Ubuntu 16.04
• Remote Extension/Connection Type: Docker

Steps to Reproduce:

1. Try to attach to running container on remote host (Ubuntu)

Does this issue occur when you try this locally?: No
Does this issue occur when you try this locally and all extensions are disabled?: No

[msalvaris]

Also quickly checked whether the port is open and was able to transmit messages using ncat. So don't think it is a port issue

[msalvaris]

Tunneling through seems to work so will use that workaround for now

[msalvaris]

For those interested

Locally create an SSH tunnel:
ssh -NL localhost:45312:/var/run/docker.sock user@server

Locally, vscode has the following in settings.json:
{
"docker.host":"tcp://localhost:45312"
}

Once the tunnel is launched, you should see the container in Remote Explorer, Containers
Attaching to it with vscode also works

[chrmarti]

What was your docker.host setting when it didn't work?

[msalvaris]

I tried multiple docker.host settings:
"docker.host":"ssh://user@server"
With the above I could see running containers and images on host but when "Attach Visual Studio Code" it would fail

I then set my docker daemon on remote host to accept connections on port 2375
"docker.host":"tcp://server:2375",
Again same as before

[chrmarti]

Could you retry with the latest VS Code Insiders (1.44 insiders) and the latest Remote-Containers extension version (0.107.0)? That only needs the Docker CLI to work (previously we also used an NPM package).

[msalvaris]

I just did and anecdotally things seem worse. I can see everything on localhost and connect easily but when I connect over ssh it is a lot more sluggish and it keeps coming up with the error message asking whether the daemon is running when I try to connect vs code. When I connect over the tunnel everything is fine. Let me know if there are any logs etc I can get to post here.

[chrmarti]

Do you have any of the DOCKER_* environment variables set? That might interfere with the docker.* settings that offer similar options.

[msalvaris]

I looked and have no DOCKER_* variables set in the local env.
I checked them using:
Get-ChildItem env:

[chrmarti]

Could you check if there is additional information in the dev console? (Help > Toggle Developer Tools)

[msalvaris]

Yep I get
error during connect: Get http://docker/v1.40/containers/a28b42e…/json: command [ssh -l mat ada -- docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=ssh_exchange_identification: read: Connection reset

[chrmarti]

What do you get for docker version on the remote host?

[msalvaris]

Yep, Docker version 19.03.6, build 369ce74a3c

[devrite]

I can confirm that. For me it looks like the attach to running container part ignores the docker.host setting in settings.json .

I'm using it to debug on a ssh target inside a container (X86-64 host, ARM64 target). When attaching to the running container it tries to rebuild the docker image (Dockerfile) for some reason and since the base image is ARM64 I get an exec format error on the first RUN of a native command.

When opening in a fresh container it works as far as i can tell.

[chrmarti]

@devrite Could you attach the log? (F1 > Remote-Containers: Show Log or click on the link in the progress notification in the lower right while it is connecting.)

[devrite]

@chrmarti here you go.

Docker host: "docker.host":"ssh://[email protected]". Remote user is also set to user in devcontainer.json.

The container is listed correctly from the remote yet the extension starts a new build of the image locally.

[9 ms] Start: Resolving remote
[9 ms] Setting up container for folder or workspace: <some-host-local-path>

[11 ms] Start: Run: git rev-parse --show-toplevel
[17 ms] Start: Check Docker is running
[17 ms] Start: Run: docker info
[98 ms] Start: Run: docker ps -q -a --filter label=vsch.local.folder=/<some-host-local-path> --filter label=vsch.quality=stable
[133 ms] Start: Run: docker build -f <some-host-local-path>/.devcontainer/Dockerfile -t vsc-folder-09a20399feba8293e2f63ab171a6e563 /<some-host-local-path>/.devcontainer
Sending build context to Docker daemon   12.8kB
Step 1/10 : FROM nvcr.io/nvidia/deepstream-l4t:4.0.2-19.12-base
 ---> 0cd4a7589de6
Step 2/10 : ENV DEBIAN_FRONTEND=noninteractive
 ---> Using cache
 ---> 35a7c20631e2
Step 3/10 : COPY jetson-ota-public.key nvidia-l4t-apt-source.list /
 ---> Using cache
 ---> d05576582099
Step 4/10 : RUN mv /nvidia-l4t-apt-source.list /etc/apt/sources.list.d &&  cat /jetson-ota-public.key | apt-key add - && rm /jetson-ota-public.key
 ---> Running in a8a0de0f6a95
standard_init_linux.go:211: exec user process caused "exec format error"
The command '/bin/sh -c mv /nvidia-l4t-apt-source.list /etc/apt/sources.list.d &&  cat /jetson-ota-public.key | apt-key add - && rm /jetson-ota-public.key' returned a non-zero code: 1
[1200 ms] Command failed: docker build -f <some-host-local-path>.devcontainer/Dockerfile -t vsc-folder-09a20399feba8293e2f63ab171a6e563 <some-host-local-path>.devcontainer

Thanks

[devrite]

I can also note that if I install extension into the container I am able to reload the window afterwards.

[devrite]

So setting the ENV var just worked for reconnecting with the container. The reconnect/(re)attach step does not consider the setting in settings.json while open in a new container does. So until it is resolved I can use that as a workaround.

[MichaelVasseurBE]

Hi @chrmarti ,

I've got a similar issue, but I'm using the DOCKER_HOST (set to tcp://localhost:3008 which is a local port forwarded via SSH) variable to specifiy the docker host.

When starting VS Code with this variable set in the environment, I can see the containers in the Remote Explorer and in the Docker extension.

However, when I try to attach to a container, VS Code tell me that it can't find the container, giving its name.

I checked the envrionment of newly created VS Code with Process Explorer and see that the DOCKER_HOST variable seems lost.

Do you have any clue ?

Can I start VS Code with a kind of command line switch to make it attach directly to a container ?

Regards,

[serefarikan]

Here is my observations on this issue:

Code 1.48.2 running on Ubuntu 18.04
Remote containers: 0.134.1
Remote Development: 0.20.0

Setting the docker.host property in settings.json (F1 -> preferences: open settings) works only partially. Containers view in remote explorer won't display remote containers but will display local ones.

Docker's own view will display remote containers though, but attempting to attach vs code to a container won't work. Attaching a terminal (from docker view, right click on container) will work though.

My scenario is attaching vs code to a running container, so I don't know what would happen if I attempted to use devcontainers.

Note that settings.json is the only way I'd expect to use this feature, I cannot open a folder, it is on the remote machine, either in the docker container or mounted to docker container from a local dir on the remote machine.

Setting the DOCKER_HOST environment variable and running code from the terminal works. Remote explorer now has my remote containers listed and I can attach to them.
The same remote docker host is used from another instance of code on windows 10 without any problems so whatever the issue is, at least for me, it is specific to linux version of code.

Hope this helps, and thanks to everyone who helped with the remote development feature, it really is great.

[awesomebytes]

I'm affected by this too. We develop in remote ARM64 machines which run Docker containers.
Setting up settings.json every time we want to open a Visual Studio Code for the remote docker container is not very friendly, and very not intuitive, given that Visual Studio Code does show the running containers in the remote machine when opening a Remote SSH Visual Studio Code window. We can also open a terminal, the only thing that doesn't work is actually attaching Visual Studio Code to it.

[devrite]

@awesomebytes Hey, for now I worked around it by exporting the DOCKER_HOST before launching the vscode window (which I want to attach or create a remote container). As long as this window stays open it will continue to use the DOCKER_HOST even if you reopen/reload the window locally and reattach later.

export DOCKER_HOST="ssh://[email protected]"
#should display  your remote hostname
docker info | grep -i Name
code

You could always set it using your bashrc or write a custom launch/env script for code. I hope that helps until this is resolved. On the remote host you should not require to do anything (except by either checking out a git repo, or provide a mountpoint/volume to your code). Note that your remote code (when edited) is not automatically synced to your local code (don't forget to commit and push/sync your code).

Another way to do it may be to rsync your code to the remove and mount it (via volume or volume bind mount into the container). This is useful if you do not have a git repo or your remote machine does not have access to it (Note that you can always clone your local bare repo via ssh if you want and ssh is setup properly instead of cloning from the upstream repo).

[MichaelVasseurBE]

@devrite This is working but if you try to attach VS Code to a container from this VS Code instance, the attachment will fail because the newly created VS COde instance does NOT have the DOCKER_HOST in its own environment.

[devrite]

@awesomebytes I do not know if I understand your requirement. Once you opened/attached you are on the remote (within its environment). If you want to reuse the code instance you must first detach (reopen locally) and then attach to another container on the same host. I think what you want is a new feature (cascaded/recursive attach and/or multiple configured hosts and/or environment sharing). I would suggest a new issue? What you can do now is start a second instance locally with a different docker host or the same and attach to another container. Each new window is its own instance (if not started from within that window at least). This was my impression correct me if I am wrong.

[awesomebytes]

@devrite My requirement would be that once I open a remote VS Code instance (e.g. it used SSH to develop with 'native' feeling in a remote machine) I would get in the Containers section (in the dropdown where I can choose SSH Targets or Containers) the containers running in the machine where that VS Code instance is running/connected to.

From there I would right-click and attach VS Code to one of those running instances. I think calling it 'cascaded attach' makes a lot of sense.

I've made a video to illustrate better my expectations:

First I open VS Code on a local project (which I may do some changes and push them), then I want to try this project in a remote machine so I use this awesome plugin to open an instance of VS Code in that machine. I must run this code inside of a Docker container, so when I use the drop-down to choose/refresh the container list, I expect the containers of that machine to appear (as I'm getting this native feeling of being in that machine).

Additionally, and I understand this is unrelated to this repository, so I apologize for the noise but I think it is interesting for anyone with the same workflow wish, if using the Docker extension I get almost the behavior I expect. I made another video to illustrate it:

I first use the plugin from this repo to get a VS Code instance in a remote machine, then I use the Docker plugin where I see the remote containers that are available. Moreover, I can attach a shell to that remote docker container. But when I try to attach a VS Code instance it silently fails. I'm not sure if this is an issue with this plugin, with the docker plugin, or with the interaction of both. I'd love to have this fixed as it feels to me that it's super close to work in a very user-friendly way.

As a final note, my use-case is the following: I develop robotics software on amd64 computers first, then test/deploy on arm64 (TX2/Raspberry Pi). It may not be the most typical use-case, but VS Code has made this way more friendly than it was years ago!

Let me know if I wasn't too clear or you want me to test anything. Very open to help!

[mlabour]

Same issue as @awesomebytes . It used to fail silently. I upgraded Docker & VSCode and now it fails w/ a popup "There are no running containers to attach to". Image enclosed.

[chrmarti]

Note that connecting to a container while being connecting via Remote-SSH is tracked as #2994.

[awesomebytes]

I've found a workaround that works for my use-case. I open VSCode from the command line. I've documented the process here: https://gist.github.com/awesomebytes/8494fb216c42bc4a2fcaef6b4937a07e

I copy the relevant bit in this issue too in case others may find it useful:

# With 192.168.1.100 being the IP of the machine with the docker container running
export DOCKER_HOST="ssh://[email protected]"
CONTAINER_NAME=zen_gates
FOLDER=/home/dockeruser/myfolder
HEX_CONFIG=$(printf {\"containerName\":\"/$CONTAINER_NAME\"} | od -A n -t x1 | tr -d '[\n\t ]')
code --folder-uri "vscode-remote://attached-container+$HEX_CONFIG$FOLDER"

[yangyhua]

I have same problem.
local machine: windows 10, docker desktop v19.03.13
remote machine: centos 7.7, docker 1.13.1

I use vscode 1.50.0
extension: remote-ssh 0.55.0, docker 1.6.0, remote-containers 0.145.0

I open vscode in the remote machine, and want to attach vscode for the container in remote machine.
It notes me "there are no running containers to attach to", but if I click attach shell, it can work in terminal.

[chrmarti]

Note that connecting to a container while being connecting via Remote-SSH is tracked as #2994.

[chrmarti]

Continuing in #4072.