Can't attach to a container running on a remote machine

[alexkirsz]

• VSCode Version: 1.52.1
• Local OS Version: macOS 10.15.7
• Remote OS Version: Ubuntu 20.04.1
• Remote Extension/Connection Type: SSH, then Docker

Steps to Reproduce:

1. Create a docker-compose.yml with the following contents:

version: '3.9'

services:
  hello:
    image: debian:buster
    entrypoint: "/bin/sh -c 'while :; do sleep 1000; done'"

2. Run docker-compose up -d

3. Attach to running container -> Select the hello container.

4. The following error occurs:

[2 ms] Remote-Containers 0.155.1 in VS Code 1.52.1 (ea3859d4ba2f3e577a159bc91e3074c5d85c0523).
[17 ms] Start: Resolving Remote
[19 ms] Setting up container /project_hello_1

[19 ms] Start: Run: docker inspect --type container /project_hello_1
[2201 ms] Start: Check Docker is running
[2201 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[3741 ms] Server API version: 1.41
[3742 ms] Start: Inspecting container
[3742 ms] Start: Run: docker inspect --type container b235cf4125de22c61dd57caab1007066fd4c4656e25b50ddb220f15585383281
[5888 ms] Start: Run: docker exec -i -u root -e VSCODE_REMOTE_CONTAINERS_SESSION=15558f45-8d97-45fe-b99f-64d37f0778fc1612416965003 b235cf4125de22c61dd57caab1007066fd4c4656e25b50ddb220f15585383281 /bin/sh
[5892 ms] Start: Run in container: uname -m
[8186 ms] Start: Run in container: cat /etc/passwd
[8186 ms] Stdin closed!
[8188 ms] Shell server terminated (code: 1, signal: null)

command [ssh -- devserver docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=kex_exchange_identification: read: Connection reset by peer

Sometimes, clicking "Retry" will go further but then throw another error down the line (case 1):

[2 ms] Remote-Containers 0.155.1 in VS Code 1.52.1 (ea3859d4ba2f3e577a159bc91e3074c5d85c0523).
[14 ms] Start: Resolving Remote
[15 ms] Setting up container /project_hello_1

[15 ms] Start: Run: docker inspect --type container /project_hello_1
[1770 ms] Start: Check Docker is running
[1770 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[3268 ms] Server API version: 1.41
[3269 ms] Start: Inspecting container
[3269 ms] Start: Run: docker inspect --type container b235cf4125de22c61dd57caab1007066fd4c4656e25b50ddb220f15585383281
[5554 ms] Start: Run: docker exec -i -u root -e VSCODE_REMOTE_CONTAINERS_SESSION=a6ba1e6c-9ed1-4213-98e3-a86b4eef65f81612417264464 b235cf4125de22c61dd57caab1007066fd4c4656e25b50ddb220f15585383281 /bin/sh
[5558 ms] Start: Run in container: uname -m
[8569 ms] x86_64
[8570 ms] 
[8570 ms] Start: Run in container: (cat /etc/os-release || cat /usr/lib/os-release) 2>/dev/null
[8592 ms] PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
[8592 ms] 
[8592 ms] Start: Run in container: cat /etc/passwd
[8615 ms] Start: Run in container: test -d /root/.vscode-server
[8637 ms] 
[8638 ms] 
[8638 ms] Exit code 1
[8638 ms] Start: Run in container: test -d /root/.vscode-remote
[8660 ms] 
[8660 ms] 
[8660 ms] Exit code 1
[8660 ms] Start: Run in container: set -o noclobber ; mkdir -p '/root/.vscode-server/data/Machine' && { > '/root/.vscode-server/data/Machine/.writeMachineSettingsMarker' ; } 2> /dev/null
[8682 ms] 
[8683 ms] 
[8683 ms] Start: Run in container: cat /root/.vscode-server/data/Machine/settings.json
[8704 ms] 
[8705 ms] cat: /root/.vscode-server/data/Machine/settings.json: No such file or directory
[8705 ms] Exit code 1
[8705 ms] Start: Run in container: test -d /root/.vscode-server/bin/ea3859d4ba2f3e577a159bc91e3074c5d85c0523
[8725 ms] 
[8726 ms] 
[8726 ms] Exit code 1
[8726 ms] Installing VS Code Server for commit ea3859d4ba2f3e577a159bc91e3074c5d85c0523
[8726 ms] Start: Run in container: mkdir -p /root/.vscode-server/bin/ea3859d4ba2f3e577a159bc91e3074c5d85c0523_1612417273735
[8747 ms] 
[8747 ms] 
[8747 ms] Start: Installing VS Code Server
[8766 ms] Start: Run in container: (dd iflag=fullblock bs=8192 count=3723 2>/dev/null; dd iflag=fullblock bs=895 count=1 2>/dev/null) | tar --no-same-owner -xz --strip-components 1 -C /root/.vscode-server/bin/ea3859d4ba2f3e577a159bc91e3074c5d85c0523_1612417273735
[24642 ms] 
[24642 ms] 
[24642 ms] Start: Run in container: mv -n /root/.vscode-server/bin/ea3859d4ba2f3e577a159bc91e3074c5d85c0523_1612417273735 /root/.vscode-server/bin/ea3859d4ba2f3e577a159bc91e3074c5d85c0523
[24665 ms] 
[24665 ms] 
[24665 ms] Start: Launching Remote-Containers helper.
[24665 ms] Start: Run: gpgconf --list-dir agent-extra-socket
[24671 ms] Start: Run in container: # Test for /root/.ssh/known_hosts and ssh
[24691 ms] ssh not found
[24691 ms] 
[24691 ms] Exit code 1
[24691 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-b9b9520be8fdd1d9aa9e254cf0231f9781fdf0dc.js' >/tmp/vscode-remote-containers-b9b9520be8fdd1d9aa9e254cf0231f9781fdf0dc.js
[24742 ms] 
[24742 ms] 
[24742 ms] Start: Run in container: command -v git >/dev/null 2>&1 && git config --system credential.helper '!f() { /root/.vscode-server/bin/ea3859d4ba2f3e577a159bc91e3074c5d85c0523/node /tmp/vscode-remote-containers-b9b9520be8fdd1d9aa9e254cf0231f9781fdf0dc.js $*; }; f' || true
[24766 ms] 
[24766 ms] 
[24766 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-server-b9b9520be8fdd1d9aa9e254cf0231f9781fdf0dc.js' >/tmp/vscode-remote-containers-server-b9b9520be8fdd1d9aa9e254cf0231f9781fdf0dc.js
[24817 ms] 
[24817 ms] 
[24817 ms] Start: Run in container: # Test for /root/.gitconfig and git
[24817 ms] Start: Run: docker exec -i -u root -e REMOTE_CONTAINERS_SOCKETS=["/tmp/vscode-ssh-auth-b9b9520be8fdd1d9aa9e254cf0231f9781fdf0dc.sock"] -e REMOTE_CONTAINERS_IPC=/tmp/vscode-remote-containers-ipc-b9b9520be8fdd1d9aa9e254cf0231f9781fdf0dc.sock b235cf4125de22c61dd57caab1007066fd4c4656e25b50ddb220f15585383281 /root/.vscode-server/bin/ea3859d4ba2f3e577a159bc91e3074c5d85c0523/node /tmp/vscode-remote-containers-server-b9b9520be8fdd1d9aa9e254cf0231f9781fdf0dc.js
[24842 ms] git not found
[24842 ms] 
[24842 ms] Exit code 1
[24842 ms] Start: Run in container: command -v git >/dev/null 2>&1 && git config --global credential.helper '!f() { /root/.vscode-server/bin/ea3859d4ba2f3e577a159bc91e3074c5d85c0523/node /tmp/vscode-remote-containers-b9b9520be8fdd1d9aa9e254cf0231f9781fdf0dc.js $*; }; f' || true
[24866 ms] 
[24866 ms] 
[24866 ms] Start: Preparing Extensions
[24867 ms] Start: Run in container: set -o noclobber ; mkdir -p '/root/.vscode-server/data/Machine' && { > '/root/.vscode-server/data/Machine/.installExtensionsMarker' ; } 2> /dev/null
[24891 ms] 
[24891 ms] 
[24893 ms] Start: Run in container: test -d /root/.vscode-server/extensionsCache && ls /root/.vscode-server/extensionsCache || true
[24917 ms] 
[24917 ms] 
[24918 ms] Start: Run in container: for pid in `cd /proc && ls -d [0-9]*`; do { echo $pid ; readlink /proc/$pid/cwd ; readlink /proc/$pid/ns/mnt ; cat /proc/$pid/stat | tr "
[24967 ms] Start: Starting VS Code Server
[24967 ms] Start: Run in container: /root/.vscode-server/bin/ea3859d4ba2f3e577a159bc91e3074c5d85c0523/server.sh --force-disable-user-env --port 0 --extensions-download-dir /root/.vscode-server/extensionsCache --start-server
[27920 ms] 

*
* Visual Studio Code Server
*
* Reminder: You may only use this software with Visual Studio family products,
* as described in the license https://aka.ms/vscode-remote/license
*

Extension host agent listening on 44135

[27921 ms] Start: Run in container: echo 44135 >/root/.vscode-server/data/Machine/.devport
[27943 ms] 
[27943 ms] 
[27946 ms] Forwarding local port 44135 to container port 44135
[27947 ms] Start: Run: docker exec -i -u root -w /root/.vscode-server/extensions b235cf4125de22c61dd57caab1007066fd4c4656e25b50ddb220f15585383281 /bin/sh -c # Watch installed extensions
[27947 ms] Start: Run: docker exec -i -u root -w /root/.vscode-server/data/Machine b235cf4125de22c61dd57caab1007066fd4c4656e25b50ddb220f15585383281 /bin/sh -c # Watch machine settings
[27966 ms] Start: Run: docker exec -i -u root -e VSCODE_REMOTE_CONTAINERS_SESSION=a6ba1e6c-9ed1-4213-98e3-a86b4eef65f81612417264464 b235cf4125de22c61dd57caab1007066fd4c4656e25b50ddb220f15585383281 /root/.vscode-server/bin/ea3859d4ba2f3e577a159bc91e3074c5d85c0523/node -e 
[27970 ms] Start: Run: docker exec -i -u root -e VSCODE_REMOTE_CONTAINERS_SESSION=a6ba1e6c-9ed1-4213-98e3-a86b4eef65f81612417264464 b235cf4125de22c61dd57caab1007066fd4c4656e25b50ddb220f15585383281 /root/.vscode-server/bin/ea3859d4ba2f3e577a159bc91e3074c5d85c0523/node -e 
[27994 ms] [05:41:32] Extension host agent started.
[29220 ms] Start: Run in container: cat /root/.vscode-server/data/Machine/settings.json
[29233 ms] Start: Run: docker exec -i -u root -e VSCODE_AGENT_FOLDER=/root/.vscode-server -w /root/.vscode-server/bin/ea3859d4ba2f3e577a159bc91e3074c5d85c0523 b235cf4125de22c61dd57caab1007066fd4c4656e25b50ddb220f15585383281 /root/.vscode-server/bin/ea3859d4ba2f3e577a159bc91e3074c5d85c0523/server.sh --list-extensions
[29244 ms] 
[29245 ms] cat: /root/.vscode-server/data/Machine/settings.json: No such file or directory
[29245 ms] Exit code 1
[29921 ms] 
[29921 ms] error during connect: Get http://docker/v1.24/containers/b235cf4125de22c61dd57caab1007066fd4c4656e25b50ddb220f15585383281/json: command [ssh -- devserver docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=kex_exchange_identification: read: Connection reset by peer

Other times, clicking "Retry" will simply show a popup with the following message (case 2):

Cannot attach to the container with name/id /project_hello_1, it no longer exists.

And the option to "Choose container". Clicking on "Choose container" will then show the following error at the bottom right of the screen:

Failed to connect to the remote extension host server (Error: Command failed: docker ps -q)

With the following in the terminal:

[2 ms] Remote-Containers 0.155.1 in VS Code 1.52.1 (ea3859d4ba2f3e577a159bc91e3074c5d85c0523).
[14 ms] Start: Resolving Remote
[16 ms] Setting up container /project_hello_1

[16 ms] Start: Run: docker inspect --type container /project_hello_1
[101614 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[101731 ms] 
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
[101732 ms] Start: Run: docker ps -q

I've verified that the container is still running in all of these cases. Whether one case or the other occurs seems up to chance, but I've noticed that case 2 occurs much more frequently.

---

I've also tried this workflow with Insiders but couldn't even get past "Attach to Running Containers", where it displays the following error in the bottom right of my screen:

Command 'Remote-Containers: Attach to Running Container...' resulted in an error (Running the contributed command: 'remote-containers.attachToRunningContainer' failed.)

---

In both cases, the docker extension properly registers the docker daemon and the running container.

[chrmarti]

I wonder if something hangs up the connection (firewall or something) due to the many ssh connections being established. Could you try reusing a single connection by adding the following to your ~/.ssh/config:

Host *
    ControlMaster auto
    ControlPath ~/.ssh/sockets/%r@%h-%p
    ControlPersist 600

(You can also use a specific host.)

This should also speed things up.

[alexkirsz]

@chrmarti You weren't kidding. Everything works smoothly now. I'm behind a corporate VPN + Firewall which I go through to access the remote machine.

Thanks a lot!

Unrelated, but is there any progress for support for building and opening a container from a devcontainer.json configuration file on remote machines? (without having to boot up a docker agent on my local machine)

[chrmarti]

More general SSH support is tracked as #2994. The ability to attach to a container in a Remote-SSH session is a recent step towards full support.