Reconnection attempts after "internal libpod error" when starting a project with Podman

[jeremyn]

Often after starting VS Code with Remote-Containers in a project using a Podman container, there are many reconnection attempts due to an internal libpod error, also included in the log below:

Error: timed out waiting for file /home/$HOST_USER/.local/share/containers/storage/overlay-containers/390f007317f11a0d1a4a9c6f92177711fa3ccde63c50e3a064a0d41fa30048df/userdata/6f0c6d8b91e8de7dd13bffb2544e957c60acc3b27147a6dc8b25a020735cf6c0/exit/390f007317f11a0d1a4a9c6f92177711fa3ccde63c50e3a064a0d41fa30048df: internal libpod error

Sometimes it will stop reconnecting and the project will successfully load in VS Code after only a few reconnection attempts, as in the log below, and sometimes it tries indefinitely. In that case I can shut down VS Code and start it again, and it may or may not work, and if not I can shut down VS Code again and start it again. Eventually it works. I don't know why it works sometimes and not other times. You can see from the log below that it takes about 7 seconds to cycle between attempts.

Other people have reported the same problem in #6685 and I'm creating a new issue as requested by @chrmarti in #6685 (comment).

• VSCode Version: 1.71.0
• Local OS Version: Ubuntu 22.04 (on VMware Workstation Pro on Windows 10), using Podman v3.4.4 from Ubuntu repositories
• Remote OS Version: Ubuntu 22.04
• Remote Extension/Connection Type: Remote-Containers v0.251.0
• Logs: From the end of a container log after startup, in this case there is only one reconnection attempt and then the project loads successfully. $CONTAINER_USER and $HOST_USER have been added by me.

[5128 ms] *
* Visual Studio Code Server
*
* By using the software, you agree to
* the Visual Studio Code Server License Terms (https://aka.ms/vscode-server-license) and
* the Microsoft Privacy Statement (https://privacy.microsoft.com/en-US/privacystatement).
*
[5134 ms] Server bound to 127.0.0.1:38777 (IPv4)
Extension host agent listening on 38777

[5134 ms] Start: Run in container: echo 38777 >'/home/$CONTAINER_USER/.vscode-server/data/Machine/.devport-784b0177c56c607789f9638da7b6bf3230d47a8c'
[5135 ms] 
[5135 ms] 
[5135 ms] Port forwarding for container port 38777 starts listening on local port.
[5136 ms] Port forwarding local port 38777 to container port 38777
[5193 ms] userEnvProbe PATHs:
Probe:     '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
Container: '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
[5229 ms] Port forwarding connection from 42220 > 38777 > 38777 in the container.
[5230 ms] Start: Run in container: /home/$CONTAINER_USER/.vscode-server/bin/784b0177c56c607789f9638da7b6bf3230d47a8c/node -e 
[5530 ms] [13:13:29] Extension host agent started.
[5666 ms] Port forwarding 42220 > 38777 > 38777 stderr: Connection established
[5742 ms] Port forwarding connection from 42236 > 38777 > 38777 in the container.
[5742 ms] Start: Run in container: /home/$CONTAINER_USER/.vscode-server/bin/784b0177c56c607789f9638da7b6bf3230d47a8c/node -e 
[5849 ms] [13:13:29] ComputeTargetPlatform: linux-x64
[5916 ms] [13:13:29] [127.0.0.1][60ff6de4][ManagementConnection] New connection established.
[6057 ms] Port forwarding 42236 > 38777 > 38777 stderr: Connection established
[6424 ms] [13:13:30] [127.0.0.1][67f561c1][ExtensionHostConnection] New connection established.
[6430 ms] [13:13:30] [127.0.0.1][67f561c1][ExtensionHostConnection] <179> Launched Extension Host Process.
[12613 ms] Port forwarding 42220 > 38777 > 38777 stderr: Error: timed out waiting for file /home/$HOST_USER/.local/share/containers/storage/overlay-containers/390f007317f11a0d1a4a9c6f92177711fa3ccde63c50e3a064a0d41fa30048df/userdata/6f0c6d8b91e8de7dd13bffb2544e957c60acc3b27147a6dc8b25a020735cf6c0/exit/390f007317f11a0d1a4a9c6f92177711fa3ccde63c50e3a064a0d41fa30048df: internal libpod error
[12616 ms] Port forwarding 42220 > 38777 > 38777 terminated with code 255 and signal null.
[12617 ms] Port forwarding 42220 > 38777 > 38777: Local close
[12626 ms] Start: Reconnection attempt 1
[12626 ms] Start: Run: podman inspect --type container 390f007317f11a0d1a4a9c6f92177711fa3ccde63c50e3a064a0d41fa30048df
[12856 ms] Port forwarding connection from 47004 > 38777 > 38777 in the container.
[12857 ms] Start: Run in container: /home/$CONTAINER_USER/.vscode-server/bin/784b0177c56c607789f9638da7b6bf3230d47a8c/node -e 
[12971 ms] Port forwarding 47004 > 38777 > 38777 stderr: Connection established
[12984 ms] [13:13:36] [127.0.0.1][60ff6de4][ManagementConnection] The client has reconnected.
[14534 ms] Start: Run in container: cat /proc/233/environ

Steps to Reproduce:

1. Start VS Code with Remote-Containers on a project in a Podman container.
2. Sometimes it cycles through a few or many reconnection attempts, maybe eventually working, maybe not, as described above.

[chrmarti]

Which version of Podman are you using?

[jeremyn]

@chrmarti From my first comment: "Podman v3.4.4 from Ubuntu repositories", which is to say the current version from apt install podman.

[chrmarti]

Could be containers/podman#13227. Could you try installing the most recent Podman release? (The fix for that issue appears to be in Podman 4.1.0, the latest version is 4.2.0.)

[jeremyn]

@chrmarti That's a good find. Unfortunately it doesn't look like Podman 4.x is available on any version of Ubuntu through any official channel. The instructions at https://podman.io/getting-started/installation for Ubuntu, other than using the official repos, are broken because they point to a not-for-production-use Kubic "testing" repository that is only available through 21.10 and contains Podman 3.4.2. Issue containers/podman#14302 talks about the difficulties in getting Podman 4.x into Ubuntu.

In that issue someone who appears to be a Red Hat employee has created a Kubic "unstable" repository with Podman 4.x for Ubuntu, see containers/podman#14302 (comment). I did install that and restarted VS Code several times and the reconnection problem didn't happen, so it looks like the current version of Podman from that repository (4.2.1) does fix the problem. However I don't think we can say that this build of a not-for-production-use repository that is still considered so unstable that it's not even documented on the Podman site is a valid solution to this VS Code issue.

I see two approaches:

1. There's a suggestion in the issue you linked, and in at least one of the issues linked to that issue, that the internal libpod error problem has to do with Podman logging. Perhaps VS Code could pass some logging option to Podman that could work around the problem.

2. Make it clear that VS Code support on Podman is experimental/glitchy/only on certain platforms etc.

What do you think?

[RReverser]

Thanks for the suggestions here, I've upgraded to podman 4.3 via provided Kubic links and can confirm that finally fixes the regular reconnection issues.

I'm okay with the not-for-production-use caveat as it's just for my development machine anyway.

[jeremyn]

Just an update: Ubuntu 22.10 "Kinetic" came out last month, and both it and the previous version 22.04 "Jammy" have the same Podman version, v3.4.4. This shouldn't be a surprise given issue containers/podman#14302 that I linked in my previous comment, but it strongly suggests that the included version will be stuck at v3.4.4 for a while.

[jeremyn]

I asked on the Podman issue I linked, see containers/podman#14302 (comment) and following, and it seems there is no official plan by Red Hat or Canonical to keep Podman updated on Ubuntu for any version. Instead we have the undocumented and unofficial Kubic repository and the usual best efforts by part-time unpaid Debian volunteers.

So for anyone who doesn't want to use the Kubic repository, I expect this problem will persist at least until the next Ubuntu LTS (April 2024), if not later, unless the VS Code team takes some action.

[dawidd6]

You can also install Homebrew on any Linux distro and get the latest and greatest rootless podman from there. If you don't mind another package manager ofc.

I'm using it since a couple of months I believe, together with flatpak'ed VSCode and this duo works just fine.

[jeremyn]

That's a great idea, I hadn't considered that. The Podman documentation even recommends using Homebrew for Mac in the installation instructions, and the formula page for Podman says Linux is supported.

I'll go ask over in the other issue whether they might update the documentation to recommend Homebrew for Linux.

[jeremyn]

@chrmarti I wouldn't say this is definitely an upstream bug, for all we know the VS Code extension is doing something wrong that the later version of Podman handles more gracefully than the earlier version.

[aloisklink]

VS Code extension is doing something wrong

I believe the issue (at least in my case) is that VS Code is outputting too much on stdout and stderr and older versions of Podman are timing out when trying to log the stdout/stderr output in journald. This bug seems to have been fixed in Podman 4.2, see containers/podman#13779 (comment).

In my case, since I want to stay on podman version 3.4.4 (that's the version in the Ubuntu 22.04 repos), using --log-driver=none seems to be working as a workaround. You can add it to .devcontainer/devcontainer.json as {"runArgs": ["--log-driver=none"]}. The only issue is that then podman logs won't work, but 🤷 Using --log-driver=none also has the benefit of keeping journalctl tidier and more performant, so it's worth doing anyway if you never look at podman logs.