Port forwarding not working (ERR_EMPTY_RESPONSE in browser)

[DanTup]

I have a Dockerfile in .devcontainer like this:

FROM ruby:2

RUN gem install github-pages

ENV SHELL /bin/bash

If I load my project in the container and then run jekyll serve in the shell, I see:

    Server address: http://127.0.0.1:4000
  Server running... press ctrl-c to stop.

If I then run "Forward port from container" it correctly lists 4000 and claism to forward it, however when I open in my browser I get:

This page isn’t working
localhost didn’t send any data.
ERR_EMPTY_RESPONSE

However if I open another terminal window and run curl localhost:4000 then I see the HTML output as expected.

I don't know if it's related, but there's this in the developer tools console.

[DanTup]

Running in Jekyll in verbose mode shows when a client connects (for ex. when using curl in another terminal in the container), but doesn't show anything here, suggesting the connection isn't getting to Jekyll.

[DanTup]

Hmmm, I tried it on a different port (8123) and it seems to work. Is there something weird about 4000?

[chrmarti]

Did you have something listen on your local 4000? In that case the forwarding will pick a random port locally, there is a notification telling you which port it ended up with.

[DanTup]

No, there was nothing local and I used the "open in browser" button in the forwarding notification. What's weird is I was sure it worked the first time, then it broke - I think when I tried changing it to using appPort in JSON. But now even after rebuilding the container, rebooting, and not having it in appPort it still gets empty responses.

I'll try another reboot and test again later just to be sure.

[DanTup]

It's working now :/

If I see it again, I'll try and do some more digging. If there's anything you think I should try/check, let me know.

[DanTup]

Ok, there's definitely something not working right here. I'm seeing it again on a completely different image. I'm using rocker/verse which contains RStudio that listens on 8787, and it worked the first time I ran it, and now I just get ERR_EMPTY_RESPONSE.

Interestingly, if I stop the container/close VS Code, then I get a connection error (eg. nothing is bound to the port), but when I open VS Code, let the container start, then forward the port (which shows 8787), then I get ERR_EMPTY_RESPONSE. So it definitely seems like the port forwarding is doing something, but it's not working correctly.

I'm on Windows 10 Pro. Dockerfile just looks like:

FROM rocker/verse

ENV SHELL /bin/bash

devcontainer.json looks like this:

{
	"name": "RStudio",
	"dockerFile": "Dockerfile",
	"runArgs": [
		"-e",
		"PASSWORD=rrrstudio"
	]
}

I've tried both with appPort: 8787 and also without (just using the Forward command). I've also tried using "image": "rocker/verse" instead of having a Dockerfile.

Restarting Docker did not help.

[DanTup]

Hmm, interestingly using wget from inside the container I get "connection refused", so now I'm wondering if RStudio is even running properly (though if it's not, I'm not sure why 8787 shows up in the port forwarding list... is that just a list of what the container exposes, are is it things that are definitely open?).

Running docker run -e PASSWORD=rrrstudio --rm -p 8787:8787 rocker/verse outside of VS Code containers to work fine.

[DanTup]

I found the full command VS Code is running in the container log, and I added -p 8787:8787 to it and ran from the terminal:

docker run -p 8787:8787 -a STDOUT -a STDERR
    --mount type=bind,source=m:\Dev\RStudio,target=/workspaces/RStudio
    -v /host_mnt/c/Users/danny/.gitconfig:/root/.gitconfig -l vsch.quality=stable
    -l vsch.local.folder=m:\Dev\RStudio -l vsch.remote.devPort=0
    -e PASSWORD=rrrstudio rocker/verse

This all worked fine. So is VS Code's port forwarding different to doing -p 8787:8787? (I guess it's too late to map the port, so maybe it's tunneled? should the behaviour be the same?)

[chrmarti]

The port forwarding shows the list of currently open ports and uses tunneling through docker exec as you suggest.

Do you see the empty response only when inside the container you get the connection refused? (The port forwarding can't "forward" the refused connection because it has to accept the connection on the host machine first before it can do/attempt the actual forwarding.)

[jswny]

I am also getting the same problem trying to run the Webpack development server in the VSCode terminal after connecting to the node:12 dev container as follows: webpack-dev-server --config webpack.config.dev.js --mode development. You can see the repo I am running this on here: https://github.com/jswny/j1.io-v2.

Upon further investigation, it works just fine if I exec into the container and curl from there:

root@e5582278f81d:/# curl 127.0.0.1:8080
<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Joe Sweeney</title>
  </head>
  <body>
    <div id="main"></div>
  <script type="text/javascript" src="bundle.js"></script></body>
</html>root@e5582278f81d:/#

But when I curl from outside the container I get an empty response:

➜  j1.io-v2 git:(vscode-devcontainer) ✗ curl localhost:8080
curl: (52) Empty reply from server

EDIT: I fixed it! Turns out I had to enable the Webpack dev server to be accessed outside localhost (which is the default and would explain why it would only be accessible from inside the container). As per this issue: moby/moby#2522, I passed --host 0.0.0.0 to Webpack dev server. I would recommend that if you are also having this problem in the same way that I was to ensure that the server you are running inside the devcontainer is available outside the container.

@DanTup I think if you pass --host=0.0.0.0 to your serve command it might work for you too?

[chrmarti]

@jswny That is indeed a limitation that Docker's port forwarding has which you get when using "appPort" in the devcontainer.json. When using Forward Port from Container... while already connected to the container, you get the dynamic port forwarding we set up through docker exec and that does not have that same limitation.

[DanTup]

Do you see the empty response only when inside the container you get the connection refused?

Sorry for the delay. I can repro this issue now, but indeed I see connection refused inside (in the terminal) when I'm seeing empty_response from outside.

I can't figure out what's going wrong though - the fact that the port showed up in VS Code's "forward" list suggests that something is bound to the port, and if I run the docker container manually with a Docker port binding everything works. Any ideas?

Here's my devcontainer config:

{
	"name": "RStudio",
	"image": "rocker/verse",
	"runArgs": [
		"-e",
		"PASSWORD=rrrstudio"
	],
	"workspaceMount": "src=M:\\Dev\\RStudio\\code,dst=/home/rstudio,type=bind",
	"workspaceFolder": "/home/rstudio"
}

I'm no longer using my own Dockerfile, just the image set above. There's nothing obvious in the Dev Container terminal output to tell if the container is running correctly though. It's possible there's something wrong with the RStudio docker image that's causing it to fail, but it always runs without any problems from the command line with the port mapped directly:

docker run -d --rm -p 8787:8787 --mount src=M:\Dev\RStudio\code,dst=/home/rstudio,type=bind -e PASSWORD=rrrstudio rocker/verse

[chrmarti]

It might need to run something as part of its CMD or ENTRYPOINT which we override by default. Try "overrideCommand": false in the devcontainer.json to test that.

[DanTup]

Aha! That didn't seem to make any difference at first (I deleted all my docker containers and images except the rocker/verse image assuming that'd clear all state), but running "rebuild container" it seems like that did fix it.

I'm not sure why when I originally hit this it seemed to work "sometimes" though. It's long enough ago that I can't say for certain I didn't get confused with launching it from the command line.

If this is a known limitation, then there might be nothing to do here (though it'd obviously be nice if the user didn't need to know when this applied :-)).

[chrmarti]

The problem we were trying to solve with overriding CMD/ENTRYPOINT by default is that these would oftentimes start the server you want to work on and our assumption is that you would rather start that manually (in debug mode) once the project is set up (e.g., with npm install). Apparently that assumption doesn't hold here.

[DanTup]

Yeah, this use case was probably a bit weird. When I just started, I planned to use VS code to edit R files and run them (in the usual way you might use this container support), but it turned out that R had a nice web IDE so I switched to a container that included that and essentially ended up trying to use VS Code just as a convenient way to start the docker container and mount a local folder into it (but wasn't actually using VS Code for the editing). This was obviously pointless, so I then switched to just using a PoSh alias that ran Docker directly with the flags (that I stole from VS Code log) to mount the folder directly.

So the example above doesn't seem like a particularly valid use case. I don't know if there are "more-valid" workflows that would be hit by this though, so I'll leave it to you to decide whether it's an issue (or how much of an issue).

In any case, it may be worth expanding on the docs for overrideCommand (specifically calling out that it may prevent things from running that would otherwies run). I don't think it's totally clear (to a relative Docker noob like me, at least) from the existing text that might happen (the text suggests it exists only to stop a container quitting immediately).

[Chuxel]

@DanTup We just updated docs to include an explicit mention of overrideCommand in the in-depth section for setting up and image/Dockerfile and and clarified in the devcontainer.json reference.

[DanTup]

LGTM :-)

[Karrq]

I ended up here looking for why appPort didn't seem to be working correctly.
In this issue I found that, to apply that change I need to rebuild the container (since normally the EXPOSE isn't respected from the Dockerfile). This detail was missing from the docs and it would have certainly helped having it mentioned.

[Chuxel]

@Karrq Rebuild Container is mentioned in many different places. Where did you expect to see it but didn't find it?

[Karrq]

you are right. I was completely blind... It's written in multiple places!

Maybe not enough emphasis?
Something like

Make sure you rebuild your container if you edit your configuration

would probably have done it for me...

[Chuxel]

I updated this PR microsoft/vscode-docs#2901 to add it in a few more spots. We're also discussing prompting you about whether you want to rebuild if you reload after making modifications which likely is the right long term fix.

I'm going to close this issue since the original feedback is now live in the docs.