.bashrc updates are not available in tasks (unlike locally)

[AndersonSFerreira]

• VSCode Version: 1.46.0
• Local OS Version: Manjaro Lyria 20.0.3
• Remote OS Version: Debian GNU/Linux 9 (stretch)

Using vscode and .NET Core 2.1, I created a new ASP.NET Core Web App (Angular), when trying to utilize VS Code Remote Development (Containers). I copied the Dockerfile from the vscode-remote-try-dotnetcore example. Everything seems to compile just fine, but I get a runtime error:

fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
An unhandled exception has occurred while executing the request.
System.AggregateException: One or more errors occurred. (One or more errors occurred. (Failed to start 'npm'. To resolve this:.

[1] Ensure that 'npm' is installed and can be found in one of the PATH directories.
Current PATH enviroment variable is: /home/vscode/.vscode-server/bin/a5d1cc28bb5da32ec67e86cc50f84c67cc690321/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Make sure the executable is in one of those directories, or update your PATH.

[2] See the InnerException for further details of the cause.)) ---> System.AggregateException: One or more errors occurred. (Failed to start 'npm'. To resolve this:.

[1] Ensure that 'npm' is installed and can be found in one of the PATH directories.
Current PATH enviroment variable is: /home/vscode/.vscode-server/bin/a5d1cc28bb5da32ec67e86cc50f84c67cc690321/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Make sure the executable is in one of those directories, or update your PATH.

[2] See the InnerException for further details of the cause.) ---> System.InvalidOperationException: Failed to start 'npm'. To resolve this:.

Running $PATH on remote:

vscode@a266b7d784a7:/workspaces/[WebApp]$` echo $PATH

/home/vscode/.nvm/versions/node/v12.18.1/bin:/home/vscode/.vscode-server/bin/a5d1cc28bb5da32ec67e86cc50f84c67cc690321/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

As you can see, the $PATH does include /home/vscode/.nvm/versions/node/v12.18.1/bin, though I'm not sure why this doesn't match the path that the command npm spits out.

I'm not sure what to do here to get it to work.

[AndersonSFerreira]

Dockerfile

[chrmarti]

Does the vscode-remote-try-dotnetcore example work for you?

[AndersonSFerreira]

yes, the example works, but it only uses net core. Would you have an example that uses npm? or some project created by

dotnet new angular

[Chuxel]

@chrmarti It looks like .bashrc is not getting used by default in tasks. Shouldn't VS Code server have this fired?

[chrmarti]

@Chuxel We disabled that for #544. We could make that an option or maybe there is a way of running the task inside a shell. @AndersonSFerreira What does your tasks.json look like?

[Chuxel]

@chrmarti To be clear, things are fine in the terminal, it's just tasks that fail. The issue you reference is talking about the terminal. This is actually going to be a problem for using nvm in tasks in general since it has to source the script.

These are pre-created scripts from .NET Core and its triggered by the CLI itself, so I'm not sure we can modify it.

Here's the generated tasks.json:

{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "build",
            "command": "dotnet",
            "type": "process",
            "args": [
                "build",
                "${workspaceFolder}/dotnetcore-3.0.csproj",
                "/property:GenerateFullPaths=true",
                "/consoleloggerparameters:NoSummary"
            ],
            "problemMatcher": "$msCompile"
        },
        {
            "label": "publish",
            "command": "dotnet",
            "type": "process",
            "args": [
                "publish",
                "${workspaceFolder}/dotnetcore-3.0.csproj",
                "/property:GenerateFullPaths=true",
                "/consoleloggerparameters:NoSummary"
            ],
            "problemMatcher": "$msCompile"
        },
        {
            "label": "watch",
            "command": "dotnet",
            "type": "process",
            "args": [
                "watch",
                "run",
                "${workspaceFolder}/dotnetcore-3.0.csproj",
                "/property:GenerateFullPaths=true",
                "/consoleloggerparameters:NoSummary"
            ],
            "problemMatcher": "$msCompile"
        }
    ]
}

EDIT: The dotnet command internally calls npm and node. Repro is simple, just grab a .NET Core definition and run dotnet new angular in a terminal and hit F5.

[Chuxel]

@AndersonSFerreira Here's a workaround:

1. Add this to devcontainer.json (or modify the settings inside the container):

   "settings": {
   	"terminal.integrated.shell.linux": "/bin/bash",
   	"terminal.integrated.shellArgs.linux": [ "-i" ]
   },
   

2. Update task.json to change "type": "process", to "type": "shell".

@chrmarti I don't understand why the VS Code process wouldn't be passing along environment variables set in .bashrc. Doesn't the host process for VS Code Server get these vars set?

[chrmarti]

@Chuxel We start the server without using a shell to avoid relying on specific shell behavior. This is similar to what tasks does for "type": "process".

If NVM had a symlink to the current Node install, we could use ENV PATH=... in the Dockerfile to point to that, but it looks like NVM updates PATH to point to the different Node install folders.

The terminal now has a setting to turn off inheriting the environment variables. Maybe that is a better fix for #544 than to not read the shell environment. Or: We make reading the shell environment an option, but ideally we don't add more options if we don't have to.

[Chuxel]

but it looks like NVM updates PATH to point to the different Node install folders.

Correct - there's no way to use nvm without sourcing the init script.

We start the server without using a shell to avoid relying on specific shell behavior. This is similar to what tasks does for "type": "process".

The terminal now has a setting to turn off inheriting the environment variables. Maybe that is a better fix for #544 than to not read the shell environment. Or: We make reading the shell environment an option, but ideally we don't add more options if we don't have to.

Here's the thing, even with type process, nvm works locally, users will expect it to work in this case as well. The fact it also works in a terminal in the container will really throw people off. I'm actually unclear how nvm works with tasks on my mac if the env vars are wiped out for this type. What am I missing?

I'm also not 100% sure we should be removing the env vars as much as merging as was proposed in #544. (I'm potentially missing some offline convos on where this landed.)

[chrmarti]

It works locally with tasks on Mac because VS Code reads the env with bash -lic '...': https://github.com/microsoft/vscode/blob/687898822424805156ea02579b63160b2f73d239/src/vs/code/node/shellEnv.ts#L33

The same code can be run by the VS Code Server, but we disabled it for #544. The problem in #544 is specific to containers (or at least these minimal Linux installs) because the PATH is overwritten instead of augmented by the shell profile (that may seem unexpected, I think it just doesn't matter usually with containers). Merging two PATH values is non-trivial because the order matters.

Maybe a better fix for #544 would be to change the shell profile to not overwrite the existing PATH but add to it.

[Chuxel]

The same code can be run by the VS Code Server, but we disabled it for #544. The problem in #544 is specific to containers (or at least these minimal Linux installs) because the PATH is overwritten instead of augmented by the shell profile (that may seem unexpected, I think it just doesn't matter usually with containers). Merging two PATH values is non-trivial because the order matters.

Maybe a better fix for #544 would be to change the shell profile to not overwrite the existing PATH but add to it.

Got it. Yeah, it seems like we would want things already in the path to take precedence over what we append during startup. That's why I'd assumed that was a pretty trivial change (it's just concatenation at that point). The only other alternative I can think of is to preserve the original PATH in an alternate one (ORIG_PATH) that can be added to processes as you'd see locally along with all other env vars.

[Chuxel]

@chrmarti Ok - shall we flip this to a bug to investigate?

I did find another workaround for nvm specifically, but I think this is a general problem to investigate. There is a NVM_SYMLINK_CURRENT env var that can be set. I'll add that into the definitions we have for now as a tactical first step.

@AndersonSFerreira Just tweak your Dockerfile as by changing:

ENV NVM_DIR=/home/vscode/.nvm

...to...

ENV NVM_DIR=/home/vscode/.nvm
ENV NVM_SYMLINK_CURRENT=true
ENV PATH=${NVM_DIR}/current/bin:${PATH}

[chrmarti]

debian:buster and alpine:3.12 both overwrite PATH in their /etc/profile and using -l triggers that:

docker run --rm -it -e PATH=/bin:/usr/bin:/test debian:buster /bin/bash -lic 'echo $PATH'

Outputs:

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Without the -l:

/bin:/usr/bin:/test

ubuntu:focal does not overwrite PATH.

It seems one possible fix would be to use only -i and not -l. That would source .bashrc instead of the profile files (looking at https://medium.com/@rajsek/zsh-bash-startup-files-loading-order-bashrc-zshrc-etc-e30045652f2e).

[Chuxel]

@chrmarti Yeah that is what I have been doing in the vscode-dev-containers test scripts actually (#!/bin/bash -i).

[Chuxel]

@chrmarti I think @edgonmsft heard about this on the Codespaces side as well, but from the terminal point of view. This also seems to break ENV PATH based updates as well in the terminal. So basically, tasks.json only works with ENV PATH, but Terminal only works with .bashrc.

EDIT: ^- This may be Codespaces specific.

[chrmarti]

In case this helps Codespaces: You can start server.sh with --disable-user-env-probe to avoid pulling in the environment from <shell> -lic ....

We might want to do our own probing for Remote-Containers using only -i to address the issue discussed here.

[AndersonSFerreira]

@chrmarti Ok - shall we flip this to a bug to investigate?

I did find another workaround for nvm specifically, but I think this is a general problem to investigate. There is a NVM_SYMLINK_CURRENT env var that can be set. I'll add that into the definitions we have for now as a tactical first step.

@AndersonSFerreira Just tweak your Dockerfile as by changing:

ENV NVM_DIR=/home/vscode/.nvm

...to...

ENV NVM_DIR=/home/vscode/.nvm
ENV NVM_SYMLINK_CURRENT=true
ENV PATH=${NVM_DIR}/current/bin:${PATH}

adding these lines made the project work

[chrmarti]

Interestingly NVM doesn't work in local VS Code's tasks either because it is a function and these are not extracted by shellEnv.ts. E.g., this task doesn't work:

{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "testenv",
            "command": "nvm",
            "type": "process",
            "args": [
                "ls"
            ]
        }
    ]
}

[Chuxel]

@chrmarti Interesting - what I did to see if I got the right version of Node was this:

{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "testenv",
            "command": "which",
            "type": "process",
            "args": [
                "node"
            ]
        }
    ]
}

For me returns /Users/chuck/.nvm/versions/node/v10.16.3/bin/node. There is a system version as well, but it's at /usr/local/bin/node.

I'll have to try on a Linux machine to see if things are different there.

[chrmarti]

I see our Node 12 DevContainer image take 500-600ms for running $SHELL -ic printenv. (Plain Alpine and Debian take 150-200ms.) That is 22% increase when rebuilding the container. The PATH is then set up as expected. I'm not sure we should enable it by default given its overhead.

[jhampson-dbre]

I think we are being impacted by .bashrc not being used by Remote Container extension (v0.122.1) in a similar way, but unrelated to tasks.

We have been using Remote SSH extension with custom built "Automation Tools" virtual machine built from Centos 7. We are working on moving to a Docker container based on centos:7 using Remote Container extension to reduce host machine overhead, improve startup time, decrease download size, etc.

Centos 7 base repo has an old Git 1.8, so we use the Git 2.18 from Software Collection Library. This gets installed in /opt/rh/rh-git218/root/usr/bin/git and is added to the PATH inside the container with this line in .bashrc

source scl_source enable rh-git218

We had no problems with this with Remote SSH extension, but in our dev container, we get a message on startup that says "Git could not be found". Running, which git from the terminal shows git as expected.

Adding git.path to devcontainer.json works as a partial workaround, but we still see this error in the Git log about missing libraries, which are normally added to LD_LIBRARY_PATH in .bashrc from source scl_source enable rh-git218

> git fetch
/opt/rh/rh-git218/root/usr/libexec/git-core/git-remote-https: error while loading shared libraries: libcurl-httpd24.so.4: cannot open shared object file: No such file or directory

Now we have added LD_LIBRARY_PATH to remoteEnv in devcontainer.json to fix this error.

We have several other environment variables and Software Collections Library packages we use and it seems we may be heading down a path where we are basically going to duplicate our .bashrc in the remoteEnv setting in a roundabout way, since we have to run the source scl_source enable command to determine what the environment variables will look like to add them to remoteEnv.

For us, it would be preferable for Remote Containers to use our .bashrc so that it "just works".

[chrmarti]

To avoid everyone seeing a performance impact from a fix to this we could add a userEnvProbe property with possible values of none, loginShell and interactiveShell to the devcontainer.json. none would remain the default, loginShell would be what Remote-SSH does and interactiveShell would skip the profiles and instead source the RC files to get around #544.

[Chuxel]

@chrmarti That sounds like a reasonable approach!

[chrmarti]

Using loginInteractiveShell instead of loginShell because that is what VS Code is using. These two flags can be set independently and we might add loginShell (i.e., non-interactive) later.

[smhc]

Is there any way we can specify the type of 'env probe' at the container/image level without having to configure every named container we start? It is a bit of a hassle to connect, modify the named container config, disconnect, then reconnect every time we start a container with a different name/image.

e.g Could there be a 'VSCODE_ENV_PROBE=login' machine based environment variable we set in the Dockerfile/image? Or just a default type of 'env probe' for all containers?

As an aside, it seems like an unusual name for the configuration option given it is effectively determining the way vscode connects rather than 'probes'.

This is in reference to #3585

[chrmarti]

@smhc That is not possible at the moment, let's track it with #3585. Thanks.