Passing the ASP.NET Core Environment to the Dotnet Test Runner

[RehanSaeed]

Moved from xunit/xunit#857. It would be incredibly useful if I could do something like this in my functional tests:

dotnet test --environment 'Production'

Instead of having to set the ASPNETCORE_ENVIRONMENT environment variable before running the test and then resetting it at the end of the test. The other advantage is that you can run more than one set of tests at the same time which is important for build machines.

[dzirkler]

@RehanSaeed as a workaround on Mac & Linux you can do this:
ASPNETCORE_ENVIRONMENT=Production dotnet test

Unfortunately, it doesn't work on Windows.

[jmevel]

Ho thanks @dzirkler, it means I can do this on Bash on Windows 10 that's cool.
Unfortunately the issue still exist for build machines...

[lski]

Does not the following workaround work in cmd?

ASPNETCORE_ENVIRONMENT=Production && dotnet test

[abakumov-v]

This commands work for me:

• powershell

($env:ASPNETCORE_ENVIRONMENT="Staging") | dotnet test -v n

• windows batch:

Set "ASPNETCORE_ENVIRONMENT=Staging" && dotnet test -v n

In both cases environment variable ASPNETCORE_ENVIRONMENT is set for current session only - you can check that it is right by opening other powershell/bat windows and type:

• powershell:

(get-item env:ASPNETCORE_ENVIRONMENT).Value

• windows batch:

echo %ASPNETCORE_ENVIRONMENT%

[iberodev]

What's the meaning of setting an environment variable as VARIABLE = env_value && dotnet test?
Does this workaround apply for dotnet vstest?
And the last question, is it required to reset the environment variable back to ?? (don't know what's the default value when executing outside VS's launchSettings) after the dotnet test or dotnet vstest has finalised?

I am also looking to have some appsettings.Test.json settings file with values for integration/component tests and configure my Gitlab CI/CD to execute the tests under some specific enviroment so that these settings are used

[lmcarreiro]

Is there another way to define inside some json file, which environment variables will be passed to the application when running from a test runner?

When running the ASP.NET Core projects, who is responsible for reading the launchSettings.json and setting the environment variables? Is it done by dotnet-cli or is it a Visual Studio specific way of setting env vars?

Is there a way to load these vars (from launchSettings.json) when running tests?

[PedroS11]

Using the variables defined in launchSettings.json would be perfect. Is it possible?

[kbirger]

That seems to be a murky subject. From another issue which I can't find right now, the support seems like it should exist, but folks are commenting that it doesn't work.

[cescoferraro]

I have stumbled upon this yesterday. although it drove me mad, it seems reasonable that tests should not rely on the environment. I have used this library to circumvent the issue.
https://github.com/bolorundurowb/dotenv.net

[nohwnd]

Summarizing, and correcting the information in this thread so we have a clearer way going forward.

as a workaround on Mac & Linux you can do this:
ASPNETCORE_ENVIRONMENT=Production dotnet test

This is true. It will add the environment variable to the current environment variables only for the current command (dotnet test).

Unfortunately, it doesn't work on Windows.

This is partially true. On Windows you can set environment variable for Process, User or Machine, but not for current command. Workarounds exists, but they require capturing all variables before command run and resetting them after (see an implementation of this in PowerShell).

On Windows

On Windows we are able to set environment variables per process, user and (given we have enough permissions) per machine. The default is Process, and the standard way to do that in PowerShell is using the Environment drive:

$env:ASPNETCORE_ENVIRONMENT="Staging"

This will set the variable for the current process, and every process that will start from this process (child process) will inherit all of the environment variables, including this one.

Alternatively in cmd SET can be used to do the same thing:

SET ASPNETCORE_ENVIRONMENT=Staging

(We can confirm the variable was set by querying it $env:ASPNETCORE_ENVIRONMENT in PowerShell or SET ASPNETCORE_ENVIRONMENT in cmd. )

Inheriting variables

On Windows a process will inherit all environment variables from parent when it starts. We can confirm that easily by starting a powershell instance, removing all env variables except PATH and then running cmd and dump all env variables (PowerShell is not happy when we start from such a bare environment).

PS> Get-ChildItem Env:\

Name                           Value
----                           -----
...etc
CommonProgramFiles             C:\Program Files\Common Files
CommonProgramFiles(x86)        C:\Program Files (x86)\Common Files
CommonProgramW6432             C:\Program Files\Common Files
... etc 

PS> Get-ChildItem Env:\ | where Name -ne PATH | Remove-Item
PS> cmd.exe /c SET
COMSPEC=C:\WINDOWS\system32\cmd.exe
Path=C:\Python38\Scripts\;C:\Python38\;C:\Program Files\Common Files....
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.JS;.WS;.MSC
PROMPT=$P$G

The cmd in fact inherited only the Path environment variable and defined few of it's own on start. It is important to note that we are still only making changes in the process scope. No changes are done to the environment variables on the User or Machine.

Running dotnet test

In PowerShell

The proposed workaround works, but is not ideal because it relies on a PowerShell quirk. () around assignment will also output the assigned value "Staging" into the pipleline which then triggers dotnet test -v n. Failing to provide the () around the assignment will either not run the command or will report Test Run Failed..

# works 
($env:ASPNETCORE_ENVIRONMENT="Staging") | dotnet test -v n
# - DO NOT do this
$env:ASPNETCORE_ENVIRONMENT="Staging" | dotnet test -v n

A better way to do this is simply set the variable on it's own line.

$env:ASPNETCORE_ENVIRONMENT="Staging"
dotnet test
dotnet test  
dotnet test

Or for visibility, and easier editing on re-runs put the two commands on the same line and terminate the first one by the command terminator ;.

$env:ASPNETCORE_ENVIRONMENT="Staging"; dotnet test

# or even 
$env:ASPNETCORE_ENVIRONMENT="Staging"; $env:SOME_OTHER_VARIABLE=10; dotnet test 

Remember that the variables are still set per process, you are just running multiple commands on the same line.

In cmd

Here is the same thing in cmd, joined by &. In this context you can use & and && interchangably.

SET ASPNETCORE_ENVIRONMENT=Staging& SET SOME_OTHER_VARIABLE=10& dotnet test

⚠ Notice that there are no quotes around the value and no spaces after the value. The value is taken literally, and will contain "Staging"<space> if you use quotes and spaces.

REM - DO NOT do this
SET ASPNETCORE_ENVIRONMENT="Staging" & ECHO '%ASPNETCORE_ENVIRONMENT%'
'"Staging" '

Running with settings from launchSettings.json

As a quick prototype I parsed the file and tried to start dotnet test with just that environment and quickly run into issues. In theory we are able to reduce the environment only to the values provided to the user and the values required by the runtime (and our internal way of communicaiting). This is further complicated by different versions of dotnet respecting different env variables (e.g. NUGET_PACKAGES does not seem to work on my 3.1.100 and I've seen issues about other variables as well).

{
  "profiles": {
    "UnitTestProject1": {
      "commandName": "Project",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Staging"
      }
    }
  }
}

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;

namespace UnitTestProject1
{
    [TestClass]
    public class UnitTest1
    {
        [TestMethod]
        public void TestMethod1()
        {
            Assert.AreEqual("Staging", Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT"));
        }
    }
}

$projectName = "UnitTestProject1"
$launchSettingsPath = "Properties/launchSettings.json"

$startInfo = [Diagnostics.ProcessStartInfo]::new()
$startInfo.UseShellExecute = $false
$startInfo.CreateNoWindow = $false
$variablesToKeep = $startInfo.EnvironmentVariables | where { 
    $_.Name.ToUpperInvariant() -in @(
        'PATH', 'HOME', 'HOMEDRIVE', 
        'HOMEPATH', 'USERPROFILE', 
        'PROGRAMFILES', 'PROGRAMDATA', 
        'APPDATA'
    )
}

# DO NOT do this. uncomment to clear all variables, dotnet will not start
# $startInfo.EnvironmentVariables.Clear()

if (Test-Path $launchSettingsPath) {
    $launchSettings = Get-Content $launchSettingsPath | ConvertFrom-Json

    $variables = $launchSettings.profiles."$projectName".environmentVariables
    foreach ($p in $variables.PSObject.Properties) {
        $startInfo.EnvironmentVariables.Add($p.Name, $p.Value)
    }
}

# DO NOT do this. uncomment to add the kept variables after claring, dotnet test will start but fail
# to setup communication with the underlying processes
# foreach ($v in $variablesToKeep) {
#     $startInfo.EnvironmentVariables.Add($v.Name, $v.Value)
# }

# this would be better but does not work
# $startInfo.EnvironmentVariables.Add('NUGET_PACKAGES', (Resolve-Path '~/.nuget/packages').ProviderPath)

'Starting with:'
$startInfo.EnvironmentVariables

$startInfo.FileName='dotnet'
$startInfo.Arguments = 'test'
$startInfo.WorkingDirectory = $pwd.Path
$process = [Diagnostics.Process]::new()
$process.StartInfo = $startInfo

$process.Start()

This will pass as long as we are passing all environment variables. So it could be a viable workaround until this is properly implemented. 🥳🥳🥳

[nohwnd]

Update: revisited this proposal to reflect the UX guideline of dotnet cli

I identified these issues in this thread:

Set environment variables for dotnet test

I like the idea, but would propose to implement it in a way that is not specific to ASP.NET. I think the following syntax would be reasonable:

dotnet test --environment ASPNETCORE_ENVIRONMENT=Staging
dotnet test -e ASPNETCORE_ENVIRONMENT=Staging

# multiple values by repeating the same param 
dotnet test -e ASPNETCORE_ENVIRONMENT=Staging -e SOME_OTHER_VARIABLE=10
# multiple values split by comma 
dotnet test -e "ASPNETCORE_ENVIRONMENT=Staging", "SOME_OTHER_VARIABLE=10"

The multiple value approach is great because then we don't run into many issues parsing the values, but might be foreign to some. Also I am not sure if there is a guideline for dotnet tools. This option is not explicitly mentioned in the UX guideline, but it's mentioned by this getopt syntax guideline and also commonly used (e.g. git commit -m "first line" -m "next line" -m "next line", or docker run -e var1=abc -e var2=gef`).

Loading launchSettings.json

Loading launchSettings.json from the test project would take all environment variables from the specified profile. There would be two options:

--launch-settings  (default: Properties/launchSettings.json)
--launch-settings-profile (default: <projectName>)

These options would have no short for. It is tempting to do -l and -p but the first one is taken by logger, and the second one would be more suited for Project, or Platform if there is such option in the future.

dotnet test
dotnet test --launch-settings-profile Debugging
dotnet test --launch-settings launchSettings.special.json
dotnet test --launch-settings launchSettings.special.json --launch-settings-profile Debugging

This would take the launchsettings from the given profile and add the variables to the ones that are already defined. Then all the env variables would be accessible in the test.

It would NOT clean the environment, and only use the variables from the file. This is how it currently works as well. All env variables from the system, and possibly extra ones taken from Visual Studio, are available in the test.

Combining this with --environment, the launch settings would be added first and then the variables defined by --environment option.

This assumes that each csproj is loaded into it's own process is that true for dotnet core @AbhitejJohn ? I did some experiments that confirmed it, but maybe there are some edge cases.

Should there also be an option to ignore launch settings?

Run multiple test sets on a single machine

This should be a non-issue, as long the test sets are running in their own processes, because the environment variables are set per process. You could run into problem where the second process gets both ENV1 and ENV2, but that is how the OS works, and it can be solved not setting the env variable in the current process, but rather by starting the process via ProcessStartInfo & Process, and setting the env variables just for the new process as shown above.

SET ENV1=1
dotnet test

SET ENV2=2
dotnet test

[AbhitejJohn]

This assumes that each csproj is loaded into it's own process is that true for dotnet core @AbhitejJohn ? I did some experiments that confirmed it, but maybe there are some edge cases.

That is correct. Link.

[nohwnd]

Note to self: Use --launch-profile name for the parameter, dotnet run uses it.

[Haplois]

This is partly done. It's possible to pass environment variables into the dotnet test command via the --environment parameter with dotnet/sdk#13436. --launch-profile support is on hold for the time being because it's part of the dotnet run command and needs some more development by its owners; we'll implement the support on the dotnet test after the work is done.

[danielcor]

.runsettings is supposed to have an XML section for environment variables, and while Microsoft has documented that, it doesn't seem to work, and I find almost nothing on it:

A unit test in this repository has:

 var runsettingsXml = @"<RunSettings>
                          <RunConfiguration>
                            <EnvironmentVariables>
                              <RANDOM_PATH>C:\temp</RANDOM_PATH>
                            </EnvironmentVariables>
                          </RunConfiguration>
                        </RunSettings>";

but that gives me:

TpTrace Warning: 0 : 12628, 4, 2021/03/11, 16:23:38.052, 270394899679, testhost.dll, Invalid settings 'RunConfiguration'. Unexpected XmlElement: 'EnvironmentVariables'.

Thoughts? I want something that runs in Visual Studio as well as something I can run in Jenkins.

[nohwnd]

This is a new feature and won't be implemented, we are focusing on adding new features to Testing.Platform instead. https://aka.ms/testingplatform