Cannot define environment variables in .runsettings file

[Haplois]

.runsettings is supposed to have an XML section for environment variables, and while Microsoft has documented that, it doesn't seem to work, and I find almost nothing on it:

A unit test in this repository has:

 var runsettingsXml = @"<RunSettings>
                          <RunConfiguration>
                            <EnvironmentVariables>
                              <RANDOM_PATH>C:\temp</RANDOM_PATH>
                            </EnvironmentVariables>
                          </RunConfiguration>
                        </RunSettings>";

but that gives me:

TpTrace Warning: 0 : 12628, 4, 2021/03/11, 16:23:38.052, 270394899679, testhost.dll, Invalid settings 'RunConfiguration'. Unexpected XmlElement: 'EnvironmentVariables'.

Thoughts? I want something that runs in Visual Studio as well as something I can run in Jenkins.

Originally posted by @danielcor in #669 (comment)

[Haplois]

@danielcor, can you share the version of the Test Platform you're using?

A quick repro would be great to investigate this; if not, please run your tests enabling diagnostic logs and share them with us.

[Sanan07]

I just checked and it is true. We do not read EnvironmentVariables node value from RunConfiguration, only write log if meet some unexpected node name. Code is here.

[danielcor]

dotnet --version returns 5.0.200 and I'm running on Microsoft Windows
The application is dotnet core 3.1

I'm using the same version of the mstest nugets in both:

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.7.1" />
<PackageReference Include="MSTest.TestAdapter" Version="2.1.2" />
<PackageReference Include="MSTest.TestFramework" Version="2.1.2" />

I tried creating a test case, and the environment variable was set as they should be, but the diagnostic file, showed the same error, which is peculiar.

TpTrace Warning: 0 : 29356, 4, 2021/03/11, 17:09:17.457, 297788799430, testhost.dll, Invalid settings 'RunConfiguration'. Unexpected XmlElement: 'EnvironmentVariables'.

diag.host.21-03-11_17-09-16_12753_6.txt

Unfortunately my unit test is much more complicated, and I can't share it.. So will continue to try see what the difference is that causes it not to read this Xml section.

[ViktorHofer]

I just checked and it is true. We do not read EnvironmentVariables node value from RunConfiguration, only write log if meet some unexpected node name. Code is here.

That isn't true. The EnvironmentArgumentProcessor and other code paths read the environment variables from the runsettings file:

• vstest/src/vstest.console/Processors/EnvironmentArgumentProcessor.cs

  Line 127 in cda2550

  var node = this.runSettingsProvider.QueryRunSettingsNode($"RunConfiguration.EnvironmentVariables.{key}");

• vstest/src/Microsoft.TestPlatform.Utilities/InferRunSettingsHelper.cs

  Line 398 in 35d2279

  var node = runSettingsNavigator.SelectSingleNode(EnvironmentVariablesNodePath);

[danielcor]

@Sanan07

That explains the error message. What it doesn't explain is why it works sometimes and not other times. A new test project I created in my solution that uses the same .runsettings, set solution wide in Test/ConfigureRunsettings is able to read an environment variable, and another can't.

The test case that created the diagnostic above, I shared below.

testcase.zip

Works but produces an error in the diagnostics when run from the command line like:

dotnet test -d diag.txt -s .\test.runsettings .

[danielcor]

OK, I figured out why my complex unit tests weren't doing as expected. I left out the line of code where I read in the environment variables to be included in Configuration.

However, could someone fix the incorrect error message? It led me to troubleshoot the wrong thing!

[Sanan07]

@ViktorHofer You are right I missed that part. For EnvironmentArgumentProcessor, console parameter should be specified during execution. For vstest.console.exe it works in this way :

vstest.console.exe test.dll /e:Key=Value

@danielcor We can think about fixing error message for EnvironmetVariables in RunConfiguraiton. Thanks for pointing out.

[nohwnd]

dotnet/vscode-csharp#5131 (comment) Another example of how to properly set runsettings.