Testing: Add a LiveParallelizable that runs Live tests in parallel

[tg-msft]

We need to run tests sequentially when AZURE_TEST_MODE is Record or Playback. It would be nice to run tests in parallel when AZURE_TEST_MODE is Live but nUnit doesn't allow that kind of flexibility out of the box. We should investigate adding a custom LiveParallelizable attribute at the assembly level.

[vinagesh]

We recently started using the test framework and see that if there are multiple tests within a class, we cannot run the tests with the Parallelizable attribute. But we would like to run the live tests in parallel. Voting to have this feature.

[kasobol-msft]

@tg-msft @pakrym Would it make sense to make recording framework parallel-friendly instead of this? I'd appreciate faster CI builds.

FYI I was trying some parallelization here #16883 but I didn't want to pursue this further as it turned out to be quite complex to solve.

[pakrym]

I don't think we can make it parallel friendly because of how NUnit manages test class instances (not creating an instance per test).

[kasobol-msft]

@pakrym We won't be able to go as far as parallelization per single test. That is probably true for live tests (at least for Storage) due to limitation you mentioned.

However, there's a way to apply NUnit Parallelizable attribute in such a way that test classes are executed in parallel but within class methods go serially (there are few ways to achieve that). I would take ability to go parallel per class any day instead of current situation.

In my experiment I applied [Parallelizable(ParallelScope.Self)] on StorageTestBase. The issue I was facing is somewhere in this code

azure-sdk-for-net/sdk/core/Azure.Core.TestFramework/src/ClientTestFixtureAttribute.cs

Lines 58 to 67 in 52fb9b7

	var suitePermutations = GeneratePermutations();
	foreach (var (fixture, isAsync, serviceVersion, parameter) in suitePermutations)
	{
	foreach (TestSuite testSuite in fixture.BuildFrom(typeInfo, filter))
	{
	Process(testSuite, serviceVersion, isAsync, parameter);
	yield return testSuite;
	}
	}

. Somehow it makes recording/recordingtransport or something in that vicinity being shared between sync/async mode which I couldn't pin point. When I hacked this to generate only one permutation per tests I got pretty good results, i.e. whole assembly ran per-class-parallel and tests were green (in playback mode).

[pakrym]

Very good point Kamil. I think the parallelization of test classes is technically achieveable.