CoreCLR test suite optimization proposal: support for test project grouping #54512

trylek · 2021-06-21T18:52:16Z

Problem description

Current CoreCLR Pri1 test set has over 10K individual test projects. This is beyond the means of a single msbuild execution and is mitigated by partitioning the test projects into subgroups. Today at least three such partitionings exist (partitioning during test build, partitioning into XUnit wrappers, partitioning for Helix execution). While @echesakov did his best to make the Helix partitioning as good as possible, the entire logic adds enormous complexity to the test system, complicates developer ramp-up and is a constant cause of developer complaints. The 10K separate apps also mean 10K .NET Core runtime startups incurring enormous testing cost, it's not hard to imagine that the repeated .NET Core runtime initializations take an equal or greater amount of time than the actual test code execution.

Caveat - we don't yet have any hard data to substantiate this claim. I'm working on figuring out how to produce it in some form.

Ideal state

As I personally heard in presentations by @jaredpar and @stephentoub, perf optimization of Roslyn and libraries tests that took place several years ago involved the reduction of the number of separate test apps as a key step. I believe we should take the same route in CoreCLR testing; in bulk testing (local or lab Pri0 / Pri1 testing) we should run fewer than 1K test apps, ideally less than 500. Once that happens, we should be able to remove all the partitioning goo and just run the tests one by one, both locally and in Helix.

Downsides, challenges and problems to solve

Today, about 3/4 of the test suite corresponds to the JIT unit tests - a search in my runtime repo clone under src\tests\JIT for *.csproj/ilproj yields 7312 matches. If we're serious about this effort, we must tackle JIT tests first. According to the proposed ideal state, we should strive to reduce the number of separate apps to about 300~400. I think that roughly corresponds to two subdirectory levels under JIT (e.g. Methodical\divrem) but I have yet to provide more precise numbers.

While the test aggregation is expected to solve a known set of problems (test system complexity caused by the partitioning systems, performance of test build and execution), it has the potential to introduce a new set of problems we should plan ahead of and work on fixing or mitigating as part of the proposal. In particular, a larger number of tests being run as a single app can complicate debugging, profiling, TTT analysis, and JIT dump analysis; runtime and / or hard crash in one test tears down the subsequent tests in an aggregated test app, reducing test coverage in the presence of failures.

The counter-arguments clearly highlight sets of tests that are unsuitable for aggregation - typically interop tests where the individual tests sometimes tamper with the machine state (e.g. by registering COM classes), perhaps also the GC tests that are often lengthy and / or have the potential to tear down the app like in the case of negative OOM tests.

Even in cases where the test aggregation is expected to be benign, e.g. in the case of the JIT methodical tests, we still need to address the question of aggregation hampering developer productivity, typically in various diagnostic scenarios. @AndyAyersMS proposed a dual system where the tests would be aggregated by default in bulk testing but the developer could explicitly request the build of a single test case to mitigate the aforementioned complications.

Proposed solution

I have yet to make any real experiments in this space but it seems to me that we might be able to solve much of this puzzle by introduction of group projects. My initial thinking is that, for a particular test project, e.g. JIT\Methodical\divrem\div\i4div_cs_do.csproj, we would use a new property to declare that the test is a part of the test group project, say, JIT\Methodical\divrem\divrem_do.csproj (JIT tests often come in groups that require different optimization flags so that would need preserving in the groupings). Hopefully it should be possible to tweak msbuild to normally build just the group projects; these would need to use either some form of code generators or reflection to run all the relevant test “cases” represented by the grouped projects but that should no longer blow up msbuild as we could easily build the individual group projects serially.

I already have a work item on adding a new command-line option to src\tests\build.cmd/sh to let developers build just a particular test project or project subtree. It should be trivial to consolidate this option with the proposed project grouping such that in bulk testing we’d end up with just the group projects whereas targeted local scenarios would end up producing a single-test executable (as before) with the caveat that trying to build the entire tree in this “separate” mode would likely trigger an msbuild OOM or some other failure.

Proposed sequencing

I’m going to perform at least a series of local experiments to measure how much of the running time of the individual tests is coming from runtime initialization vs. actual test code execution and I’ll share them on this issue thread. I have yet to see whether this approach can be easily applied in the lab. Locally it might suffice to tweak R2RTest to use ETW mode to monitor at which point Main got executed.
Assuming the perf experiments do confirm a perf win in test grouping (especially for tiny tests like the JIT unit tests) and we agree on this proposal in some form, I’ll look into implementing its basic underpinnings in the CoreCLR test build / execution infra scripts and I’ll test the approach on a small suite of JIT tests.
Once the PR per (2) is merged in, we can trigger a “quality-week-like” combined effort to apply the technique to additional CoreCLR test areas. At this point we would be still using the pre-existing infrastructure including the XUnit wrappers and test partitionings, we’d just gradually reduce the number of test apps being run. (The proposed conservative approach doesn’t address actual test code merging i.e. the test build time win will likely be smaller if any. This is further aggravated by the fact that many of the JIT unit tests come in form of IL source code.)
The work per (3) should yield gradually accumulating benefits in form of reducing the total CoreCLR test running time, both locally and in the lab. Once the work advances enough so that we get under the envisioned 1K test projects, we can proceed to experimenting with removal of the test partitionings. At that point we may be also able to consider removing the Pri0 / Pri1 distinction and always run all the tests.

Thanks

Tomas

/cc @dotnet/runtime-infrastructure

The text was updated successfully, but these errors were encountered:

ghost · 2021-06-21T18:52:18Z

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

Problem description

Current CoreCLR Pri1 test set has over 10K individual test projects. This is beyond the means of a single msbuild execution and is mitigated by partitioning the test projects into subgroups. Today at least three such partitionings exist (partitioning during test build, partitioning into XUnit wrappers, partitioning for Helix execution). While @echesakov did his best to make the Helix partitioning as good as possible, the entire logic adds enormous complexity to the test system, complicates developer ramp-up and is a constant cause of developer complaints. The 10K separate apps also mean 10K .NET Core runtime startups incurring enormous testing cost, it's not hard to imagine that the repeated .NET Core runtime initializations take an equal or greater amount of time than the actual test code execution.

Caveat - we don't yet have any hard data to substantiate this claim. I'm working on figuring out how to produce it in some form.

Ideal state

As I personally heard in presentations by @jaredpar and @stephentoub, perf optimization of Roslyn and libraries tests that took place several years ago involved the reduction of the number of separate test apps as a key step. I believe we should take the same route in CoreCLR testing; in bulk testing (local or lab Pri0 / Pri1 testing) we should run fewer than 1K test apps, ideally less than 500. Once that happens, we should be able to remove all the partitioning goo and just run the tests one by one, both locally and in Helix.

Downsides, challenges and problems to solve

Today, about 3/4 of the test suite corresponds to the JIT unit tests - a search in my runtime repo clone under src\tests\JIT for *.csproj/ilproj yields 7312 matches. If we're serious about this effort, we must tackle JIT tests first. According to the proposed ideal state, we should strive to reduce the number of separate apps to about 300~400. I think that roughly corresponds to two subdirectory levels under JIT (e.g. Methodical\divrem) but I have yet to provide more precise numbers.

While the test aggregation is expected to solve a known set of problems (test system complexity caused by the partitioning systems, performance of test build and execution), it has the potential to introduce a new set of problems we should plan ahead of and work on fixing or mitigating as part of the proposal. In particular, a larger number tests being run as a single app can complicate debugging, profiling, TTT analysis, and JIT dump analysis; runtime and / or hard crash in one test tears down the subsequent tests in an aggregated test app, reducing test coverage in the presence of failures.

The counter-arguments clearly highlight sets of tests that are unsuitable for aggregation - typically interop tests where the individual tests sometimes tamper with the machine state (e.g. by registering COM classes), perhaps also the GC tests that are often lengthy and / or have the potential to tear down the app like in the case of negative OOM tests.

Even in cases where the test aggregation is expected to be benign, e.g. in the case of the JIT methodical tests, we still need to address the question of aggregation hampering developer productivity, typically in various diagnostic scenarios. @AndyAyersMS proposed a dual system where the tests would be aggregated by default in bulk testing but the developer could explicitly request the build of a single test case to mitigate the aforementioned complications.

Proposed solution

I have yet to make any real experiments in this space but it seems to me that we might be able to solve much of this puzzle by introduction of group projects. My initial thinking is that, for a particular test project, e.g. JIT\Methodical\divrem\div\i4div_cs_do.csproj, we would use a new property to declare that the test is a part of the test group project, say, JIT\Methodical\divrem\divrem_do.csproj (JIT tests often come in groups that require different optimization flags so that would need preserving in the groupings). Hopefully it should be possible to tweak msbuild to normally build just the group projects; these would need to use either some form of code generators or reflection to run all the relevant test “cases” represented by the grouped projects but that should no longer blow up msbuild as we could easily build the individual group projects serially.

I already have a work item on adding a new command-line option to src\tests\build.cmd/sh to let developers build just a particular test project or project subtree. It should be trivial to consolidate this option with the proposed project grouping such that in bulk testing we’d end up with just the group projects whereas targeted local scenarios would end up producing a single-test executable (as before) with the caveat that trying to build the entire tree in this “separate” mode would likely trigger an msbuild OOM or some other failure.

Proposed sequencing

I’m going to perform at least a series of local experiments to measure how much of the running time of the individual tests is coming from runtime initialization vs. actual test code execution and I’ll share them on this issue thread. I have yet to see whether this approach can be easily applied in the lab. Locally it might suffice to tweak R2RTest to use ETW mode to monitor at which point Main got executed.
Assuming the perf experiments do confirm a perf win in test grouping (especially for tiny tests like the JIT unit test) and we agree on this proposal in some form, I’ll look into implementing its basic underpinnings in the CoreCLR test build / execution infra scripts and I’ll test the approach on a small suite of JIT tests.
Once the PR per (2) is merged in, we can trigger a “quality-week-like” combined effort to apply the technique to additional CoreCLR test areas. At this point we would be still using the pre-existing infrastructure including the XUnit wrappers and test partitionings, we’d just gradually reduce the number of test apps being run. (The proposed conservative approach doesn’t address actual test code merging i.e. the test build time win will likely be smaller if any. This is further aggravated by the fact that many of the JIT unit test come in form of IL source code.)
The work per (3) should yield gradually accumulating benefits in form of reducing the total CoreCLR test running time, both locally and in the lab. Once the work advances enough so that we get under the envisioned 1K test projects, we can proceed to experimenting with removal of the test partitionings. At that point we may be also able to consider removing the Pri0 / Pri1 distinction and always run all the tests.

Thanks

Tomas

/cc @dotnet/runtime-infrastructure

Author:	trylek
Assignees:	-
Labels:	`area-Infrastructure-coreclr`
Milestone:	-

trylek · 2021-06-21T18:53:49Z

/cc @dotnet/jit-contrib

trylek · 2021-06-21T18:59:37Z

/cc @dotnet/gc

trylek · 2021-06-21T19:09:31Z

/cc @agocke @jkotas @janvorli @mangod9

trylek · 2021-06-21T19:22:23Z

/cc @tommcdon @hoyosjs

trylek · 2021-06-21T19:24:43Z

/cc @naricc @fanyang-mono

jkotas · 2021-06-21T19:55:21Z

we should strive to reduce the number of separate apps to about 300~400.

This feels still way too much. I think we should be shooting for < 40.

It is common to have thousand of tests per tests app in the libraries partition. Having a few hundred of tests per test app would still be less that what you regularly see in libraries.

the test aggregation

There are two independent aggregations:

Build aggregation: how many test .exes and .dlls are built.
Execution aggregation: how many times is the .exe executed. One exe can be executed multiple times to deal with the isolation problem that you have pointed out. Libraries tests have RemoteExecutor helper for that.

I think we should deal with both types of aggregation at the same time, so that it is solved once for good. I think the ideal state is:

Build produces .dlls that contains 100s of tests each
By default, each .dll is executed once to run all tests. Tests that need to run in seperate process can use RemoteExecutor or something similar.

We would need to change how the tests are authored to make this happen. The tests cannot use the regular Main method as the entrypoint anymore since you cannot have multiple Main methods per binary.

My proposal would be:

Use the regular XUnit [Fact] attributes to mark test entrypoints. We would bulk edit the tests and change their Main method to be a regular method with [Fact] attribute.
Create a source generator that enumerates all methods marked with [Fact] attribute in the project and creates Main method that calls all of them. Optionally, the Main method can take a test name and switch on it so that it is still easy to run individual test (would it be enough to address the diagnostic concern?).
- https://github.com/dotnet/runtime/tree/main/src/libraries/Common/tests/StaticTestGenerator can be used as an inspiration for this source generator

The reason for using source generator and not XUnit runner to discover the tests is debuggability. XUnit runner is a reflection stress test and thus it is not suitable as a test driver for the low-level runtime.

The nice side-effect of using the standard XUnit attributes for runtime tests is that the authoring of core runtime tests will become more similar to authoring of libraries tests.

hoyosjs · 2021-06-21T20:24:06Z

One thing I was thinking about this approach is: does this mean catastrophic failures in one test will take down the whole work item execution? Maybe this is something the remote executor can help with. Also, with the generated Main approach we would probably need to work out the reporting mechanism + coredump mechanism as what we have today would fall short and helix wouldn't report these.

trylek · 2021-06-21T21:15:27Z

I can theoretically imagine that we might be able to tweak the test scripts such that, when the aggregate test app crashes in a catastrophic manner, we'd run it a second time to execute the individual test cases one by one as separate apps, I guess that's what roughly corresponds to the remote executor. For the test authoring, I guess the biggest challenge is the JIT IL tests, I was originally thinking we might be able to keep them unchanged but if that doesn't work, I'm definitely open to other ideas.

jkoritzinsky · 2021-06-21T22:37:33Z

My main request if we go a remote-executor route would be that there is some mode to have the remote executor spit out the command line required to launch the process it is starting. One of the hardest problems with RemoteExecutor is being able to figure out how to debug the child process.

Additionally, if we go the route of a source-generated xunit-esque test execution runner with RemoteExecutor-esque features for tests that require out-of-proc launching, I'd like it if we could design the support such that a test author could also reuse whatever infra we have for launching the child process and capturing diagnostics for specialized cases (like the COM tests with native entry-points that test activation)

trylek · 2021-06-21T22:57:33Z

Frankly speaking, I think we should work hard to avoid child process executions whenever possible as I believe it makes a crucial difference w.r.t. test perf. For isolation-sensitive tests like interop tests we'll add specific provisions based on auditing where process isolation is required.

naricc · 2021-06-22T02:30:26Z

We are already doing a kind of build-aggregation of tests for Android and iOS tests, because it was simply impractical to package up each test as a separate app. (@fanyang-mono and @imhameed worked on this repsecitvely) I think this will need to be true for wasm-aot as well, because each individual wasm-app takes a long time to compile.

If we do this "test group" things, we may be able to also put each group in an app, which would simplify the design of those test lanes. But I am not sure if the tradeoffs are the same/compatilbe (i.e. how many tests can go in each app).

BruceForstall · 2021-06-22T23:48:54Z

A JIT team requirement is to execute as little managed code as possible before getting to the test being debugged. It sounds like the proposal above might mostly achieve this even with aggregated tests, for most debugging scenarios. A counter-example is the libraries tests, where debugging them involves JITing and running gobs of xunit w/ reflection, which is super slow and interferes with debugging (e.g., set a JIT/GC stress mode, xunit is stressed also before even getting to the desired code). I like the proposal that tests could optionally be built standalone, if possible. Small, standalone tests help greatly in platform bring-up scenarios.

I like Jan's suggestion about mass grouping, noting that the build grouping doesn't necessarily need to reflect Helix run-time grouping: if we have X built test assemblies and want to parallelize runs on Y machines, we don't need X == Y : especially if we can choose which subset of tests in a test assembly get run in any particular invocation. E.g., copy X.dll to two Helix machines, run half of the tests in X.dll on one machine, half on the other. This might not work quite so transparently, however, for crossgen tests, which will crossgen the entire test assembly no matter what subset of tests is run.

Grouping the tests probably makes is easier/simpler to copy tests between machines, e.g., from a Linux/x64 box doing cross-compilation to a Linux/arm32 "run" box.

The "test driver" will need to be very clear about which test is being run, which has passed/failed, how to rerun a failure (or pass). Of course, we need the results surfaced to Helix/AzDO properly.

How will per-test timeouts work? Will they only be per-test-assembly? That could cause a misbehaving test early in the run to prevent getting results from tests later in the run sequence.

trylek · 2021-06-24T20:01:02Z

Thanks @BruceForstall for your detailed and insightful feedback. I don't yet have all the answers to your question; as a first step I'm trying to collect some actual perf numbers and as part of this task I noticed a bunch of test duplicates. Would you be fine with cleaning these up as a preparatory step or is there some more subtle distinction to what I perceive as mere duplication? Examples:

https://github.com/dotnet/runtime/blob/main/src/tests/JIT/Methodical/NaN/arithm32_cs_d.csproj
https://github.com/dotnet/runtime/blob/main/src/tests/JIT/Methodical/NaN/arithm32_d.csproj

(and seven other pairs in the same folder)

https://github.com/dotnet/runtime/blob/main/src/tests/JIT/Methodical/xxobj/operand/refanyval.csproj
https://github.com/dotnet/runtime/blob/main/src/tests/JIT/Methodical/xxobj/operand/_dbgrefanyval.csproj

I see about two dozen similar cases and my local tooling should let me automate their identification. If you agree to the preparatory cleanup, I'll work on putting up a PR.

Thanks

Tomas

BruceForstall · 2021-06-24T20:11:12Z

Those look like dups to me.

Note that src\tests\JIT\CheckProjects\CheckProjects.cs is a tool to ensure test tests set various properties correctly. I haven't run it recently (and I'm not sure it runs in an automated fashion anywhere).

trylek · 2021-06-25T23:53:56Z

I have performed a simple experiment to get an initial reading on the perf implications of proposed test merging. The results seem to indicate potential for substantial build time speedup; I'm also seeing some runtime speedup but frankly not as pronounced as I expected. Most of the motivation in the issue description remains in place, I just have less hope that the change will automatically translate to drastic reduction of test running time - there are still chances the change will substantially speed up Helix execution by means of reducing the payloads but that's speculation at this point.

As the target for my initial experiment I picked the JIT\Methodical tests that seem well suited for merging. The Pri1 suite contains about 2K tests in this subtree approximately half of which are csproj and the other half are ilproj projects. I have limited the initial experiment to csproj as the IL is much harder to transform. I have basically open-coded a simple managed app that uses some shortcuts to try to mimic the proposed transformation - changing the Main methods in the individual tests to be normal methods and directly calling them from a generated wrapper project / source file. I have excluded about 50 tests that use constructs incompatible with my simplistic C# / project rewriter. The runtime perf results on my laptop are as follows (x64 release):

Vanilla src\tests\run release - 33 seconds.
Test wrapper calling into the original test assemblies - 29 seconds.
All tests compiled into a single assembly - 26 seconds.

It's probably worth noting that I'm respecting the d/r/do/ro distinctions so I'm actually generating four projects and running them in sequence. As you can see, the "fastest" variant (putting all tests in the subtree in a single assembly) reduces the execution time by about 22%.

On the other hand, putting all tests in a single assembly does reduce test build time in a substantial manner. On my laptop the managed test build portion of src\tests\build release -priority=1 (still limited to csproj projects under JIT\Methodical) takes about 190 seconds while the build of the four combined projects representing JIT\Methodical tests in the four build combinations (d / do / r / do) only takes about 24 seconds i.e. about 8 times improvement.

Summary: merging many tests together does have a drastic effect on total test build time; runtime perf improvement is also measurable but much less pronounced. Please note this could still be a big win for PR / CI runs as the test build constitutes a non-trivial portion of the total running time of these pipelines. In the latest CI run Pri0 test build took about 23 minutes; in the last outerloop run, Pri1 test build took about 47 minutes. It is also worth noting that this part is fundamental as all the Helix runs depend on it.

If purely hypothetically we were able to reduce test build time 8 times as the results for JIT\Methodical tests suggest, i.e. from 23 to ~3 minutes and from 47 to ~6 minutes, that would directly translate into total running times of the CoreCLR pipelines. This estimate has many caveats, e.g. many Interop tests use native components with different build time characteristics, merging multiple ilproj tests into a single assembly requires more complex IL transformations etc. but I believe there definitely is potential for improvement along the lines of this proposal.

BruceForstall · 2021-06-26T00:02:24Z

Note that we build and run the tests in a Checked config almost 100% of the time, so I'd measure that instead of release. There, I'd expect a bigger improvement.

jkotas · 2021-06-26T00:11:56Z

Vanilla src\tests\run release - 33 seconds.
Test wrapper calling into the original test assemblies - 29 seconds.
All tests compiled into a single assembly - 26 seconds.

Was the CPU utilization same between the different cases?

+1 on measuring checked JIT and runtime flavor

trylek · 2021-06-26T00:40:20Z

Thanks Bruce and Jan for your additional feedback. You're right on both accounts. In checked mode, the three numbers are:

Vanilla src\tests\run checked - 202 seconds.
Test wrapper calling into the original test assemblies - 96 seconds.
All tests compiled into a single assembly - 89 seconds.

For now I just observed CPU utilization in the task manager while running the tests. In (1), xUnit is obviously running the tests in parallel - after the initial test discovery CPU utilization quickly goes to 100% and stays there for the entire test duration. In contrast, both my "new" measurements per (2) and (3) involve CPU happily sitting at 22~25% utilization corresponding to just 1 out of my 4 cores being used. In other words, by fully leveraging parallelism we should be able to further improve case (2) and (3) to 96/4 ~ 24 seconds (about 8 times speedup). I assume that the difference between (2) and (3) is less pronounced in checked mode as the slower JIT and runtime in general dwarf the OS loader time needed to load the multiple assemblies in case (2).

trylek · 2021-06-26T00:58:04Z

In case anyone's interested in further experiments in this area, I have put the tool I wrote on our internal share

\\clrmain\public\writable\users\trylek\TestGrouping.zip

It basically receives a path into the GIT clone as its command-line argument (e.g. D:\git\runtime\src\tests\JIT\Methodical), rewrites the C# code and projects in the subtree and generates the eight wrapper projects - four projects per (2) and four per (3) - into the folder. This way git checkout / git clean -xdf can be easily used to undo the transformations when experimenting with the tool. The wrapper projects conform to the normal CoreCLR test project style so that they can be individually built using dotnet msbuild ...csproj and executed using the generated cmd script. I'll be happy to carry out any additional measurements using the tool based on your suggestions, at the end of the day it's actually quite easy to use.

trylek · 2021-08-16T21:23:24Z

As a next step in my experiments I have recently managed to leverage the internal iDNA technology to measure that in the archetypal "tiny JIT" test I'm always mentioning, i4div_cs_do, (on Windows x64 release) we carry out about 70M instructions before entering Main and then about 15M instructions within it. While anecdotal, I believe it further confirms that there is at least some value in test merging for lab testing purposes.

For our oncoming Quality week (next week of 8/23) I have proposed starting the initial preparatory steps, in particular cleaning up test duplicates and renaming tests to remove entrypoint name duplicates (getting rid of pairs of tests with the same qualified entrypoint name i.o.w. where the assembly, class and entrypoint are the same). Once this is done, I'll start working on the next step actually converting tests to XUnit style and on support for their merging.

For now there's no functional change to the behavior of the tests, I have just copied the bits to inject from Jeremy's example in his pending PR. I have spot-checked that some of the tests use Main with the command-line args argument. I'm not changing them in this PR, the signature only becomes important once we start actually merging the IL tests and I presume we'll clean that up at that point. Thanks Tomas Contributes to: dotnet#54512

fanyang-mono · 2021-11-12T23:34:34Z

Thanks @fanyang-mono for the suggestion, that sounds reasonable. @jkoritinsky is currently reviewing PR implementing the wrapper generator, I think that it should be compatible with dotnet publish but we'll need to validate that as part of the refactoring / grouping.

May I have the link to the PR?

trylek · 2021-11-12T23:35:32Z

Sure, here you are: #60846

naricc · 2021-11-14T13:32:00Z

@trylek So the wasm tests currently rely on taking the already-built tests and making a WasmApp bundle from all the assemblies at test run time (in the generated shell script), then running it. We get one AppBundle per test.

I am also struggling to bring up the wasm-aot lane as well: #57963

This will rely on actually being able to publish the test projects.

For now there's no functional change to the behavior of the tests, I have just copied the bits to inject from Jeremy's example in his pending PR. Thanks Tomas Contributes to: #54512

For now there's no functional change to the behavior of the tests, I have just copied the bits to inject from Jeremy's example in his pending PR. I have spot-checked that some of the tests use Main with the command-line args argument. I'm not changing them in this PR, the signature only becomes important once we start actually merging the IL tests and I presume we'll clean that up at that point. Thanks Tomas Contributes to: #54512

trylek · 2021-11-17T00:58:46Z

Thanks Nathan for your feedback. I think that the work Jeremy's been doing in #60846 is very clean w.r.t. standard dotnet build / publish behavior (much cleaner than the msbuild-based C# generator we're using today anyway); I guess it may turn out we need to fix some bits and pieces but I don't expect that to be a big deal.

For my part, by now I believe I have injected the [Fact] attributes on all ilproj tests where they're harmless. For csproj tests we can only start the conversion once Jeremy's PR has been merged in as otherwise we wouldn't have a runner for the new-style tests. As I said before, once the change is in, I plan to use it to switch over the Loader/classloader/TypeGeneratorTests first to get an initial reading on the perf deltas.

For the il tests in general, I'm wondering whether we want to keep the .module lines in the tests. I noticed arbitrary typos in the module names and they still refer to exes so I guess they are likely ignored. The only time I actually recall .module clauses being of some use was when catching some obscure build / runtime bugs related to tests with differing assembly vs. module names - as I'm under the impression it's not an officially supported scenario, I wonder to what extent we need to care moving forward.

I believe that despite its singular nature this is worth a separate review to make sure we're in agreement regarding the conversion principles. There are several things worth noting here: 1) According to the current plan we're continuously documenting in dotnet#54512 we want to remove command-line parameters from all test entrypoints. Variant tests driven by command-line parameters should be turned into multiple test cases marked with separate [Fact] attributes. 2) For ilproj tests, to facilitate local debugging, our current plan is to keep them runnable as standalone executables. This implies that ilproj tests comprising several [Fact] test entrypoints require a new entrypoint that just calls into the individual test cases. 3) The Roslyn-generated merged wrapper for the tests won't care about the "composite" main (that is for local debugging only), it will directly identify and use the individual test cases marked with [Fact] attributes. 4) In accordance with this scheme, such composite ILPROJ tests are specific in not having their entrypoint method itself marked with the [Fact] attribute. 5) Funnily enough this example nicely demonstrates the implied cleanup - the entire command-line machinery is only used for a handwritten switch to choose one of the three variants; moreover we only exercised two out of the three variants, possibly due to an authoring bug when creating the variant test, potentially caused by previous complexity of such endeavor. Thanks Tomas

trylek · 2021-11-17T23:10:58Z

I'm currently working on variant tests that pass command-line arguments from the msbuild scripts to the managed test execution. We're trying to eradicate those cases in order to simplify test authoring. I believe there are basically two patterns emerging:

We have multiple tests calling into the same final test exe with variations of command line. The vast majority of those in the runtime repo are the GC simulator tests.
We have unique tests that don't end up converging to the same root test with different command-line arguments, someone just happened to author the test such that it expects a certain argument.

For the GC simulator tests, I would love to hear feedback from @Maoni0 and @PeterSolMS how these should be ideally modelled. We're trying to make tests more similar to library tests, maybe in some of these cases we could use the [Theory] concept to call something multiple times with argument variations. It's not pressing at the moment but I believe it to be an opportunity to make the test suite more meaningful.

For the command-line test singletons I'm working on analyzing what exactly they're trying to achieve with the goal to refactor them to stop needing them.

Thanks

Tomas

trylek · 2021-11-17T23:11:40Z

SINGLE TESTS WITH COMMAND-LINE ARGUMENTS
----------------------------------------
d:\git\runtime5\src\tests\baseservices\exceptions\sharedexceptions\emptystacktrace\OOMException01.csproj -> -trustedexe
d:\git\runtime5\src\tests\baseservices\threading\regressions\beta1\347011.csproj -> 240
d:\git\runtime5\src\tests\GC\API\GC\Collect_Default_1.csproj -> 0
d:\git\runtime5\src\tests\GC\API\GC\Collect_Default_2.csproj -> 1
d:\git\runtime5\src\tests\GC\API\GC\Collect_Default_3.csproj -> 2
d:\git\runtime5\src\tests\GC\API\GC\Collect_Forced_1.csproj -> 0
d:\git\runtime5\src\tests\GC\API\GC\Collect_Forced_2.csproj -> 1
d:\git\runtime5\src\tests\GC\API\GC\Collect_Forced_3.csproj -> 2
d:\git\runtime5\src\tests\GC\API\GC\Collect_Optimized_1.csproj -> 0
d:\git\runtime5\src\tests\GC\API\GC\Collect_Optimized_2.csproj -> 1
d:\git\runtime5\src\tests\GC\API\GC\Collect_Optimized_3.csproj -> 2
d:\git\runtime5\src\tests\GC\API\WeakReference\multipleWRs_1.csproj -> 10000 track
d:\git\runtime5\src\tests\GC\API\WeakReference\multipleWRs.csproj -> 10000
d:\git\runtime5\src\tests\GC\Features\HeapExpansion\bestfit_1.csproj -> 1 1000 50000
d:\git\runtime5\src\tests\GC\Features\HeapExpansion\bestfit.csproj -> 1 1000 50000
d:\git\runtime5\src\tests\GC\LargeMemory\Allocation\finalizertest.csproj -> 2048
d:\git\runtime5\src\tests\GC\LargeMemory\Allocation\largeexceptiontest.csproj -> 2048
d:\git\runtime5\src\tests\GC\LargeMemory\API\gc\collect.csproj -> 2048
d:\git\runtime5\src\tests\GC\LargeMemory\API\gc\getgeneration.csproj -> 2048
d:\git\runtime5\src\tests\GC\LargeMemory\API\gc\gettotalmemory.csproj -> 2048
d:\git\runtime5\src\tests\GC\LargeMemory\API\gc\keepalive.csproj -> 2048
d:\git\runtime5\src\tests\GC\LargeMemory\API\gc\reregisterforfinalize.csproj -> 2048
d:\git\runtime5\src\tests\GC\LargeMemory\API\gc\suppressfinalize.csproj -> 2048
d:\git\runtime5\src\tests\GC\Scenarios\Dynamo\dynamo.csproj -> 1000 40 191919
d:\git\runtime5\src\tests\GC\Scenarios\GCBase1\gc_base1_1.csproj -> 8 100
d:\git\runtime5\src\tests\GC\Scenarios\GCBase1\gc_base1.csproj -> 3 100
d:\git\runtime5\src\tests\GC\Scenarios\GCSimulator\GCSimulator.csproj -> -t 1 -tp 0 -dz 17 -sdz 17 -dc 20000 -sdc 8000 -lt 2 -dp 0.0 -dw 0.4
d:\git\runtime5\src\tests\GC\Scenarios\RanCollect\rancollect.csproj -> 7 40 4 77
d:\git\runtime5\src\tests\GC\Scenarios\ServerModel\servermodel.csproj -> /numrequests:100
d:\git\runtime5\src\tests\GC\Stress\Framework\ReliabilityFramework.csproj -> -unittest
d:\git\runtime5\src\tests\JIT\Regression\JitBlue\GitHub_26491\GitHub_26491_MultipleReturns.ilproj -> MultipleReturns
d:\git\runtime5\src\tests\JIT\Stress\ABI\pinvokes_d.csproj -> --pinvokes --num-calls 1000 --no-ctrlc-summary
d:\git\runtime5\src\tests\JIT\Stress\ABI\pinvokes_do.csproj -> --pinvokes --num-calls 1000 --no-ctrlc-summary
d:\git\runtime5\src\tests\JIT\Stress\ABI\stubs_do.csproj -> --instantiatingstubs --unboxingstubs --sharedgenericunboxingstubs --num-calls 100 --max-params 5 --no-ctrlc-summary
d:\git\runtime5\src\tests\JIT\Stress\ABI\tailcalls_d.csproj -> --tailcalls --num-calls 1000 --no-ctrlc-summary
d:\git\runtime5\src\tests\JIT\Stress\ABI\tailcalls_do.csproj -> --tailcalls --num-calls 1000 --no-ctrlc-summary
d:\git\runtime5\src\tests\Loader\regressions\polyrec\Polyrec.csproj -> 4 50
d:\git\runtime5\src\tests\readytorun\r2rdump\BasicTests\R2RDumpTest.csproj -> $(CoreClrDir)tests\src\readytorun\r2rdump\files\$(TargetOS).$(TargetArchitecture).$(Configuration)\

TEST GROUPS WITH VARIANT ARGUMENTS
----------------------------------
d:\git\runtime5\src\tests\GC\Scenarios\GCSimulator\GCSimulator.csproj
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 17 -dc 20000 -sdc 8000 -lt 4 -f -dp 0.0 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 8517 -sdz 17 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.8
    -> -t 1 -tp 2 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.4
    -> -t 1 -tp 2 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.8
    -> -t 1 -tp 2 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.8 -dw 0.8
    -> -t 1 -tp 2 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.4
    -> -t 1 -tp 2 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 4 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.4
    -> -t 1 -tp 4 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.4 -dw 0.4
    -> -t 1 -tp 4 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.4
    -> -t 1 -tp 4 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 4 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.0
    -> -t 5 -tp 0 -dz 17 -sdz 17 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.8
    -> -t 5 -tp 0 -dz 17 -sdz 17 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.8
    -> -t 5 -tp 0 -dz 17 -sdz 17 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.4
    -> -t 7 -tp 0 -dz 17 -sdz 17 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 17 -dc 30000 -sdc 6000 -lt 3 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 17 -dc 20000 -sdc 6000 -lt 2 -f -dp 0.0 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 17 -dc 20000 -sdc 8000 -lt 2 -f -dp 0.0 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 17 -dc 20000 -sdc 8000 -lt 4 -f -dp 0.0 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 17 -dc 20000 -sdc 8000 -lt 2 -dp 0.0 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.0 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.4 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.0 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.4 -dw 0.8
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.4 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.4 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.4 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.4 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.8 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.8 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.8 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.8 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 17 -dc 20000 -sdc 8000 -lt 4 -f -dp 0.0 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 8517 -sdz 17 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.8 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.8
    -> -t 1 -tp 2 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.4
    -> -t 1 -tp 2 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.8
    -> -t 1 -tp 2 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.8 -dw 0.8
    -> -t 1 -tp 2 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.4
    -> -t 1 -tp 2 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 4 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.4
    -> -t 1 -tp 4 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.4
    -> -t 1 -tp 4 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.4
    -> -t 1 -tp 4 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.4 -dw 0.0
    -> -t 1 -tp 4 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.0
    -> -t 5 -tp 0 -dz 17 -sdz 17 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.8
    -> -t 5 -tp 0 -dz 17 -sdz 17 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.8
    -> -t 5 -tp 0 -dz 17 -sdz 17 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.4
    -> -t 7 -tp 0 -dz 17 -sdz 17 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -dp 0.0 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 17 -dc 30000 -sdc 6000 -lt 3 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 17 -dc 20000 -sdc 6000 -lt 2 -f -dp 0.0 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 17 -dc 20000 -sdc 8000 -lt 2 -f -dp 0.0 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 17 -dc 20000 -sdc 8000 -lt 4 -f -dp 0.0 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 7 -tp 0 -dz 17 -sdc 1024 -dc 10000 -sdz 17 -lt 2 -dp 0.1 -dw 0.0 -f
    -> -t 8 -tp 0 -dz 17 -sdc 1024 -dc 10000 -sdz 17 -lt 2 -dp 0.2 -dw 0.0 -f
    -> -t 10 -tp 0 -dz 17 -sdc 1024 -dc 10000 -sdz 17 -lt 2 -dp 0.2 -dw 0.0 -f
    -> -t 10 -tp 0 -dz 17 -sdc 1024 -dc 10000 -sdz 17 -lt 2 -dp 0.3 -dw 0.0 -f
    -> -t 8 -tp 0 -dz 17 -sdc 1024 -dc 10000 -sdz 17 -lt 2 -dp 0.3 -dw 0.1 -f
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.4 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.4 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.4 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.0 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.8 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.0 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.0 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.4 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.4 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.4 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.4 -dw 0.8
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.4 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.4 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.4 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.4 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.8 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.8 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.8 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.8 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8517 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.8 -dw 0.0
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.8 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -f -dp 0.8 -dw 0.4
    -> -t 3 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -f -dp 0.8 -dw 0.4
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 4 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 5 -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 2 -f -dp 0.0 -dw 0.0
    -> -t 1 -tp 0 -dz 17 -sdz 8500 -dc 10000 -sdc 5000 -lt 3 -f -dp 0.0 -dw 0.0

    -> -trustedexe
    -> 240
    -> 0
    -> 1
    -> 2
    -> 0
    -> 1
    -> 2
    -> 0
    -> 1
    -> 2
    -> 10000 track
    -> 10000
    -> 1 1000 50000
    -> 1 1000 50000
    -> 2048
    -> 2048
    -> 2048
    -> 2048
    -> 2048
    -> 2048
    -> 2048
    -> 2048
    -> 1000 40 191919
    -> 8 100
    -> 3 100
    -> -t 1 -tp 0 -dz 17 -sdz 17 -dc 20000 -sdc 8000 -lt 2 -dp 0.0 -dw 0.4
    -> 7 40 4 77
    -> /numrequests:100
    -> -unittest
    -> MultipleReturns
    -> --pinvokes --num-calls 1000 --no-ctrlc-summary
    -> --pinvokes --num-calls 1000 --no-ctrlc-summary
    -> --instantiatingstubs --unboxingstubs --sharedgenericunboxingstubs --num-calls 100 --max-params 5 --no-ctrlc-summary
    -> --tailcalls --num-calls 1000 --no-ctrlc-summary
    -> --tailcalls --num-calls 1000 --no-ctrlc-summary
    -> 4 50
    -> $(CoreClrDir)tests\src\readytorun\r2rdump\files\$(TargetOS).$(TargetArchitecture).$(Configuration)\
d:\git\runtime5\src\tests\baseservices\exceptions\simple\ParallelCrash.csproj
    -> 1
    -> 2
d:\git\runtime5\src\tests\JIT\Regression\JitBlue\GitHub_26491\GitHub_26491_MultipleReturns.ilproj
    -> SingleReturnSynchronized

…1750) I believe that despite its singular nature this is worth a separate review to make sure we're in agreement regarding the conversion principles. There are several things worth noting here: 1) According to the current plan we're continuously documenting in #54512 we want to remove command-line parameters from all test entrypoints. Variant tests driven by command-line parameters should be turned into multiple test cases marked with separate [Fact] attributes. 2) For ilproj tests, to facilitate local debugging, our current plan is to keep them runnable as standalone executables. This implies that ilproj tests comprising several [Fact] test entrypoints require a new entrypoint that just calls into the individual test cases. 3) The Roslyn-generated merged wrapper for the tests won't care about the "composite" main (that is for local debugging only), it will directly identify and use the individual test cases marked with [Fact] attributes. 4) In accordance with this scheme, such composite ILPROJ tests are specific in not having their entrypoint method itself marked with the [Fact] attribute. 5) Funnily enough this example nicely demonstrates the implied cleanup - the entire command-line machinery is only used for a handwritten switch to choose one of the three variants; moreover we only exercised two out of the three variants, possibly due to an authoring bug when creating the variant test, potentially caused by previous complexity of such endeavor. Thanks Tomas

trylek · 2022-04-12T09:51:44Z

By now we're approaching merge of the JIT/Methodical tests switchover (currently blocked on the GC bug #64263 taking down all r2r-extra runs on Unix) after which we'll have 3574 tests switched over or about 1/3 of the entire Pri1 test tree. Before we start a next phase merging additional test subtrees, I believe it's useful to analyze post-mortem observations from the previous phases and discuss solid design for cases we initially had to work around or express in suboptimal manner. Follows an initial list of aspects that merit additional design / cleanup.

(*) Most problems revolve around architecture-conditional ILPROJ tests (e.g. JIT/Methodical/ELEMENT_TYPE_IU/i_array_merge_TARGET_64Bit_il_d.ilproj):

(*) Putting ConditionalFactAttribute on IL code is extremely ugly as it requires pasting the binary representation of the attribute into the source code.

(*) Today the tests need to be tagged as out-of-process because R2R precompilation of the "wrong architecture" variant crashes JIT with an assertion failure. The original design was to compile all code and skip wrong architecture code at runtime using the ConditionalFact statements; we need to decide whether it still holds (and we somehow fix JIT to work with the wrong bitness code) or design something different.

(*) Marking the tests as out of process also makes them unusable for testing on those Mono platforms that don't support process creation (e.g. WASM).

(*) There are additional test project properties that are used for conditional test execution - e.g. GCStressIncompatible or UnloadabilityIncompatible. Today these also need to be marked as out-of-proc just because we have no way to express these conditions via attributes. Perhaps we would need something like a
IncompatibleWithAttribute to mark tests as conditionally dependent on execution mode. As a first cut it could have a string argument where the developer could put a semicolon-separated list of blocking conditions for the test, e.g. GCStress or Unloadability even though it may be useful to have specialized attributes for the most common cases to reduce the risk of typos in free-form string arguments.

(*) We had to substantially massage many IL source files by removing the .module directives and by putting .assembly directives in sync with the source name to achieve consistency w.r.t. handling of dependencies of the merged wrappers. The logic is quite fragile and Jeremy believes many problems of this type could be solved by autogenerating the .assembly directive into a file that would form a second input to ilasm compilation. This merits sound design as it relies on previously unused functionality (multiple inputs to ilasm).

(*) In many cases there is a pair of _il_r and _il_d project using the same source just with different symbol info settings; as the .assembly directive is naturally the same (it's the same source), we're using the trick of transforming the output assembly name by removing the _il_r / _il_d prefix; perhaps this is also solvable via the generated .assembly directive.

This is the list of issues I remember off the top of my head. I'll add more after going over the various conversations and PRs related to this task.

Thanks

Tomas

jakobbotsch · 2022-04-12T10:56:06Z

(*) Putting ConditionalFactAttribute on IL code is extremely ugly as it requires pasting the binary representation of the attribute into the source code.

There is undocumented syntax to specify attribute values without having to deal with binary blobs. I have used it in this test.

…tests Due to optimizing effort of building and running runtime tests #54512, some tests don't generate `*.sh` files by default after build. It is necessary to `export BuildAsStandalone=true` or modify the test's `.csproj` file. --- Contributes to: #90254

trylek added the area-Infrastructure-coreclr label Jun 21, 2021

dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Jun 21, 2021

trylek removed the untriaged New issue has not been triaged by the area owner label Jun 21, 2021

hoyosjs added this to the 7.0.0 milestone Jun 21, 2021

trylek mentioned this issue Aug 16, 2021

Add support for building just a subset of runtime tests #57142

Merged

trylek mentioned this issue Nov 12, 2021

Add [Fact] attributes to methodical JIT tests #61536

Merged

trylek mentioned this issue Nov 13, 2021

Add [Fact] attributes to JIT regression tests #61540

Merged

trylek mentioned this issue Nov 16, 2021

Remove XUnit reference metadata from tests I switched over last week #61691

Merged

trylek mentioned this issue Nov 17, 2021

Fix the only ilproj test that uses multiple command-line variants #61750

Merged

trylek mentioned this issue Nov 18, 2021

Fix a few special cases of projects with command line arguments #61758

Merged

trylek mentioned this issue Apr 12, 2022

Mark GCStressIncompatible and similar tests as out-of-proc #67876

Merged

trylek mentioned this issue Jul 6, 2022

Merge the remaining groups of runtime tests #71732

Open

33 tasks

agocke added this to Runtime Infra Jul 19, 2022

jeffschwMSFT modified the milestones: 7.0.0, 8.0.0 Aug 10, 2022

agocke modified the milestones: 8.0.0, 9.0.0 Sep 18, 2023

matouskozak mentioned this issue Dec 15, 2023

[mono][docs] Reflect current workflow of building individual runtime tests #96054

Merged

agocke modified the milestones: 9.0.0, 10.0.0 Aug 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CoreCLR test suite optimization proposal: support for test project grouping #54512

CoreCLR test suite optimization proposal: support for test project grouping #54512

trylek commented Jun 21, 2021 •

edited

Loading

ghost commented Jun 21, 2021

trylek commented Jun 21, 2021

trylek commented Jun 21, 2021

trylek commented Jun 21, 2021

trylek commented Jun 21, 2021

trylek commented Jun 21, 2021

jkotas commented Jun 21, 2021 •

edited

Loading

hoyosjs commented Jun 21, 2021

trylek commented Jun 21, 2021

jkoritzinsky commented Jun 21, 2021

trylek commented Jun 21, 2021

naricc commented Jun 22, 2021

BruceForstall commented Jun 22, 2021

trylek commented Jun 24, 2021

BruceForstall commented Jun 24, 2021

trylek commented Jun 25, 2021

BruceForstall commented Jun 26, 2021 •

edited

Loading

jkotas commented Jun 26, 2021

trylek commented Jun 26, 2021 •

edited

Loading

trylek commented Jun 26, 2021

trylek commented Aug 16, 2021 •

edited

Loading

fanyang-mono commented Nov 12, 2021

trylek commented Nov 12, 2021

naricc commented Nov 14, 2021 •

edited

Loading

trylek commented Nov 17, 2021

trylek commented Nov 17, 2021

trylek commented Nov 17, 2021

trylek commented Apr 12, 2022

jakobbotsch commented Apr 12, 2022

CoreCLR test suite optimization proposal: support for test project grouping #54512

CoreCLR test suite optimization proposal: support for test project grouping #54512

Comments

trylek commented Jun 21, 2021 • edited Loading

ghost commented Jun 21, 2021

trylek commented Jun 21, 2021

trylek commented Jun 21, 2021

trylek commented Jun 21, 2021

trylek commented Jun 21, 2021

trylek commented Jun 21, 2021

jkotas commented Jun 21, 2021 • edited Loading

hoyosjs commented Jun 21, 2021

trylek commented Jun 21, 2021

jkoritzinsky commented Jun 21, 2021

trylek commented Jun 21, 2021

naricc commented Jun 22, 2021

BruceForstall commented Jun 22, 2021

trylek commented Jun 24, 2021

BruceForstall commented Jun 24, 2021

trylek commented Jun 25, 2021

BruceForstall commented Jun 26, 2021 • edited Loading

jkotas commented Jun 26, 2021

trylek commented Jun 26, 2021 • edited Loading

trylek commented Jun 26, 2021

trylek commented Aug 16, 2021 • edited Loading

fanyang-mono commented Nov 12, 2021

trylek commented Nov 12, 2021

naricc commented Nov 14, 2021 • edited Loading

trylek commented Nov 17, 2021

trylek commented Nov 17, 2021

trylek commented Nov 17, 2021

trylek commented Apr 12, 2022

jakobbotsch commented Apr 12, 2022

trylek commented Jun 21, 2021 •

edited

Loading

jkotas commented Jun 21, 2021 •

edited

Loading

BruceForstall commented Jun 26, 2021 •

edited

Loading

trylek commented Jun 26, 2021 •

edited

Loading

trylek commented Aug 16, 2021 •

edited

Loading

naricc commented Nov 14, 2021 •

edited

Loading