Native AOT Testing Improvements

[agocke]

• Increase NativeAOT testing when NativeAOT is changed #72419
• Expand CoreCLR testing #67743
  • Consolidated tests have IsMergedTestRunnerAssembly
• Expand libraries testing #67741
  • Make xunit assert aot-compatible
  • Add AOT-compatible XUnit runner #91069
• ILLink+NativeAOT testing integration #70395

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

Issue Details

---

• Increase NativeAOT testing when NativeAOT is changed #72419
• Expand CoreCLR testing #67743
• Expand libraries testing #67741

Author:	agocke
Assignees:	-
Labels:	area-NativeAOT-coreclr
Milestone:	-

[kant2002]

I would like to ask opinions here first. Currently other teams like EF Core and ASP.NET seems to be plan to work on NativeAOT. From what I know they are using xUnit. Both does not have infra for NativeAOT yet.

I would like to ask what's status of testing of NativeAOT using xUnit, for example. If I want contribute to EF Core + NativeAOT for example, I obviously may try to create sample apps and work, but that seems to be inefficient.

It would be good to start discussing on guidance for developers who would like to tests their libraries.

[MichalStrehovsky]

In general, the experience we're shooting for with trimming and AOT is that of compile time warnings. If there are no warnings, the code should work and there's no need to do separate functional testing with trimming or AOT. The behaviors should be identical, modulo bugs in the runtime/compiler. It's the same for e.g. PublishReadyToRun. One doesn't specifically run unit tests with PublishReadyToRun either.

This only breaks down if one has a trimming/AOT incompatible dependency that the analyzer doesn't see. One could say that the testing would make sense to cover those. But the act of placing the code into a unit test makes trimming/AOT work differently (because the surrounding code is different). It may well be that the code will work in a test, but fail in the app, or vice versa. Using unit testing as a way to verify trimming/AOT compatibility doesn't work.

That said, we have #78342 tracking a trimming/AOT friendly xUnit runner for the purposes of the runtime repo. The runtime repo does run unit tests with NativeAOT, but the purpose is to catch actual NativeAOT runtime or compiler bugs. We're already using xUnit with its reflection based runner to run 200+ libraries test assemblies in this repo. But the experience is subpar and we have many rd.xml files to make things work in the runner.

[agocke]

Closing this out as done for 8.0. Further testing improvements will be brought through in 9.0