Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Android] Introduce NetTraceToMibcConverter task & streamline testing targets #72394

Merged
merged 16 commits into from
Aug 4, 2022

Conversation

steveisok
Copy link
Member

@steveisok steveisok commented Jul 18, 2022

NetTraceToMibcConverter

  • Used in profiled AOT scenarios where a .nettrace file is given as input and is converted to a .mibc file that can be fed into the AOT compiler. This previously was in the AotCompiler task, but for clarity purposes is now separated out.

Streamline Android testing targets

  • The testing targets function the same, but are now structured similarly to iOS and Wasm.

  • Introduced new testing properties to support profiled AOT:

    NetTraceFilePath - The path to a .nettrace file that will be converted into a .mibc file and fed into the aot compiler

    RuntimeComponents - The list of native components to include in the test app build (diagnostics_tracing)

    DiagnosticsPorts - The ip address:port where the runtime will listen when running diagnostic tooling

    DiagnosticStartupMode - The mode the runtime will use at startup for diagnostic scenarios. Suspend will halt the app very early and wait, while nosuspend will wait for a connection, but not halt the runtime

@ghost
Copy link

ghost commented Jul 18, 2022

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

NetTraceToMibcConverter

  • Used in profiled AOT scenarios where a .nettrace file is given as input and is converted to a .mibc file that can be fed into the AOT compiler. This previously was in the AotCompiler task, but for clarity purposes is now separated out.

Streamline Android testing targets

  • The testing targets function the same, but are now structured similarly to iOS and Wasm.

  • Introduced new testing properties to support profiled AOT:

    RunProfileAOT - true/false to control the mode the aot compiler is set in.

    NetTracePath - The path to a .nettrace file that will be converted into a .mibc file and fed into the aot compiler

    RuntimeComponents - The list of native components to include in the test app build (diagnostics_tracing)

    DiagnosticsPorts - The ip address:port where the runtime will listen when running diagnostic tooling

    DiagnosticStartup - The mode the runtime will use at startup for diagnostic scenarios. Suspend will halt the app very early and wait, while nosuspend will wait for a connection, but not halt the runtime

Author: steveisok
Assignees: -
Labels:

area-Infrastructure-mono

Milestone: -

… targets

NetTraceToMibcConverter

- Used in profiled AOT scenarios where a .nettrace file is given as input and is converted to a .mibc file that can be fed into the AOT compiler. This previously was in the AotCompiler task, but for clarity purposes is now separated out.

Streamline Android testing targets

- The testing targets function the same, but are now structured similarly to iOS and Wasm.

- Introduced new testing properties to support profiled AOT:

RunProfileAOT - true/false to control the mode the aot compiler is set in.

NetTracePath - The path to a .nettrace file that will be converted into a .mibc file and fed into the aot compiler

RuntimeComponents - The list of native components to include in the test app build (diagnostics_tracing)

DiagnosticsPorts - The ip address:port where the runtime will listen when running diagnostic tooling

DiagnosticStartup - The mode the runtime will use at startup for diagnostic scenarios. Suspend will halt the app very early and wait, while nosuspend will wait for a connection, but not halt the runtime
src/mono/msbuild/android/build/AndroidApp.InTree.props Outdated Show resolved Hide resolved
eng/testing/tests.android.targets Outdated Show resolved Hide resolved
eng/testing/tests.android.targets Show resolved Hide resolved
Comment on lines +58 to +61
<_PublishAssemblies Include="$(PublishDir)\**\*.dll" Exclude="$(PublishDir)\**\*.resources.dll" />
<_SatelliteAssemblies Include="$(PublishDir)\**\*.resources.dll" />

<AndroidAssembliesToBundle Include="@(_PublishAssemblies)">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit since publish assesmblies naming implis all assemblies, and it doesnt seem like its being used elsewhere, so is there a benefit to having it in a separate itemgroup rather than explicitly defining AndroidAssembliesToBundle?

Suggested change
<_PublishAssemblies Include="$(PublishDir)\**\*.dll" Exclude="$(PublishDir)\**\*.resources.dll" />
<_SatelliteAssemblies Include="$(PublishDir)\**\*.resources.dll" />
<AndroidAssembliesToBundle Include="@(_PublishAssemblies)">
<_SatelliteAssemblies Include="$(PublishDir)\**\*.resources.dll" />
<AndroidAssembliesToBundle Include="$(PublishDir)\**\*.dll" Exclude="@(_SatelliteAssemblies)">

Is there a particular reason we are keeping track of SatelliteAssemblies in an itemgroup?
If its not going to be used, maybe just
<AndroidAssembliesToBundle Include="$(PublishDir)\**\*.dll" Exclude="$(PublishDir)\**\*.resources.dll" />

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could be wrong, but I think the paths need expanded in an item first before you can condition them like we do in AndroidAssembliesToBundle

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah do you mean the metadata getting set properly?

I think it still works properly before getting expanded. With a console project with a.txt b.txt c.txt in the same directory

<ItemGroup>
   <TestItemGroup Include="$(MSBuildThisFileDirector)*.txt">
     <_InternalForceInterpret Condition="'%(FileName)%(Extension)' != 'b.txt'">true</_InternalForceInterpret>
   </TestItemGroup>
 </ItemGroup>

evaluates to
Screen Shot 2022-07-19 at 16 51 29

So it looks like the metadata is still set properly

eng/testing/tests.android.targets Show resolved Hide resolved
src/mono/msbuild/android/build/AndroidApp.targets Outdated Show resolved Hide resolved
src/mono/msbuild/android/build/AndroidApp.targets Outdated Show resolved Hide resolved
src/mono/msbuild/android/build/AndroidApp.targets Outdated Show resolved Hide resolved
/// NetTrace file to use when invoking dotnet-pgo for
/// </summary>
[Required]
public string NetTracePath { get; set; } = "";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we extend NetTracePath to be multiple NetTrace files?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does dotnet-pgo support this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No I dont think so, but it does support merging multiple .mibc files. I'm not sure if its a big desire to input multiple .nettrace files, but if it is a use case we could generate multiple .mibc and merge it, or perhaps dotnet-pgo could be extended to convert multiple .nettrace files.

@steveisok
Copy link
Member Author

/azp run runtime-extra-platforms

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

Copy link
Member

@mdh1418 mdh1418 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it looks good if the build order is good https://github.com/dotnet/runtime/pull/72394/files#r924949694,
https://github.com/dotnet/runtime/pull/72394/files#r924955076
my mistake, I got the BeforeTargets and AfterTargets mixed up again, updating the comments.
#72394 (comment)

Can the HelloWorld desktop sample be updated as well? https://github.com/dotnet/runtime/blob/main/src/mono/sample/HelloWorld/HelloWorld.csproj

src/mono/msbuild/android/build/AndroidApp.targets Outdated Show resolved Hide resolved
@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

Copy link
Member

@mdh1418 mdh1418 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we update HelloWorld.csproj to use the MonoAOTCompiler.cs and NetTraceToMibcConverter.cs changes to ensure things are working (in terms of correct parameter name etc).

If everything is functioning as expected, it looks good to me!

@steveisok
Copy link
Member Author

Can we update HelloWorld.csproj to use the MonoAOTCompiler.cs and NetTraceToMibcConverter.cs changes to ensure things are working (in terms of correct parameter name etc).

Yes, I'd rather do it in a follow up.

Copy link
Member

@akoeplinger akoeplinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some comments but looks good overall


protected override string GenerateFullPathToTool()
{
return ToolPath;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@steveisok ok but we could at least make ToolExe default to dotnet-pgo so you don't need to explicitly pass that in. That reminds me, do we need to append .exe on Windows?

<PublishTestAsSelfContainedDependsOn>Publish</PublishTestAsSelfContainedDependsOn>
</PropertyGroup>

<PropertyGroup>
<RunAOTCompilation Condition="'$(TargetOS)' == 'iOS' or '$(TargetOS)' == 'tvOS'">true</RunAOTCompilation>
</PropertyGroup>

<PropertyGroup>
<DotnetPgoToolDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsBinDir)', 'coreclr', '$(TargetOS).$(TargetArchitecture).$(Configuration)', 'dotnet-pgo'))</DotnetPgoToolDir>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this could be unified with ILAsmToolPath which references the same path in Directory.Build.props

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since ILAsmToolPath is only active when it's source build or s390x, I made a CoreCLRToolPath property and used that instead.

eng/testing/tests.mobile.targets Outdated Show resolved Hide resolved
@steveisok
Copy link
Member Author

The wasm failures are known and unrelated.

@steveisok steveisok merged commit 209c040 into dotnet:main Aug 4, 2022
@steveisok steveisok deleted the nettrace-mibc-converter branch August 4, 2022 17:02
radical added a commit to radical/runtime that referenced this pull request Aug 5, 2022
.. builds.

dotnet#72394 disabled the cache always,
even when it was intended to be used. This reverses that change.

Fixes dotnet#73419
radical added a commit that referenced this pull request Aug 5, 2022
…al (#73427)

.. builds.

#72394 disabled the cache always,
even when it was intended to be used. This reverses that change.

Fixes #73419
@uweigand
Copy link
Contributor

Hi @steveisok, it looks like this broke native build on s390x:

/home/uweigand/dotnet-net7-preview5/sdk/7.0.100-preview.5.22307.18/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.PackTool.targets(122,5): error MSB4018: The "GenerateShims" task failed unexpectedly. [/home/uweigand/runtime/src/coreclr/tools/dotnet-pgo/dotnet-pgo.csproj]
/home/uweigand/dotnet-net7-preview5/sdk/7.0.100-preview.5.22307.18/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.PackTool.targets(122,5): error MSB4018: Microsoft.NET.HostModel.AppHost.AppHostMachOFormatException: Exception of type 'Microsoft.NET.HostModel.AppHost.AppHostMachOFormatException' was thrown. [/home/uweigand/runtime/src/coreclr/tools/dotnet-pgo/dotnet-pgo.csproj]
/home/uweigand/dotnet-net7-preview5/sdk/7.0.100-preview.5.22307.18/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.PackTool.targets(122,5): error MSB4018:    at Microsoft.NET.HostModel.AppHost.MachOUtils.Verify(Boolean condition, MachOFormatError error) [/home/uweigand/runtime/src/coreclr/tools/dotnet-pgo/dotnet-pgo.csproj]
/home/uweigand/dotnet-net7-preview5/sdk/7.0.100-preview.5.22307.18/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.PackTool.targets(122,5): error MSB4018:    at Microsoft.NET.HostModel.AppHost.MachOUtils.RemoveSignature(FileStream stream) [/home/uweigand/runtime/src/coreclr/tools/dotnet-pgo/dotnet-pgo.csproj]
/home/uweigand/dotnet-net7-preview5/sdk/7.0.100-preview.5.22307.18/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.PackTool.targets(122,5): error MSB4018:    at Microsoft.NET.HostModel.AppHost.HostWriter.<>c__DisplayClass2_0.<CreateAppHost>b__2() [/home/uweigand/runtime/src/coreclr/tools/dotnet-pgo/dotnet-pgo.csproj]
/home/uweigand/dotnet-net7-preview5/sdk/7.0.100-preview.5.22307.18/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.PackTool.targets(122,5): error MSB4018:    at Microsoft.NET.HostModel.RetryUtil.RetryOnIOError(Action func) [/home/uweigand/runtime/src/coreclr/tools/dotnet-pgo/dotnet-pgo.csproj]
/home/uweigand/dotnet-net7-preview5/sdk/7.0.100-preview.5.22307.18/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.PackTool.targets(122,5): error MSB4018:    at Microsoft.NET.HostModel.AppHost.HostWriter.CreateAppHost(String appHostSourceFilePath, String appHostDestinationFilePath, String appBinaryFilePath, Boolean windowsGraphicalUserInterface, String assemblyToCopyResorcesFrom, Boolean enableMacOSCodeSign) [/home/uweigand/runtime/src/coreclr/tools/dotnet-pgo/dotnet-pgo.csproj]
/home/uweigand/dotnet-net7-preview5/sdk/7.0.100-preview.5.22307.18/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.PackTool.targets(122,5): error MSB4018:    at Microsoft.NET.Build.Tasks.GenerateShims.ExecuteCore() in /home/uweigand/dotnet-s390x-net7preview5/sdk/src/Tasks/Microsoft.NET.Build.Tasks/GenerateShims.cs:line 134 [/home/uweigand/runtime/src/coreclr/tools/dotnet-pgo/dotnet-pgo.csproj]
/home/uweigand/dotnet-net7-preview5/sdk/7.0.100-preview.5.22307.18/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.PackTool.targets(122,5): error MSB4018:    at Microsoft.NET.Build.Tasks.TaskBase.Execute() in /home/uweigand/dotnet-s390x-net7preview5/sdk/src/Tasks/Common/TaskBase.cs:line 47 [/home/uweigand/runtime/src/coreclr/tools/dotnet-pgo/dotnet-pgo.csproj]
/home/uweigand/dotnet-net7-preview5/sdk/7.0.100-preview.5.22307.18/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.PackTool.targets(122,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() in /home/uweigand/dotnet-s390x-net7preview5/msbuild/src/Build/BackEnd/TaskExecutionHost/TaskExecutionHost.cs:line 564 [/home/uweigand/runtime/src/coreclr/tools/dotnet-pgo/dotnet-pgo.csproj]
/home/uweigand/dotnet-net7-preview5/sdk/7.0.100-preview.5.22307.18/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.PackTool.targets(122,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) in /home/uweigand/dotnet-s390x-net7preview5/msbuild/src/Build/BackEnd/Components/RequestBuilder/TaskBuilder.cs:line 825 [/home/uweigand/runtime/src/coreclr/tools/dotnet-pgo/dotnet-pgo.csproj]

The problem seems to be that the new mono.tools subset now builds dotnet-pgo.csproj, which tries to create an osx-x64 shim apphost:

    <PackAsToolShimRuntimeIdentifiers>win-x64;win-x86;win-arm;osx-x64</PackAsToolShimRuntimeIdentifiers>

This in turn pulls in code in src/installer/managed/Microsoft.NET.HostModel/AppHost/MachOUtils.cs to manipulate MachO binaries, but that whole file unfortunately fundamentally seems to assume little-endian byte order, as it just mmaps the binary and then overlays structures over the mmap.

So the question is, what is the best way to fix this? Of course, we could try to rewrite MachOUtils.cs to be endian-independent. But I'm wondering if we even need to build this on s390x? Do you think it would make sense to skip building dotnet-pgo completely (or at least building the osx-x64 shim), either whenever PrimaryRuntimeFlavor is Mono or specificially just on s390x due to the endian issue?

Are the tools in dotnet-pgo even useful at all on a system where we're not supporting CoreCLR at all?

@steveisok
Copy link
Member Author

@akoeplinger @directhex can you help Uli with this?

@ghost ghost locked as resolved and limited conversation to collaborators Sep 9, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants