Move Linker repo to Runtime repo

[tlakollo]

Move linker repo to runtime repo

This issue tracks progress and discussion for moving the dotnet/linker repo into dotnet/runtime repo.

Progress

• Selected the location of the repository as runtime\src\tools\linker
• Ability to transfer Linker history to runtime using the consolidation script
• Add cecil as an arcade package in order to avoid having a submodule
  • Rename package to DotNet.Cecil
• Be able to run dotnet format to apply runtime formatting to linker repo
• Create subsets (tools.linker & tools.linkertests) in order to be able to run and test the repo
• Run in CI
• Add additional label
• Rename linker to illink
• Build a package from the new location or use live bits

Follow-up work

• Tools illink should use runtime formatting #78050
• Rename linker to illink #78059
• Use runtime build infrastructure for tools.illink subset #78334
• Unify type name parser in linker once is in runtime #77868
• Re-enable illink tests which rely on checked-in binaries #78344
• Ensure illink packages flow to source build #80319
• Change github illink links #80407
• Build tools subset to generate artifacts #80406

Motivation

• Code Sharing. At this moment NativeAOT and linker share a lot of code via a straight copy. NativeAOT consumes the ILLink.Shared project which is a shared project between linker, trimming analyzer and NativeAOT. It also shares the same testing infrastructure and even shares additional files that are not included in the shared project since they are only shared between linker and NativeAOT (not for the trimming analyzer). The plan is that once we move the linker repository to runtime we can take advantage of having only one ILLink.Shared project, only one testing infrastructure, and we can improve code sharing between linker and NativeAOT, even having the same formats and analyzers applied to the codebases improve the current state of code sharing.
• Versioning history. Backward compatibility of tools like linker and NativeAOT is an ongoing issue (see PublishTrimmed should use a linker that matches the TargetFramework linker#3029) having both repositories in the runtime simplifies the solution for both of the tools.

Select the location for the linker project

So far there have been some locations that seem relevant

• src\coreclr\tools\ folder, although it would be closer to the AOT tools linker is not strictly a part of the coreclr
• Create a new folder under src\tools for tools that are agnostic to the clr. Likely the best option.
• Maybe somewhere in the mono partition?

Avoiding having a cecil submodule

Part of the linker workflow is to have a .gitmodules file that specifies a mono/cecil module, for runtime this won't fit the runtime needs. So we need to maybe ship cecil packages from mono/cecil and consume them in the runtime to later be used by linker

Linker style of code format

Currently, linker repo uses mono style of code. This makes already codesharing with NativeAOT more difficult since we cannot share code as is. Linker already transition from the mono space to the dotnet space, which leaves in the air if it should still use the mono guidelines. Once moving to the runtime repo and giving that we want to share code, the most likely solution is to just use runtime format for that repo. Note: mono directories under runtime still use the mono style of code guidelines.

Legs for execution

Need a definition on when should ci build linker and run linker tests. Most likely will remain as a subset just like currently clr.tools and clr.toolstests, and run only if changes in the linker repo are made. For source build, building and testing the linker repo seems inevitable unless we create a different leg that only runs linker.

Add an additional label

Add an additional label to runtime to identify linker issues

Keep a history of the linker changes

There are some scripts that were used to consolidate different repositories into runtime, although they are tailored to work with certain repositories we could configure it to keep the history from linker (see https://github.com/dotnet/consolidation)

Migration of github issues

Previous runtime consolidation work had the help of GitHub to migrate all issues from the different repos to the new dotnet/runtime repo, and then lock down creation of additional issues, such that the repos will no longer be used for issue management. They also inhibit the creation of PRs to the main branch, effectively making old repos an archive for read-only review of history. Plan is to do the same for linker

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[tlakollo]

cc @dotnet/linker-contrib @dotnet/ilc-contrib

[tlakollo]

cc @marek-safar @radekdoulik

[ghost]

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Move linker repo to runtime repo

This issue tracks progress and discussion for moving the dotnet/linker repo into dotnet/runtime repo.

Select the location for the linker project

So far there have some locations that seem relevant

• src\coreclr\tools\ folder, although it would be closer to the AOT tools linker is not strictly a part of the coreclr
• Create a new folder under src\tools for tools that are agnostic to the clr. Likely the best option.
• Maybe somewhere in the mono partition?

Keep a history of the linker changes

There are some scripts that were used to consolidate different repositories into runtime, although they are tailored to work with certain repositories we could configure it to keep the history from linker (see https://github.com/dotnet/consolidation)

Avoiding having a cecil submodule

Part of the linker workflow is to have a .gitmodules file that specifies a mono/cecil module, for runtime this won't fit the runtime needs. So we need to maybe ship cecil packages from mono/cecil and consume them in the runtime to later be used by linker

Linker style of code format

Currently, linker repo uses mono style of code. This makes already codesharing with NativeAOT more difficult since we cannot share code as is. Linker already transition from the mono space to the dotnet space, which leaves in the air if it should still use the mono guidelines. Once moving to the runtime repo and giving that we want to share code, the most likely solution is to just use runtime format for that repo. Note: mono directories under runtime still use the mono style of code guidelines.

Legs for execution

Need a definition on when should ci build linker and run linker tests. Most likely will remain as a subset just like currently clr.tools and clr.toolstests, and run only if changes in the linker repo are made. For source build, building and testing the linker repo seems inevitable unless we create a different leg that only runs linker.

Add an additional label

Add an additional label to runtime to identify linker issues

Author:	tlakollo
Assignees:	-
Labels:	area-Meta
Milestone:	8.0.0

[jkotas]

Create a new folder under src\tools

That looks better to me.

we need to maybe ship cecil packages from mono/cecil and consume them in the runtime to later be used by linker

We need to make sure that this is compatible with our servicing policies and source build. If we were to keep cecil in separate repo, I expect that we would need to move it to dotnet/cecil at least.

[vitek-karas]

Moving Cecil to runtime repo is problematic. So far we've used mono/cecil as only a way to ship predictable bits and have a way to quickly fix issues without waiting for upstream merge. But we always send all our changes upstream and most of the time mono/cecil and upstream are identical. If we were to move cecil into runtime repo, it would have to be a mirror, which seems impractical.

[marek-safar]

We need to make sure that this is compatible with our servicing policies and source build. If we were to keep cecil in separate repo, I expect that we would need to move it to dotnet/cecil at least.

I don't think mono/ org should be a problem for source build (we have or had sources used by source build there before) but I have no objection to creating another fork under dotnet/ org. We just need to be careful about versioning of the internal cecil package/dlls.

[ghost]

Tagging subscribers to this area: @agocke, @sbomer, @vitek-karas
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Move linker repo to runtime repo

This issue tracks progress and discussion for moving the dotnet/linker repo into dotnet/runtime repo.

Progress

• Selected the location of the repository as runtime\src\tools\linker
• Ability to transfer Linker history to runtime using the consolidation script
• Add cecil as an arcade package in order to avoid having a submodule
  • Rename package to DotNet.Cecil
• Be able to run dotnet format to apply runtime formatting to linker repo
• Create subsets (tools.linker & tools.linkertests) in order to be able to run and test the repo
• Run in CI
• Add additional label
• Rename linker to illink
• Build a package from the new location or use live bits

Follow-up work

• Tools illink should use runtime formatting #78050
• Rename linker to illink #78059
• Use runtime build infrastructure for tools.illink subset #78334
• Unify type name parser in linker once is in runtime #77868
• Re-enable illink tests which rely on checked-in binaries #78344
• Ensure illink packages flow to source build #80319
• Change github illink links #80407
• Build tools subset to generate artifacts #80406

Motivation

• Code Sharing. At this moment NativeAOT and linker share a lot of code via a straight copy. NativeAOT consumes the ILLink.Shared project which is a shared project between linker, trimming analyzer and NativeAOT. It also shares the same testing infrastructure and even shares additional files that are not included in the shared project since they are only shared between linker and NativeAOT (not for the trimming analyzer). The plan is that once we move the linker repository to runtime we can take advantage of having only one ILLink.Shared project, only one testing infrastructure, and we can improve code sharing between linker and NativeAOT, even having the same formats and analyzers applied to the codebases improve the current state of code sharing.
• Versioning history. Backward compatibility of tools like linker and NativeAOT is an ongoing issue (see PublishTrimmed should use a linker that matches the TargetFramework linker#3029) having both repositories in the runtime simplifies the solution for both of the tools.

Select the location for the linker project

So far there have been some locations that seem relevant

• src\coreclr\tools\ folder, although it would be closer to the AOT tools linker is not strictly a part of the coreclr
• Create a new folder under src\tools for tools that are agnostic to the clr. Likely the best option.
• Maybe somewhere in the mono partition?

Avoiding having a cecil submodule

Part of the linker workflow is to have a .gitmodules file that specifies a mono/cecil module, for runtime this won't fit the runtime needs. So we need to maybe ship cecil packages from mono/cecil and consume them in the runtime to later be used by linker

Linker style of code format

Currently, linker repo uses mono style of code. This makes already codesharing with NativeAOT more difficult since we cannot share code as is. Linker already transition from the mono space to the dotnet space, which leaves in the air if it should still use the mono guidelines. Once moving to the runtime repo and giving that we want to share code, the most likely solution is to just use runtime format for that repo. Note: mono directories under runtime still use the mono style of code guidelines.

Legs for execution

Need a definition on when should ci build linker and run linker tests. Most likely will remain as a subset just like currently clr.tools and clr.toolstests, and run only if changes in the linker repo are made. For source build, building and testing the linker repo seems inevitable unless we create a different leg that only runs linker.

Add an additional label

Add an additional label to runtime to identify linker issues

Keep a history of the linker changes

There are some scripts that were used to consolidate different repositories into runtime, although they are tailored to work with certain repositories we could configure it to keep the history from linker (see https://github.com/dotnet/consolidation)

Migration of github issues

Previous runtime consolidation work had the help of GitHub to migrate all issues from the different repos to the new dotnet/runtime repo, and then lock down creation of additional issues, such that the repos will no longer be used for issue management. They also inhibit the creation of PRs to the main branch, effectively making old repos an archive for read-only review of history. Plan is to do the same for linker

Author:	tlakollo
Assignees:	tlakollo
Labels:	area-Tools-ILLink
Milestone:	8.0.0

[agocke]

Closing as complete