Support reference assemblies in build flows #1986

rainersigwald · 2017-04-17T19:32:36Z

rainersigwald · 2017-04-17T19:59:33Z

Implementation option: find ref assembly by filename transformation convention.

For example: after RAR and before CoreCompile, sub @(ReferencePath->'%(RootDir)%(Directory)\ref\%(Filename)%(Extension)') into the ReferencePath item if it exists.

➕ minimal MSBuild dataflow changes
➖ convention based (the compiler option allows arbitrary naming of the ref asm)

rainersigwald · 2017-04-17T20:38:48Z

Implementation option: pass ref assembly path as metadata on the primary compiler output

For example: after RAR and before CoreCompile, sub @(ReferencePath->'%(RootDir)%(Directory)\ref\%(Filename)%(Extension)') into the ReferencePath item if it exists.

➕ allows user flexibility in path conventions
➖ requires passing data though the convoluted p2p + RAR process
➖ Design time p2p resolution doesn't return an item but just the output path (currently)

rainersigwald · 2017-04-25T22:08:11Z

Ok, I've gotten a proof-of-concept put together based on the just-look-for-a-filename-pattern strategy, and it looks fairly reasonable.

Very little of this is production-ready, but it does work as hoped for: a whitespace change in src\Compilers\Core\Portable\AssemblyUtilities.cs causes recompilation for CodeAnalysis.csproj but doesn't propagate the ref assembly:

15:04:27.747     7>CopyFilesToOutputDirectory: (TargetId:67)
                     Reference assembly "S:\roslyn\Binaries\Debug\Dlls\CodeAnalysis\Microsoft.CodeAnalysis.interface.dll" already has latest information. Leaving it untouched. (TaskId:33)

diff --git a/MSBuild/15.0/Bin/MSBuild.exe.config b/MSBuild/15.0/Bin/MSBuild.exe.config
index eafd444..bb0f31a 100644
--- a/MSBuild/15.0/Bin/MSBuild.exe.config
+++ b/MSBuild/15.0/Bin/MSBuild.exe.config
@@ -46,6 +46,15 @@
           <assemblyIdentity name="XamlBuildTask" culture="neutral" publicKeyToken="31bf3856ad364e35" />
           <bindingRedirect oldVersion="4.0.0.0" newVersion="15.0.0.0" />
         </dependentAssembly>
+        <dependentAssembly>
+          <assemblyIdentity name="System.Collections.Immutable" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
+          <bindingRedirect oldVersion="0.0.0.0-1.2.1.0" newVersion="1.2.1.0" />
+        </dependentAssembly>
+        <dependentAssembly>
+          <assemblyIdentity name="System.IO.FileSystem" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
+          <bindingRedirect oldVersion="0.0.0.0-4.0.2.0" newVersion="4.0.2.0" />
+        </dependentAssembly>
+
 
         <!-- Workaround for crash in C++ CodeAnalysis scenarios due to https://github.com/Microsoft/msbuild/issues/1675 -->
         <dependentAssembly>
diff --git a/MSBuild/15.0/Bin/Microsoft.Common.CurrentVersion.targets b/MSBuild/15.0/Bin/Microsoft.Common.CurrentVersion.targets
index 8aa426d..6f58d49 100644
--- a/MSBuild/15.0/Bin/Microsoft.Common.CurrentVersion.targets
+++ b/MSBuild/15.0/Bin/Microsoft.Common.CurrentVersion.targets
@@ -347,6 +347,7 @@ Copyright (C) Microsoft Corporation. All rights reserved.
   </PropertyGroup>
   <ItemGroup>
     <IntermediateAssembly Include="$(IntermediateOutputPath)$(TargetName)$(TargetExt)"/>
+    <IntermediateRefAssembly Include="$(IntermediateOutputPath)$(TargetName).interface$(TargetExt)"/>
     <FinalDocFile Include="@(DocFileItem->'$(OutDir)%(Filename)%(Extension)')"/>
   </ItemGroup>
 
@@ -4003,6 +4004,17 @@ Copyright (C) Microsoft Corporation. All rights reserved.
 
     </Copy>
 
+    <!-- Copy the build product (.dll or .exe). -->
+    <CopyRefAssembly
+        SourcePath="@(IntermediateRefAssembly)"
+        DestinationPath="$(OutDir)@(IntermediateRefAssembly->'%(Filename)%(Extension)')"
+        Condition="'$(CopyBuildOutputToOutputDirectory)' == 'true' and '$(SkipCopyBuildProduct)' != 'true' and Exists('@(IntermediateRefAssembly)')"
+            >
+
+      <Output TaskParameter="DestinationPath" ItemName="ReferenceAssembly"/>
+
+    </CopyRefAssembly>
+
     <Message Importance="High" Text="$(MSBuildProjectName) -&gt; @(MainAssembly->'%(FullPath)')" Condition="'$(CopyBuildOutputToOutputDirectory)' == 'true' and '$(SkipCopyBuildProduct)'!='true'" />
 
     <!-- Copy the additional modules. -->
diff --git a/MSBuild/15.0/Bin/Microsoft.Common.tasks b/MSBuild/15.0/Bin/Microsoft.Common.tasks
index d284df7..4c7fbee 100644
--- a/MSBuild/15.0/Bin/Microsoft.Common.tasks
+++ b/MSBuild/15.0/Bin/Microsoft.Common.tasks
@@ -172,5 +172,6 @@
     <!-- Roslyn tasks are now in an assembly owned and shipped by Roslyn -->
     <UsingTask TaskName="Microsoft.CodeAnalysis.BuildTasks.Csc"                       AssemblyFile="$(RoslynTargetsPath)\Microsoft.Build.Tasks.CodeAnalysis.dll" Condition="'$(MSBuildAssemblyVersion)' != ''" />
     <UsingTask TaskName="Microsoft.CodeAnalysis.BuildTasks.Vbc"                       AssemblyFile="$(RoslynTargetsPath)\Microsoft.Build.Tasks.CodeAnalysis.dll" Condition="'$(MSBuildAssemblyVersion)' != ''" />
+    <UsingTask TaskName="Microsoft.CodeAnalysis.BuildTasks.CopyRefAssembly"           AssemblyFile="$(RoslynTargetsPath)\Microsoft.Build.Tasks.CodeAnalysis.dll" Condition="'$(MSBuildAssemblyVersion)' != ''" />
 </Project>
 
diff --git a/MSBuild/15.0/Bin/Roslyn/Microsoft.CSharp.Core.targets b/MSBuild/15.0/Bin/Roslyn/Microsoft.CSharp.Core.targets
index 5612c75..020adbb 100644
--- a/MSBuild/15.0/Bin/Roslyn/Microsoft.CSharp.Core.targets
+++ b/MSBuild/15.0/Bin/Roslyn/Microsoft.CSharp.Core.targets
@@ -1,13 +1,20 @@
 ∩╗┐<?xml version="1.0" encoding="utf-8"?>
 <!-- Copyright (c)  Microsoft.  All Rights Reserved.  Licensed under the Apache License, Version 2.0.  See License.txt in the project root for license information. -->
 <Project ToolsVersion="14.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <Target Name="MungeRefs" BeforeTargets="CoreCompile">
+    <!--<Warning Text="RP: @(ReferencePath) %(RootDir)%(Directory)%(Filename).interface%(Extension)" />-->
+    <ItemGroup>
+      <TrueReferences Include="@(ReferencePath)" Condition="!Exists('%(RootDir)%(Directory)%(Filename).interface%(Extension)')" />
+      <TrueReferences Include="@(ReferencePath->'%(RootDir)%(Directory)%(Filename).interface%(Extension)')" Condition="Exists('%(RootDir)%(Directory)%(Filename).interface%(Extension)')" />
+    </ItemGroup>
+  </Target>
   <Target Name="CoreCompile"
           Inputs="$(MSBuildAllProjects);
                   @(Compile);
                   @(_CoreCompileResourceInputs);
                   $(ApplicationIcon);
                   $(AssemblyOriginatorKeyFile);
-                  @(ReferencePath);
+                  @(TrueReferences);
                   @(CompiledLicenseFile);
                   @(LinkResource);
                   @(EmbeddedDocumentation);
@@ -117,7 +124,7 @@
          Prefer32Bit="$(Prefer32Bit)"
          PreferredUILang="$(PreferredUILang)"
          ProvideCommandLineArgs="$(ProvideCommandLineArgs)"
-         References="@(ReferencePath)"
+         References="@(TrueReferences)"
          ReportAnalyzer="$(ReportAnalyzer)"
          Resources="@(_CoreCompileResourceInputs);@(CompiledLicenseFile)"
          ResponseFiles="$(CompilerResponseFile)"

Explanations:

Binding redirects: the Roslyn task assembly now has (transitive) references to these two assemblies that don't match the versions delivered in MSBuild or Roslyn. The plan is to move the task assembly to its own folder with a possibly-different closure. MSBuild might still want to set these binding redirects to match VS as part of Deliver CoreFx façades next to MSBuild.exe #1542.
Ref file pattern: I got lazy and did this instead of mkdir ref/. It's likely a requirement to keep the filename the same.
Extra copy task does the ref magic incremental copy. Initially I was thinking of telling the compiler to emit to a temporary location and doing the magic copy from there to obj, but now I think it's sufficient to always update obj and only update bin when necessary, because ProjectReferences use the output from bin.
The target here probably belongs in common rather than C# targets. I did notice it not working for VB in my testing.
TrueReferences and MungeRefs are terrible, no-good, very-bad names.

rainersigwald · 2017-04-26T21:32:30Z

Design points to be settled:

We talked about making this "default when Deterministic is turned on". Where should that be expressed? Currently common targets don't know about Deterministic. Set a property in CSharp.Core.targets that's observed in common, or just bake it into common?
- 💬 I propose just baking it into Common. It's already a mess and this doesn't make it appreciably worse.
What's the convention for paths? Based on conversation with @jaredpar, it's likely useful to make sure that the reference assembly's filename matches the implementation assembly's.
- 💬 I propose what we'd talked about initially $(OutDir)\ref\$(TargetFileName).
Should the reference assembly be passed along explicitly (via metadata) or inferred by probing for an assembly at the ref-assembly-relative-path convention?
- 💬 I propose passing it along via metadata. GetTargetPath already returned a metadata-decorated item, and it seems low-risk to do the same from Build, giving us the freedom to have arbitrary paths to the reference assembly (if user-desired) and "push" the knowledge of the ref asm from the producing project instead of inferring it from the consuming side.
Should the Csc task consume metadata on the existing @(ReferencePath), a permanently changed ReferencePath, a temporarily changed ReferencePath, or a different item?
- Tasks consume metadata:
  - ➕ Scoped impact
  - ➖ Incrementality on CoreCompile becomes hard: the "real" inputs are either the path in Identity or the path in ReferenceAssembly of the references.
- Permanently changed ReferencePath:
  - ➖ Unclear what targets besides CoreCompile consume this (including 3rd-party extensions).
- Temporarily changed ReferencePath:
  - ➕ Scoped impact
  - ➖ No other target could opt into using the ref assemblies
  - ➖ Two extra targets would be run to replace the list and to restore it.
- New item:
  - ➕ Scoped impact
  - ➕ Other targets could opt in to ref-assembly-only list
  - ➖ Targets would have to opt in
  - ➖ Microsoft.CSharp.Core.targets would need a compat shim to be able to run with older common targets.
- 💬 I propose using a new item, with a compat shim in C# core.
Is there a better name than ReferenceAssembly for the MSBuildification of these concepts? ReferenceAssembly is already used to find full-framework reference assemblies, for example in GetReferenceAssemblyPaths.
- 💬 It's confusing but the concepts are very similar. I think I prefer just using ReferenceAssembly.

@jaredpar @jcouv @Microsoft/msbuild-maintainers: does that sound good? Anything I've forgotten?

current implementation

diff --git a/MSBuild/15.0/Bin/MSBuild.exe.config b/MSBuild/15.0/Bin/MSBuild.exe.config
index eafd444..bb0f31a 100644
--- a/MSBuild/15.0/Bin/MSBuild.exe.config
+++ b/MSBuild/15.0/Bin/MSBuild.exe.config
@@ -46,6 +46,15 @@
           <assemblyIdentity name="XamlBuildTask" culture="neutral" publicKeyToken="31bf3856ad364e35" />
           <bindingRedirect oldVersion="4.0.0.0" newVersion="15.0.0.0" />
         </dependentAssembly>
+        <dependentAssembly>
+          <assemblyIdentity name="System.Collections.Immutable" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
+          <bindingRedirect oldVersion="0.0.0.0-1.2.1.0" newVersion="1.2.1.0" />
+        </dependentAssembly>
+        <dependentAssembly>
+          <assemblyIdentity name="System.IO.FileSystem" publicKeyToken="b03f5f7f11d50a3a" culture="neutral"/>
+          <bindingRedirect oldVersion="0.0.0.0-4.0.2.0" newVersion="4.0.2.0" />
+        </dependentAssembly>
+
 
         <!-- Workaround for crash in C++ CodeAnalysis scenarios due to https://github.com/Microsoft/msbuild/issues/1675 -->
         <dependentAssembly>
diff --git a/MSBuild/15.0/Bin/Microsoft.Common.CurrentVersion.targets b/MSBuild/15.0/Bin/Microsoft.Common.CurrentVersion.targets
index 8aa426d..4881519 100644
--- a/MSBuild/15.0/Bin/Microsoft.Common.CurrentVersion.targets
+++ b/MSBuild/15.0/Bin/Microsoft.Common.CurrentVersion.targets
@@ -281,6 +281,8 @@ Copyright (C) Microsoft Corporation. All rights reserved.
     <!-- Example, c:\MyProjects\MyProject\bin\debug\MyAssembly.dll -->
     <TargetPath Condition=" '$(TargetPath)' == '' ">$(TargetDir)$(TargetFileName)</TargetPath>
 
+    <TargetRefPath Condition=" '$(TargetRefPath)' == '' and '$(Deterministic)' == 'true' ">$([System.IO.Path]::Combine(`$([System.IO.Path]::GetDirectoryName($([System.IO.Path]::GetFullPath(`$(TargetPath)`))))`, 'ref', `$(TargetFileName)`))</TargetRefPath>
+
     <!-- Example, c:\MyProjects\MyProject\ -->
     <ProjectDir Condition=" '$(ProjectDir)' == '' ">$(MSBuildProjectDirectory)\</ProjectDir>
 
@@ -350,6 +352,13 @@ Copyright (C) Microsoft Corporation. All rights reserved.
     <FinalDocFile Include="@(DocFileItem->'$(OutDir)%(Filename)%(Extension)')"/>
   </ItemGroup>
 
+  <ItemGroup Condition="'$(Deterministic)' == 'true'">
+    <!-- TODO: should this be configurable? Default path obeys conventions. -->
+    <IntermediateRefAssembly Include="$(IntermediateOutputPath)ref\$(TargetName)$(TargetExt)" Condition="'@(IntermediateRefAssembly)' == ''" />
+    <CreateDirectory Include="@(IntermediateRefAssembly->'%(RootDir)%(Directory)')" />
+    <CreateDirectory Include="$(OutDir)ref" />
+  </ItemGroup>
+
   <ItemGroup Condition="'$(_DebugSymbolsProduced)' == 'true'">
     <_DebugSymbolsIntermediatePath Include="$(IntermediateOutputPath)$(TargetName).compile.pdb" Condition="'$(OutputType)' == 'winmdobj' and '@(_DebugSymbolsIntermediatePath)' == ''"/>
     <_DebugSymbolsIntermediatePath Include="$(IntermediateOutputPath)$(TargetName).pdb" Condition="'$(OutputType)' != 'winmdobj' and '@(_DebugSymbolsIntermediatePath)' == ''"/>
@@ -771,7 +780,7 @@ Copyright (C) Microsoft Corporation. All rights reserved.
       Name="Build"
       Condition=" '$(_InvalidConfigurationWarning)' != 'true' "
       DependsOnTargets="$(BuildDependsOn)"
-      Returns="$(TargetPath)" />
+      Returns="@(TargetPathWithTargetPlatformMoniker)" />
 
   <!--
     ============================================================
@@ -1797,6 +1806,7 @@ Copyright (C) Microsoft Corporation. All rights reserved.
       <TargetPathWithTargetPlatformMoniker Include="$(TargetPath)">
         <TargetPlatformMoniker>$(TargetPlatformMoniker)</TargetPlatformMoniker>
         <TargetPlatformIdentifier>$(TargetPlatformIdentifier)</TargetPlatformIdentifier>
+        <ReferenceAssembly Condition="'$(TargetRefPath)' != ''">$(TargetRefPath)</ReferenceAssembly>
       </TargetPathWithTargetPlatformMoniker>
     </ItemGroup>
   </Target>
@@ -2015,8 +2025,18 @@ Copyright (C) Microsoft Corporation. All rights reserved.
       <Output TaskParameter="FilesWritten" ItemName="FileWrites"/>
       <Output TaskParameter="DependsOnSystemRuntime" PropertyName="DependsOnSystemRuntime"/>
     </ResolveAssemblyReference>
+
+    <ItemGroup>
+      <ReferencePathWithInterfaceOnlyAssemblies Include="@(ReferencePath->'%(ReferenceAssembly)')" />
+    </ItemGroup>
   </Target>
 
+  <ItemDefinitionGroup>
+    <ReferencePath>
+      <ReferenceAssembly>%(FullPath)</ReferenceAssembly>
+    </ReferencePath>
+  </ItemDefinitionGroup>
+
   <!--
     ====================================================================================================
 
@@ -4003,6 +4023,17 @@ Copyright (C) Microsoft Corporation. All rights reserved.
 
     </Copy>
 
+    <!-- Copy the build product (.dll or .exe). -->
+    <CopyRefAssembly
+        SourcePath="@(IntermediateRefAssembly)"
+        DestinationPath="$(TargetRefPath)"
+        Condition="'$(CopyBuildOutputToOutputDirectory)' == 'true' and '$(SkipCopyBuildProduct)' != 'true' and Exists('@(IntermediateRefAssembly)')"
+            >
+
+      <Output TaskParameter="DestinationPath" ItemName="ReferenceAssembly"/>
+
+    </CopyRefAssembly>
+
     <Message Importance="High" Text="$(MSBuildProjectName) -&gt; @(MainAssembly->'%(FullPath)')" Condition="'$(CopyBuildOutputToOutputDirectory)' == 'true' and '$(SkipCopyBuildProduct)'!='true'" />
 
     <!-- Copy the additional modules. -->
diff --git a/MSBuild/15.0/Bin/Microsoft.Common.tasks b/MSBuild/15.0/Bin/Microsoft.Common.tasks
index d284df7..4c7fbee 100644
--- a/MSBuild/15.0/Bin/Microsoft.Common.tasks
+++ b/MSBuild/15.0/Bin/Microsoft.Common.tasks
@@ -172,5 +172,6 @@
     <!-- Roslyn tasks are now in an assembly owned and shipped by Roslyn -->
     <UsingTask TaskName="Microsoft.CodeAnalysis.BuildTasks.Csc"                       AssemblyFile="$(RoslynTargetsPath)\Microsoft.Build.Tasks.CodeAnalysis.dll" Condition="'$(MSBuildAssemblyVersion)' != ''" />
     <UsingTask TaskName="Microsoft.CodeAnalysis.BuildTasks.Vbc"                       AssemblyFile="$(RoslynTargetsPath)\Microsoft.Build.Tasks.CodeAnalysis.dll" Condition="'$(MSBuildAssemblyVersion)' != ''" />
+    <UsingTask TaskName="Microsoft.CodeAnalysis.BuildTasks.CopyRefAssembly"           AssemblyFile="$(RoslynTargetsPath)\Microsoft.Build.Tasks.CodeAnalysis.dll" Condition="'$(MSBuildAssemblyVersion)' != ''" />
 </Project>
 
diff --git a/MSBuild/15.0/Bin/Roslyn/Microsoft.CSharp.Core.targets b/MSBuild/15.0/Bin/Roslyn/Microsoft.CSharp.Core.targets
index 5612c75..2aee7ca 100644
--- a/MSBuild/15.0/Bin/Roslyn/Microsoft.CSharp.Core.targets
+++ b/MSBuild/15.0/Bin/Roslyn/Microsoft.CSharp.Core.targets
@@ -7,7 +7,7 @@
                   @(_CoreCompileResourceInputs);
                   $(ApplicationIcon);
                   $(AssemblyOriginatorKeyFile);
-                  @(ReferencePath);
+                  @(TrueReferences);
                   @(CompiledLicenseFile);
                   @(LinkResource);
                   @(EmbeddedDocumentation);
@@ -117,7 +117,7 @@
          Prefer32Bit="$(Prefer32Bit)"
          PreferredUILang="$(PreferredUILang)"
          ProvideCommandLineArgs="$(ProvideCommandLineArgs)"
-         References="@(ReferencePath)"
+         References="@(TrueReferences)"
          ReportAnalyzer="$(ReportAnalyzer)"
          Resources="@(_CoreCompileResourceInputs);@(CompiledLicenseFile)"
          ResponseFiles="$(CompilerResponseFile)"

jaredpar · 2017-04-26T21:45:13Z

These all look good to me.

jcouv · 2017-04-28T02:53:36Z

Looks good. I especially like the leveraging of existing /obj and /bin distinction. At least we don't have to create a third copy of the ref assembly just to manage timestamps :-)

rainersigwald · 2017-04-28T21:01:26Z

Had a design review with @AndyGerlicher and @jeffkl today. No changes to the plan, but a couple of options were discussed and dismissed:

Idea: extracting the MVIDs of references to compute incremental up-to-dateness. Rejected because while we now (post-Improved Incremental Build for C# and VB projects #1327) have an easy way to cause CoreCompile to run even if up-to-date-checks would otherwise skip it, there's no good way to do the opposite.
Idea: leave the reference assembly in obj, never copying it to bin. Rejected because we think the ref asm is a real project output, and because this would involve telling the compiler to generate it to an intermediate location and copying it around--so there's little gain.

Also discussed was dotnet/roslyn#19103.

I'm going to go forward with PRs now based on this design.

borgdylan · 2017-05-01T10:42:00Z

Will this be property based? I.e. will be be a property based switch that turns reference assembly making on/off? Please allow compilers other than roslyn to implement the feature.

jaredpar · 2017-05-01T16:17:45Z

There isn't anything specific to Roslyn that I'm aware of here. Most of the work is around making the target graphs able to benefit from reference assemblies in general.

Provides an item, `ReferencePathWithInterfaceOnlyAssemblies`, that consists of the reference (interface-only) versions of assemblies where available, allowing a consuming task to be incrementally up-to-date after implementation-only changes have been made to a reference. When `%(ReferenceAssembly)` metadata is unavailable for a given reference, the new item contains the implementation assembly directly. Additionally creates a new output item for the current project's reference assembly by default when `$(Deterministic)` is `true`. If this is created, it is copied to the output directory using the new `CopyRefAssembly` task, which copies only if the ref assembly is different from the current file. If present, the ref asm is passed to the project's consumes in metadata. See #1986, dotnet/roslyn#2184.

rainersigwald · 2017-05-01T19:03:25Z

@borgdylan I'm planning to implement that property next up in PR #2039.

This is absolutely available to compilers other than Roslyn--the core compilation target just needs to consume the new item rather than @(ReferencePath) to opt into using reference assemblies, and needs to define the ReferenceAssembly metadata on its output if it produces ref assemblies.

rainersigwald · 2017-05-01T19:48:45Z

@davkean @jviau I'm proposing to change the item that gets passed to Csc in CoreCompile to a new one that has interface-only reference assemblies substituted in for implementation assemblies where available. @jasonmalinowski is worried that this might break something in the project system. Do you know if the project system or CPS directly cares about @(ReferencePath)? If it does, what's the nature of that relationship?

To summarize: I'm concerned about swapping in the ref assembly everywhere because I don't know what targets (including third-party extension things) have a dependency on @(ReferencePath) containing the full output assembly of references. But we do want to use the reference assemblies for things like design-time builds.

borgdylan · 2017-05-01T19:49:18Z

Thanks for clarifying. I will try to implement this for my compiler since it would be for me to tell thr compiler to forgo generation of method bodies. That would allow me to implement the MSBuild specific bits that interface with your PR. Would MSBuild suggest a path where the reference assembly should be placed?

rainersigwald · 2017-05-01T19:53:47Z

@borgdylan Yes, in my PR the common targets suggest a default--though you'll want to opt in to the feature once I've made that possible. Csc consumes it fairly simply: https://github.com/rainersigwald/roslyn/blob/2164877d0099c291f2aee22773058ba01fa78266/src/Compilers/Core/MSBuildTask/Microsoft.CSharp.Core.targets#L126, and I would expect that you could do the same.

Also possibly relevant: dotnet/roslyn#19133 adds some requirements to the shape of the ref assembly to keep it compatible with the new incremental CopyRefAssembly task.

borgdylan · 2017-05-01T20:02:24Z

Thanks for heads up. I did not know of @(IntermediateAssembly) and $(PdbFile). I was concatenating the components myself.

rainersigwald · 2017-05-01T20:09:12Z

@borgdylan I have a meeting set up with a doc writer to figure out how to start to document some of the soup that is common targets. That kind of detail will likely be low on the priority list since it's primarily useful for compiler writers, but maybe someday. . .

borgdylan · 2017-05-01T20:21:42Z

In the mean time I will look at the CSharp targets for inspiration once this makes its way into the mono fork.

davkean · 2017-05-02T00:52:23Z

@rainersigwald I had a quick look, I don't know of anything that would break by using a different item to pass to the compiler instead @(ReferencePath).

I do suspect, however, that the legacy project system (and our yet to be written up-to-date check) would need to be taught about these concepts, preferable via MSBuild items.

Provides an item, `ReferencePathWithInterfaceOnlyAssemblies`, that consists of the reference (interface-only) versions of assemblies where available, allowing a consuming task to be incrementally up-to-date after implementation-only changes have been made to a reference. When `%(ReferenceAssembly)` metadata is unavailable for a given reference, the new item contains the implementation assembly directly. Additionally creates a new output item for the current project's reference assembly when `$(ProduceReferenceAssembly)` is true. If this is created, it is copied to the output directory using the new `CopyRefAssembly` task, which copies only if the ref assembly is different from the current file. If present, the ref asm is passed to the project's consumes in metadata. Introduce $(CompileUsingReferenceAssemblies), which can avoid passing reference assemblies to the compiler when set to false. This is not expected to be commonly used. IDE fast up-to-date checks need to be able to compare timestamps on _both_ the reference assembly (to see if a recompile is needed) and the implementation assembly (because it needs to be copied to an output location). Provide OriginalPathmetadata on the adjusted references that points back to the implementation assembly if a ref assembly was found. When a project both uses and produces reference assemblies, its primary output assembly might be up to date while (transitive) references need to be copied to its output directory. This case fooled the Visual Studio Fast Up-To-Date check, because the transitive dependencies aren't listed as project inputs (because design-time builds disable RAR's FindDependencies for performance reasons). To account for this, introduce a new file that is updated whenever _CopyFilesMarkedCopyLocal does actual work, pass it along as part of the project's output, and consider it a project input for the FUTD check. See #1986, dotnet/roslyn#2184.

rainersigwald · 2017-05-23T22:22:16Z

The initial implementation of this is complete, but opt-in.

borgdylan · 2017-06-22T15:47:51Z

@rainersigwald how does msbuild signal that it wants a reference assembly?

rainersigwald · 2017-06-22T16:39:27Z

@borgdylan

The design we landed on was:

The producing project respects $(ProduceReferenceAssembly) to determine whether or not to produce the ref assembly. If it does:
- The return value of GetTargetPath and Build should be an item that has the ReferenceAssembly and CopyUpToDateMarker metadata set to appropriate values.
The referencing project respects $(CompileUsingReferenceAssemblies) (as an emergency escape hatch), and if it's not set to false, passes @(ReferencePath->'%(ReferenceAssembly)') to the compiler instead of just @(ReferencePath).

So there's no real way to indicate from a referencing project that you want ref assemblies, it's up to the producing project to provide them or not.

Does that answer the question?

borgdylan · 2017-06-22T18:15:09Z

On the receiving end I am just using ReferencePathWithRefAssemblies since I know that I will be having v15.6 installed. I want to compile against reference assemblies whenever they are available.

Thanks for answering the part on the producing end. For now I just want to have my targets set up to handle the reference assembly stuff prior to my compiler actually emitting them. In the mean time I will be tricking msbuild by giving a full assembly regardless of what it asks for. However I plan to be writing the metadata and passing a flag to the compiler in the reference assembly production case.

borgdylan · 2017-06-22T18:16:51Z

What I did not understand: How do I change the values for the mentioned targets from within CoreCompile?

borgdylan · 2017-06-22T18:21:23Z

Also, what causes $(ProduceReferenceAssembly) to be set to true? It is always false in my builds. As I see it something must be in place to automatically trigger a ref assembly for all projects that are dependencies of the one being built.

rainersigwald · 2017-06-22T18:56:10Z

@borgdylan Right now ref assemblies are an opt-in feature, so a project must explicitly set <ProduceReferenceAssembly>true</ProduceReferenceAssembly> (or get it through an import--Directory.Build.props makes that easy).

I don't think I understand this:

As I see it something must be in place to automatically trigger a ref assembly for all projects that are dependencies of the one being built.

Why? It's not a great situation, but it's perfectly valid to have a non-ref-assembly-aware project depend on a ref assembly but not produce one in turn. You'd still get incremental build benefits since it would only rebuild when its own implementation or the reference's interface changes.

borgdylan · 2017-06-22T19:08:59Z

What I meant is that a dependency project should know that it is currently treated as a dependency and is not the focus, so it would be enough for it to produce a reference assembly only. Making both a normal and a reference assembly is too wasteful in the same build and defeats the use of reference assemblies to get a faster build.

rainersigwald · 2017-06-22T19:28:17Z

Ah, I see. That's not a scenario in the current design.

The Roslyn compiler simultaneously emits the reference and implementation assemblies, so it's not significantly slower than emitting only the reference assembly (AFAIK).

Even if that wasn't the case, I don't think I agree with your premise. If you didn't compile the implementation assembly when it was out of date but the build was triggered from a reference, you would potentially be using stale assemblies downstream.

Consider this situation (apologies if I'm butchering the Dylan):

define method adder  (a, b :: <integer>)
    a - b
end;

in one assembly adder.dylproj, and a unit test that verifies that adder adds in test.dylproj. Build test.dylproj, producing adder.dll, ref/adder.dll, and test.dll and observe that the test fails because the implementation of the method is wrong. Fix the error, then build the test project again. What you want is adder.dll', with internals, but an unmodified ref/adder.dll. Under your proposal, if only the ref assembly is regenerated, the test would be rerun against the stale copy of the assembly.

borgdylan · 2017-06-22T20:34:22Z

Apologies accepted. As things stand, I will be holding off my implementation as I will not have any advantages if I do. If a file in a project changes, the build would still trigger making a new reference assembly anyways. I may do a partial implementation to make distributable reference assemblies be possible.

Side note: dylan.NET is a misnamed language as it does not use the original "Dylan" syntax. An "adder" method would be written as

method public integer adder(var a as integer, var b as integer)
    return a + b
end method

or inspired by C# 6 arrow methods

method public integer adder(var a as integer, var b as integer) =>  a + b

rainersigwald added this to the MSBuild 15.3 milestone Apr 17, 2017

rainersigwald self-assigned this Apr 17, 2017

rainersigwald mentioned this issue Apr 28, 2017

Consider having CopyRefAssembly inherit from Copy dotnet/roslyn#19103

Open

This was referenced May 1, 2017

Common.targets support for reference assemblies #2039

Merged

Use refasm-only items for compilation dotnet/roslyn#19146

Merged

rainersigwald mentioned this issue May 12, 2017

Document new properties related to reference assemblies #2096

Closed

2 tasks

0x53A mentioned this issue May 17, 2017

Produce ref assemblies from command-line and msbuild dotnet/fsharp#3066

Closed

rainersigwald closed this as completed May 23, 2017

rainersigwald mentioned this issue Jun 1, 2017

[Breaking Change] Targets that run after ResolveReferences to manipulate references are broken after upgrading to 15.3 #2134

Closed

This was referenced Jun 1, 2017

CoreResGen should consume $(ReferencePathWithRefAssemblies) instead of $(ReferencePath) #2161

Closed

[Work items] Refining ref assemblies dotnet/roslyn#17612

Closed

rainersigwald mentioned this issue Jun 2, 2017

Some transformations on @(ReferencePath) result in mangled references #2165

Closed

MSDN-WhiteKnight mentioned this issue Sep 13, 2019

Add reference assemblies topic dotnet/docs#14393

Merged

charlesroddie mentioned this issue Nov 22, 2020

Does F# support ProduceReferenceAssembly dotnet/fsharp#8642

Closed

AR-May added the triaged label Feb 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support reference assemblies in build flows #1986

Support reference assemblies in build flows #1986

rainersigwald commented Apr 17, 2017 •

edited

Loading

rainersigwald commented Apr 17, 2017

rainersigwald commented Apr 17, 2017

rainersigwald commented Apr 25, 2017

rainersigwald commented Apr 26, 2017

jaredpar commented Apr 26, 2017

jcouv commented Apr 28, 2017

rainersigwald commented Apr 28, 2017

borgdylan commented May 1, 2017

jaredpar commented May 1, 2017

rainersigwald commented May 1, 2017

rainersigwald commented May 1, 2017

borgdylan commented May 1, 2017

rainersigwald commented May 1, 2017

borgdylan commented May 1, 2017

rainersigwald commented May 1, 2017

borgdylan commented May 1, 2017

davkean commented May 2, 2017 •

edited

Loading

rainersigwald commented May 23, 2017

borgdylan commented Jun 22, 2017

rainersigwald commented Jun 22, 2017

borgdylan commented Jun 22, 2017

borgdylan commented Jun 22, 2017

borgdylan commented Jun 22, 2017 •

edited

Loading

rainersigwald commented Jun 22, 2017 •

edited by drewnoakes

Loading

borgdylan commented Jun 22, 2017

rainersigwald commented Jun 22, 2017

borgdylan commented Jun 22, 2017 •

edited

Loading

Support reference assemblies in build flows #1986

Support reference assemblies in build flows #1986

Comments

rainersigwald commented Apr 17, 2017 • edited Loading

Design criteria

rainersigwald commented Apr 17, 2017

rainersigwald commented Apr 17, 2017

rainersigwald commented Apr 25, 2017

rainersigwald commented Apr 26, 2017

jaredpar commented Apr 26, 2017

jcouv commented Apr 28, 2017

rainersigwald commented Apr 28, 2017

borgdylan commented May 1, 2017

jaredpar commented May 1, 2017

rainersigwald commented May 1, 2017

rainersigwald commented May 1, 2017

borgdylan commented May 1, 2017

rainersigwald commented May 1, 2017

borgdylan commented May 1, 2017

rainersigwald commented May 1, 2017

borgdylan commented May 1, 2017

davkean commented May 2, 2017 • edited Loading

rainersigwald commented May 23, 2017

borgdylan commented Jun 22, 2017

rainersigwald commented Jun 22, 2017

borgdylan commented Jun 22, 2017

borgdylan commented Jun 22, 2017

borgdylan commented Jun 22, 2017 • edited Loading

rainersigwald commented Jun 22, 2017 • edited by drewnoakes Loading

borgdylan commented Jun 22, 2017

rainersigwald commented Jun 22, 2017

borgdylan commented Jun 22, 2017 • edited Loading

rainersigwald commented Apr 17, 2017 •

edited

Loading

davkean commented May 2, 2017 •

edited

Loading

borgdylan commented Jun 22, 2017 •

edited

Loading

rainersigwald commented Jun 22, 2017 •

edited by drewnoakes

Loading

borgdylan commented Jun 22, 2017 •

edited

Loading