Cannot use System.Interactive.Async v4+ in .NET Azure Function

[idg10]

If you add a reference to v4.1.1 of the System.Interactive.Async NuGet package to an Azure Functions project, you cannot use it because the RemoveRuntimeDependencies build task defined in this repo deletes it from your build output.

The code that performs the deletion lives in this repo at https://github.com/Azure/azure-functions-host/blob/1ed2a54159eb3c9aaff11a9289be2bf1ce91dcb5/tools/ExtensionsMetadataGenerator/src/ExtensionsMetadataGenerator/BuildTasks/RemoveRuntimeDependencies.cs, specifically:

azure-functions-host/tools/ExtensionsMetadataGenerator/src/ExtensionsMetadataGenerator/BuildTasks/RemoveRuntimeDependencies.cs

Lines 26 to 42 in 1ed2a54

	Assembly assembly = typeof(RemoveRuntimeDependencies).Assembly;
	using (Stream resource = assembly.GetManifestResourceStream(assembly.GetName().Name + ".runtimeassemblies.txt"))
	using (var reader = new StreamReader(resource))
	{
	string assemblyName = reader.ReadLine();
	while (!string.IsNullOrEmpty(assemblyName))
	{
	string fileName = Path.Combine(OutputPath, assemblyName);
	if (File.Exists(fileName))
	{
	File.Delete(fileName);
	}
	assemblyName = reader.ReadLine();
	}
	}

The list of files this uses is also in this repo, and the offending entry is at

azure-functions-host/tools/ExtensionsMetadataGenerator/src/ExtensionsMetadataGenerator/runtimeassemblies.txt

Line 228 in 1ed2a54

System.Interactive.Async.dll

The rationale for this appears to be that Functions already brings its own copy - see #6208

But the problem is that this appears to be a v3.x era System.Interactive.Async which uses a definition of IAsyncEnumerable<T> that is incompatible with C# 8.0's await foreach. The PR at dotnet/reactive#423 modified System.Interactive.Async to work with the new definition of this interface. This first became available in v4.0.

The main reason I was trying to use the features in System.Interactive.Async at all was to be able to use await foreach, so being prevented from using v4.x defeats the purpose for me.

(In the project I encountered this problem I was attempting to use the Buffer extension method that System.Interactive.Async defines. This Buffer method is not available on IAsyncEnumerable unless you add a reference to System.Interactive.Async, and although the v3.x version supported in Functions does define Buffer, it does it for the wrong, old version of IAsyncEnumerable<T>.)

Investigative information

Version information:

NuGet packages:

Microsoft.NET.Sdk.Functions v3.0.8
System.Interactive.Async v4.1.1

The local functions emulator displays this:

Azure Functions Core Tools (3.0.2630 Commit hash: beec61496e1c5de8aa4ba38d1884f7b48233a7ab)
Function Runtime Version: 3.0.13901.0

I'm not providing any cloud information because this problem occurs at build time. (It causes runtime problems, but you can see those in the local emulator)

Repro steps

1. Create a new Azure Functions project in Visual Studio
2. Ensure Azure Functions v3 (.NET Core) is selected
   1 Select Timer trigger
3. Click Create
4. Right-click Dependencies in project in Solution Explorer, select Manage Nuget Packages...
5. Click Updates in NuGet Package Manager
6. Select Microsoft.NET.Sdk.Functions
7. On the right, select version 3.0.8 and click Update
8. Click Browse in NuGet Package Manager
9. In Search type System.Interactive.Async
10. Select System.Interactive.Async from list, then ensure a v4 era version is selected (I've chosen 4.1.1)
11. Click Install, then OK, then I Accept
12. Open Function1.cs
13. Inside the Run method, add log.LogInformation(typeof(AsyncEnumerableEx).AssemblyQualifiedName);
14. Press F5 to run.
15. Wait for the timer to fire. (The Visual Studio template will have set it to run once every 5 minutes.)

An error occurs.

Expected behavior

The expected behaviour is that when the timer fires, it displays the assembly-qualified name of the AsyncEnumerableEx type in the log output. In fact, that's exactly what happens if you use Microsoft.NET.Sdk.Functions v3.0.3:

[09/07/2020 16:55:00] Executing 'Function1' (Reason='Timer fired at 2020-07-09T17:55:00.0104939+01:00', Id=d021a034-46ff-4487-9fe2-68217b1e36e9)
[09/07/2020 16:55:00] C# Timer trigger function executed at: 09/07/2020 17:55:00
[09/07/2020 16:55:00] System.Linq.AsyncEnumerableEx, System.Interactive.Async, Version=4.1.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263
[09/07/2020 16:55:00] Executed 'Function1' (Succeeded, Id=d021a034-46ff-4487-9fe2-68217b1e36e9)

Actual behavior

Although it works in the emulator with the 3.0.3 Microsoft.NET.Sdk.Functions package, it doesn't with 3.0.8. You get this output instead:

[09/07/2020 16:50:00] Executing 'Function1' (Reason='Timer fired at 2020-07-09T17:50:00.0048914+01:00', Id=4e3876f0-25a4-483f-a576-143529058cc3)
[09/07/2020 16:50:02] Executed 'Function1' (Failed, Id=4e3876f0-25a4-483f-a576-143529058cc3)
[09/07/2020 16:50:02] System.Private.CoreLib: Exception while executing function: Function1. FunctionsInteractiveAsyncRepro: Could not load file or assembly 'System.Interactive.Async, Version=4.1.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263'. The system cannot find the file specified.

[ankitkumarr]

Thanks for the elaborate issue!
I think it makes sense to me.

@fabiocav and @brettsam, would either of you have thoughts on this one? Wondering if we can update our reference of System.Interactive.Async to v4. Though, I am not sure if there are breaking changes that could break existing users' apps.

[xin9le]

I encountered a same problem. Currently, using IAsyncEnumerable<T> and System.Interactive.Async is unavoidable if we try to write more effective code using modern C#. Of course, we can workaround this issue like following.

<PropertyGroup>
    <_FunctionsSkipCleanOutput>true</_FunctionsSkipCleanOutput>
</PropertyGroup>

However, RemoveRuntimeDependencies build task that speed up the execution of Azure Functions is very attractive. If there're no major problems, I'd like you to remove System.Interactive.Async.dll from the removals list.

[ankitkumarr]

I really apologize for the super long delay here. I missed providing any updates on this. I don't think we could remove or update the major version of that package without being potentially breaking. As this is a Host runtime dependency, the way to keep the v4 of that dependency in the build artifacts would be one of two approaches --

1. What @xin9le above suggested. But as called out in their response, this would stop optimizing the package and can have some side effects.
2. You can add below to signal the build tasks to not remove this specific dll (recommended option) --

<ItemGroup>
    <FunctionsPreservedDependencies Include="System.Interactive.Async.dll" />
  </ItemGroup>

Note that this is only supported in Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator version 1.2.1+. So you may also need to add an explicit reference to the higher version of "ExtensionsMetadataGenerator" like --

<PackageReference Include="Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator" Version="1.2.1" />

I will leave this issue open in case people have further questions. Once again, I apologize that I missed providing any updates here.

[xin9le]

@ankitkumarr
Thanks you for your very useful reply. I'll try recommendation option you suggested. I'm happy I can remove _FunctionsSkipCleanOutput.