Microsoft.Extensions.Caching.Memory.Test hitting FailFast in CI

[jaredpar]

Console Log Summary

C:\h\w\A12508E4\w\A865097A\e>"C:\h\w\A12508E4\p\dotnet.exe" exec --runtimeconfig Microsoft.Extensions.Caching.Memory.Tests.runtimeconfig.json --depsfile Microsoft.Extensions.Caching.Memory.Tests.deps.json xunit.console.dll Microsoft.Extensions.Caching.Memory.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing -notrait category=nonnetcoreapptests -notrait category=nonwindowstests  
* Assertion at F:\workspace\_work\1\s\src\mono\mono\metadata\metadata.c:1179, condition `res_size == count' not met

Builds

Build	Pull Request	Test Failure Count
#596902	#34344	10
#597884	#34868	1
#597986	Rolling	1
#598049	#34297	1
#598077	#34877	6
#598667	#34909	1
#598691	#33924	1
#599401	#34825	1
#599550	#34931	3
#599616	Rolling	4
#599674	#34933	3
#599696	#34777	3
#599722	#34569	3
#599754	Rolling	4
#599756	#34934	3
#599759	#34935	3
#599793	#34764	3
#599829	#34930	3
#599833	#34764	3
#599836	#34939	6
#599858	#34940	3
#599878	Rolling	4
#599885	#34444	3
#599921	#34941	3

Configurations

• netcoreapp5.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.311.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.11-helix-bfcd90a-20200123191053
• netcoreapp5.0-Linux-Debug-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open
• netcoreapp5.0-Linux-Debug-x64-Mono_release-Ubuntu.1604.Amd64.Open
• netcoreapp5.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open
• netcoreapp5.0-Windows_NT-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open
• netcoreapp5.0-Windows_NT-Debug-x64-Mono_release-Windows.10.Amd64.Server19H1.ES.Open
• netcoreapp5.0-Windows_NT-Debug-x64-Mono_release-Windows.7.Amd64.Open
• netcoreapp5.0-Windows_NT-Debug-x64-Mono_release-Windows.81.Amd64.Open
• netcoreapp5.0-Windows_NT-Release-x64-Mono_release-Windows.10.Amd64.Server19H1.Open
• netcoreapp5.0-Windows_NT-Release-x64-Mono_release-Windows.10.Amd64.ServerRS5.Open
• netcoreapp5.0-Windows_NT-Release-x64-Mono_release-Windows.7.Amd64.Open
• netcoreapp5.0-Windows_NT-Release-x64-Mono_release-Windows.81.Amd64.Open
• netcoreapp5.0-Windows_NT-Release-x86-CoreCLR_checked-Windows.10.Amd64.Open

Helix Logs

Build	Pull Request	Console	Core	Test Results	Run Client
#596902	#34344	console.log			run_client.py
#596902	#34344	console.log			run_client.py
#596902	#34344	console.log			run_client.py
#596902	#34344	console.log			run_client.py
#596902	#34344	console.log			run_client.py
#596902	#34344	console.log			run_client.py
#596902	#34344	console.log			run_client.py
#596902	#34344	console.log			run_client.py
#596902	#34344	console.log			run_client.py
#596902	#34344	console.log			run_client.py
#597884	#34868	console.log			run_client.py
#597986	Rolling	console.log			run_client.py
#598049	#34297	console.log			run_client.py
#598077	#34877	console.log			run_client.py
#598077	#34877	console.log			run_client.py
#598077	#34877	console.log			run_client.py
#598077	#34877	console.log			run_client.py
#598077	#34877	console.log			run_client.py
#598077	#34877	console.log			run_client.py
#598667	#34909	console.log		testResults.xml	run_client.py
#598691	#33924	console.log			run_client.py
#599401	#34825	console.log			run_client.py
#599550	#34931	console.log			run_client.py
#599550	#34931	console.log			run_client.py
#599550	#34931	console.log			run_client.py
#599616	Rolling	console.log			run_client.py
#599616	Rolling	console.log			run_client.py
#599616	Rolling	console.log			run_client.py
#599616	Rolling	console.log			run_client.py
#599674	#34933	console.log			run_client.py
#599674	#34933	console.log			run_client.py
#599674	#34933	console.log			run_client.py
#599696	#34777	console.log			run_client.py
#599696	#34777	console.log			run_client.py
#599696	#34777	console.log			run_client.py
#599722	#34569	console.log			run_client.py
#599722	#34569	console.log			run_client.py
#599722	#34569	console.log			run_client.py
#599754	Rolling	console.log			run_client.py
#599754	Rolling	console.log			run_client.py
#599754	Rolling	console.log			run_client.py
#599754	Rolling	console.log			run_client.py
#599756	#34934	console.log			run_client.py
#599756	#34934	console.log			run_client.py
#599756	#34934	console.log			run_client.py
#599759	#34935	console.log			run_client.py
#599759	#34935	console.log			run_client.py
#599759	#34935	console.log			run_client.py
#599793	#34764	console.log			run_client.py
#599793	#34764	console.log			run_client.py
#599793	#34764	console.log			run_client.py
#599829	#34930	console.log			run_client.py
#599829	#34930	console.log			run_client.py
#599829	#34930	console.log			run_client.py
#599833	#34764	console.log			run_client.py
#599833	#34764	console.log			run_client.py
#599833	#34764	console.log			run_client.py
#599836	#34939	console.log			run_client.py
#599836	#34939	console.log			run_client.py
#599836	#34939	console.log			run_client.py
#599836	#34939	console.log			run_client.py
#599836	#34939	console.log			run_client.py
#599836	#34939	console.log			run_client.py
#599858	#34940	console.log			run_client.py
#599858	#34940	console.log			run_client.py
#599858	#34940	console.log			run_client.py
#599878	Rolling	console.log			run_client.py
#599878	Rolling	console.log			run_client.py
#599878	Rolling	console.log			run_client.py
#599878	Rolling	console.log			run_client.py
#599885	#34444	console.log			run_client.py
#599885	#34444	console.log			run_client.py
#599885	#34444	console.log			run_client.py
#599921	#34941	console.log			run_client.py
#599921	#34941	console.log			run_client.py
#599921	#34941	console.log			run_client.py

[ghost]

Tagging subscribers to this area: @ViktorHofer
Notify danmosemsft if you want to be subscribed.

[ViktorHofer]

@jaredpar thanks for reporting. When opening such issues can you please check if the bot sets the right label so that area-owners are notified?

[jaredpar]

FYI: I don't think that is the right label. The failure is due to an assertion in metadata.c. Seems like a runtime issue. Also pretty much all of CI is down right now with a FailFast exception. Guessing it is the single root cause of all the failures.

[jaredpar]

Here is the full list of failures

Common.Tests Work Item 12
Invariant.Tests Work Item 12
Microsoft.CSharp.Tests Work Item 12
Microsoft.Extensions.Caching.Memory.Tests Work Item 12
Microsoft.Extensions.Configuration.Binder.Tests Work Item 12
Microsoft.Extensions.Configuration.CommandLine.Tests Work Item 12
Microsoft.Extensions.Configuration.EnvironmentVariables.Tests Work Item 12
Microsoft.Extensions.Configuration.FileExtensions.Tests Work Item 12
Microsoft.Extensions.Configuration.Functional.Tests Work Item 12
Microsoft.Extensions.Configuration.Ini.Tests Work Item 12
Microsoft.Extensions.Configuration.Json.Tests Work Item 12
Microsoft.Extensions.Configuration.Tests Work Item 12
Microsoft.Extensions.Configuration.UserSecrets.Tests Work Item 12
Microsoft.Extensions.Configuration.Xml.Tests Work Item 12
Microsoft.Extensions.DependencyInjection.Tests Work Item 12
Microsoft.Extensions.DependencyModel.Tests Work Item 12
Microsoft.Extensions.FileProviders.Composite.Tests Work Item 12
Microsoft.Extensions.FileProviders.Physical.Tests Work Item 12
Microsoft.Extensions.FileSystemGlobbing.Tests Work Item 12
Microsoft.Extensions.Hosting.Functional.Tests Work Item 12
Microsoft.Extensions.Hosting.Unit.Tests Work Item 12
Microsoft.Extensions.Http.Tests Work Item 12
Microsoft.Extensions.Logging.EventSource.Tests Work Item 12
Microsoft.Extensions.Logging.Testing.Tests Work Item 12
Microsoft.Extensions.Logging.Tests Work Item 12
Microsoft.Extensions.Options.Tests Work Item 12
Microsoft.Extensions.Primitives.Tests Work Item 12
Microsoft.VisualBasic.Core.Tests Work Item 12
Microsoft.Win32.Primitives.Tests Work Item 12
Microsoft.Win32.Registry.AccessControl.Tests Work Item 12
Microsoft.Win32.Registry.Tests Work Item 12
Microsoft.Win32.SystemEvents.Tests Work Item 12
Microsoft.XmlSerializer.Generator.Tests Work Item 12
System.AppContext.Tests Work Item 12
System.Buffers.Tests Work Item 12
System.CodeDom.Tests Work Item 12
System.Collections.Concurrent.Tests Work Item 12
System.Collections.Immutable.Tests Work Item 12
System.Collections.NonGeneric.Tests Work Item 12
System.Collections.Specialized.Tests Work Item 12
System.Collections.Tests Work Item 12
System.ComponentModel.Annotations.Tests Work Item 12
System.ComponentModel.Composition.Registration.Tests Work Item 12
System.ComponentModel.Composition.Tests Work Item 12
System.ComponentModel.EventBasedAsync.Tests Work Item 12
System.ComponentModel.Primitives.Tests Work Item 12
System.ComponentModel.Tests Work Item 12
System.ComponentModel.TypeConverter.Tests Work Item 12
System.Composition.AttributeModel.Tests Work Item 12
System.Composition.Convention.Tests Work Item 12
System.Composition.Hosting.Tests Work Item 12
System.Composition.Runtime.Tests Work Item 12
System.Composition.Tests Work Item 12
System.Composition.TypedParts.Tests Work Item 12
System.Configuration.ConfigurationManager.Tests Work Item 12
System.Console.Manual.Tests Work Item 12
System.Console.Tests Work Item 12
System.Data.Common.Tests Work Item 12
System.Data.DataSetExtensions.Tests Work Item 12
System.Data.Odbc.Tests Work Item 12
System.Data.OleDb.Tests Work Item 12
System.Diagnostics.Contracts.Tests Work Item 12
System.Diagnostics.Debug.Tests Work Item 12
System.Diagnostics.DiagnosticSource.Tests Work Item 12
System.Diagnostics.EventLog.Tests Work Item 12
System.Diagnostics.FileVersionInfo.Tests Work Item 12
System.Diagnostics.PerformanceCounter.Tests Work Item 12
System.Diagnostics.Process.Tests Work Item 12
System.Diagnostics.StackTrace.Tests Work Item 12
System.Diagnostics.TextWriterTraceListener.Tests Work Item 12
System.Diagnostics.Tools.Tests Work Item 12
System.Diagnostics.TraceSource.Tests Work Item 12
System.Diagnostics.Tracing.Tests Work Item 12
System.DirectoryServices.AccountManagement.Tests Work Item 12
System.DirectoryServices.Protocols.Tests Work Item 12
System.DirectoryServices.Tests Work Item 12
System.Drawing.Common.Tests Work Item 12
System.Drawing.Primitives.Tests Work Item 12
System.Dynamic.Runtime.Tests Work Item 12
System.Globalization.Calendars.Tests Work Item 12
System.Globalization.CalendarsWithConfigSwitch.Tests Work Item 12
System.Globalization.Extensions.Tests Work Item 12
System.Globalization.Tests Work Item 12
System.IO.Compression.Brotli.Tests Work Item 12
System.IO.Compression.Tests Work Item 12
System.IO.Compression.ZipFile.Tests Work Item 12
System.IO.FileSystem.AccessControl.Tests Work Item 12
System.IO.FileSystem.DriveInfo.Tests Work Item 12
System.IO.FileSystem.Primitives.Tests Work Item 12
System.IO.FileSystem.Tests Work Item 12
System.IO.FileSystem.Watcher.Tests Work Item 12
System.IO.IsolatedStorage.Tests Work Item 12
System.IO.MemoryMappedFiles.Tests Work Item 12
System.IO.Packaging.Tests Work Item 12
System.IO.Pipelines.Tests Work Item 12
System.IO.Pipes.AccessControl.Tests Work Item 12
System.IO.Pipes.Tests Work Item 12
System.IO.Ports.Tests Work Item 12
System.IO.Tests Work Item 12
System.IO.UnmanagedMemoryStream.Tests Work Item 12
System.Linq.Expressions.Tests Work Item 12
System.Linq.Parallel.Tests Work Item 12
System.Linq.Queryable.Tests Work Item 12
System.Linq.Tests Work Item 12
System.Management.Tests Work Item 12
System.Memory.Tests Work Item 12
System.Net.Http.Json.Functional.Tests Work Item 12
System.Net.Http.Json.Unit.Tests Work Item 12
System.Net.Http.Unit.Tests Work Item 12
System.Net.Http.WinHttpHandler.Functional.Tests Work Item 12
System.Net.Http.WinHttpHandler.Unit.Tests Work Item 12
System.Net.HttpListener.Tests Work Item 12
System.Net.Mail.Functional.Tests Work Item 12
System.Net.Mail.Unit.Tests Work Item 12
System.Net.NameResolution.Functional.Tests Work Item 12
System.Net.NameResolution.Pal.Tests Work Item 12
System.Net.NameResolution.Unit.Tests Work Item 12
System.Net.NetworkInformation.Functional.Tests Work Item 12
System.Net.Ping.Functional.Tests Work Item 12
System.Net.Primitives.Functional.Tests Work Item 12
System.Net.Primitives.Pal.Tests Work Item 12
System.Net.Primitives.UnitTests.Tests Work Item 12
System.Net.Requests.Tests Work Item 12
System.Net.Security.Unit.Tests Work Item 12
System.Net.ServicePoint.Tests Work Item 12
System.Net.Sockets.Tests Work Item 12
System.Net.WebClient.Tests Work Item 12
System.Net.WebHeaderCollection.Tests Work Item 12
System.Net.WebProxy.Tests Work Item 12
System.Net.WebSockets.Client.Tests Work Item 12
System.Net.WebSockets.Tests Work Item 12
System.Net.WebSockets.WebSocketProtocol.Tests Work Item 12
System.Numerics.Tensors.Tests Work Item 12
System.Numerics.Vectors.Tests Work Item 12
System.ObjectModel.Tests Work Item 12
System.Private.Uri.ExtendedFunctional.Tests Work Item 12
System.Private.Uri.Functional.Tests Work Item 12
System.Private.Uri.Unit.Tests Work Item 12
System.Reflection.Context.Tests Work Item 12
System.Reflection.CoreCLR.Tests Work Item 12
System.Reflection.DispatchProxy.Tests Work Item 12
System.Reflection.Emit.ILGeneration.Tests Work Item 12
System.Reflection.Emit.Lightweight.Tests Work Item 12
System.Reflection.Emit.Tests Work Item 12
System.Reflection.Extensions.Tests Work Item 12
System.Reflection.Metadata.Tests Work Item 12
System.Reflection.MetadataLoadContext.Tests Work Item 12
System.Reflection.Tests Work Item 12
System.Reflection.TypeExtensions.Tests Work Item 12
System.Resources.Extensions.Tests Work Item 12
System.Resources.Reader.Tests Work Item 12
System.Resources.ResourceManager.Tests Work Item 12
System.Resources.Writer.Tests Work Item 12
System.Runtime.Caching.Tests Work Item 12
System.Runtime.CompilerServices.Unsafe.Tests Work Item 12
System.Runtime.CompilerServices.VisualC.Tests Work Item 12
System.Runtime.Extensions.Tests Work Item 12
System.Runtime.Handles.Tests Work Item 12
System.Runtime.InteropServices.RuntimeInformation.Tests Work Item 12
System.Runtime.InteropServices.Tests Work Item 12
System.Runtime.InteropServices.WindowsRuntime.Tests Work Item 12
System.Runtime.Loader.DefaultContext.Tests Work Item 12
System.Runtime.Loader.RefEmitLoadContext.Tests Work Item 12
System.Runtime.Loader.Tests Work Item 12
System.Runtime.Numerics.Tests Work Item 12
System.Runtime.Serialization.Formatters.Tests Work Item 12
System.Runtime.Serialization.Json.ReflectionOnly.Tests Work Item 12
System.Runtime.Serialization.Json.Tests Work Item 12
System.Runtime.Serialization.Primitives.Tests Work Item 12
System.Runtime.Serialization.Xml.ReflectionOnly.Tests Work Item 12
System.Runtime.Serialization.Xml.Tests Work Item 12
System.Runtime.WindowsRuntime.Tests Work Item 12
System.Runtime.WindowsRuntime.UI.Xaml.Tests Work Item 12
System.Security.AccessControl.Tests Work Item 12
System.Security.Claims.Tests Work Item 12
System.Security.Cryptography.Algorithms.Tests Work Item 12
System.Security.Cryptography.Cng.Tests Work Item 12
System.Security.Cryptography.Csp.Tests Work Item 12
System.Security.Cryptography.Encoding.Tests Work Item 12
System.Security.Cryptography.Pkcs.Tests Work Item 12
System.Security.Cryptography.Primitives.Tests Work Item 12
System.Security.Cryptography.ProtectedData.Tests Work Item 12
System.Security.Cryptography.X509Certificates.Tests Work Item 12
System.Security.Cryptography.Xml.Tests Work Item 12
System.Security.Permissions.Tests Work Item 12
System.Security.Principal.Windows.Tests Work Item 12
System.Security.SecureString.Tests Work Item 12
System.ServiceModel.Syndication.Tests Work Item 12
System.ServiceProcess.ServiceController.Tests Work Item 12
System.Text.Encoding.CodePages.Tests Work Item 12
System.Text.Encoding.Extensions.Tests Work Item 12
System.Text.Encoding.Tests Work Item 12
System.Text.Encodings.Web.Tests Work Item 12
System.Text.Json.Tests Work Item 12
System.Text.RegularExpressions.Tests Work Item 12
System.Threading.AccessControl.Tests Work Item 12
System.Threading.Channels.Tests Work Item 12
System.Threading.Overlapped.Tests Work Item 12
System.Threading.Tasks.Dataflow.Tests Work Item 12
System.Threading.Tasks.Extensions.Tests Work Item 12
System.Threading.Tasks.Parallel.Tests Work Item 12
System.Threading.Tasks.Tests Work Item 12
System.Threading.Tests Work Item 12
System.Threading.Thread.Tests Work Item 12
System.Threading.ThreadPool.Tests Work Item 12
System.Threading.Timer.Tests Work Item 12
System.Transactions.Local.Tests Work Item 12
System.Utf8String.Experimental.Tests Work Item 12
System.ValueTuple.Tests Work Item 12
System.Web.HttpUtility.Tests Work Item 12
System.Windows.Extensions.Tests Work Item 12
System.Xml.Linq.Axes.Tests Work Item 12
System.Xml.Linq.Events.Tests Work Item 12
System.Xml.Linq.Misc.Tests Work Item 12
System.Xml.Linq.Properties.Tests Work Item 12
System.Xml.Linq.SDMSample.Tests Work Item 12
System.Xml.Linq.Streaming.Tests Work Item 12
System.Xml.Linq.TreeManipulation.Tests Work Item 12
System.Xml.Linq.xNodeBuilder.Tests Work Item 12
System.Xml.Linq.xNodeReader.Tests Work Item 12
System.Xml.Misc.Tests Work Item 12
System.Xml.RW.CharCheckingReader.Tests Work Item 12
System.Xml.RW.CustomReader.Tests Work Item 12
System.Xml.RW.FactoryReader.Tests Work Item 12
System.Xml.RW.NameTable.Tests Work Item 12
System.Xml.RW.ReaderSettings.Tests Work Item 12
System.Xml.RW.RwFactory.Tests Work Item 12
System.Xml.RW.SubtreeReader.Tests Work Item 12
System.Xml.RW.WrappedReader.Tests Work Item 12
System.Xml.RW.XmlReader.ReadContentAs.Tests Work Item 12
System.Xml.RW.XmlReader.Tests Work Item 12
System.Xml.RW.XmlSystemPathResolver.Tests Work Item 12
System.Xml.RW.XmlWriter.Tests Work Item 12
System.Xml.RW.XmlWriterApi.Tests Work Item 12
System.Xml.Schema.Extensions.Tests Work Item 12
System.Xml.XmlDocument.Tests Work Item 12
System.Xml.XmlNodeReader.Tests Work Item 12
System.Xml.XmlResolver.Tests Work Item 12
System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests Work Item 12
System.Xml.XmlSchemaSet.Tests Work Item 12
System.Xml.XmlSerializer.ReflectionOnly.Tests Work Item 12
System.Xml.XmlSerializer.Tests Work Item 12
System.Xml.XPath.Tests Work Item 12
System.Xml.XPath.XDocument.Tests Work Item 12
System.Xml.XPath.XmlDocument.Tests Work Item 12
System.Xml.Xsl.XslCompiledTransformApi.Tests Work Item 12
System.Xml.Xsl.XslTransformApi.Tests Work Item 12
XsltCompiler.Tests Work Item 12

[ViktorHofer]

Got it, still this definitely isn't area-Infrastructure.

[akoeplinger]

The Mono test issues look like what should be fixed by #34951 but since I see some CoreCLR runs in your list those might be different issues?

[jaredpar]

@ViktorHofer

Sure I can fix the label. But next time you need to give me more than two minutes to do so. Given that CI is completely busted I had attached a higher priority at sending an alert that we needed to fix that.

@akoeplinger

How did this change get through the PR process without triggering a failure? Given that it took down all of CI I think it's important we understand what this gap is.

[akoeplinger]

How did this change get through the PR process without triggering a failure? Given that it took down all of CI I think it's important we understand what this gap is.

It did show on the PR but was still merged.

[jaredpar]

It did show on the PR but was still merged.

What was the source PR?

[akoeplinger]

#34877

[ViktorHofer]

Sure I can fix the label. But next time you need to give me more than two minutes to do so. Given that CI is completely busted I had attached a higher priority at sending an alert that we needed to fix that.

@jaredpar this was meant as constructive feedback. I didn't mean to offend you in any way.

[jaredpar]

@ViktorHofer no worries. I just got caught up digging into the tests right after I submitted. The failure assert rung a bell and I ran off in that area.

[akoeplinger]

The Mono issue was fixed so we can probably close this one.