Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Many System.Net.Sockets.Tests failed with System.NullReferenceException : Object reference not set to an instance of an object #51480

Closed
VincentBu opened this issue Apr 19, 2021 · 12 comments

Comments

@VincentBu
Copy link
Contributor

Run: runtime-libraries-coreclr outerloop 20210418.2

Failed tests:

net6.0-OSX-Release-arm64-CoreCLR_release-OSX.1100.ARM64.Open
 -System.Net.Sockets.Tests.SendReceive_MemoryArrayTask.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_MemoryArrayTask.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.SendReceive_MemoryArrayTask.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_MemoryArrayTask.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.SendReceive_MemoryNativeTask.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_MemoryNativeTask.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.SendReceive_MemoryNativeTask.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_MemoryNativeTask.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.SendReceiveUdpClient.SendToRecvFromAsync_Datagram_UDP_UdpClient(loopbackAddress: ::1)
 -System.Net.Sockets.Tests.SendReceiveUdpClient.SendToRecvFromAsync_Datagram_UDP_UdpClient(loopbackAddress: 127.0.0.1)
 -System.Net.Sockets.Tests.SendReceive_Task.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_Task.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.SendReceive_Task.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_Task.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.AcceptSyncForceNonBlocking.Accept_Success(listenAt: ::1)
 -System.Net.Sockets.Tests.AcceptSyncForceNonBlocking.Accept_Success(listenAt: 127.0.0.1)
 -System.Net.Sockets.Tests.AcceptApm.Accept_Success(listenAt: ::1)
 -System.Net.Sockets.Tests.AcceptApm.Accept_Success(listenAt: 127.0.0.1)
 -System.Net.Sockets.Tests.AcceptEap.Accept_Success(listenAt: ::1)
 -System.Net.Sockets.Tests.AcceptEap.Accept_Success(listenAt: 127.0.0.1)
 -System.Net.Sockets.Tests.SendReceive_Apm.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_Apm.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.SendReceive_Apm.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_Apm.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.SendReceive_Eap.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_Eap.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.SendReceive_Eap.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_Eap.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.AcceptTask.Accept_Success(listenAt: ::1)
 -System.Net.Sockets.Tests.AcceptTask.Accept_Success(listenAt: 127.0.0.1)
 -System.Net.Sockets.Tests.AcceptSync.Accept_Success(listenAt: ::1)
 -System.Net.Sockets.Tests.AcceptSync.Accept_Success(listenAt: 127.0.0.1)

One of Error messages:

System.NullReferenceException : Object reference not set to an instance of an object.


Stack trace
   at System.Linq.Enumerable.WhereEnumerableIterator`1.Select[TResult](Func`2 selector) in /_/src/libraries/System.Linq/src/System/Linq/Where.cs:line 135
   at System.Linq.Enumerable.Select[TSource,TResult](IEnumerable`1 source, Func`2 selector) in /_/src/libraries/System.Linq/src/System/Linq/Select.cs:line 23
   at System.Net.Sockets.Tests.SendReceive`1.SendRecv_Stream_TCP_MultipleConcurrentSends(IPAddress listenAt, Boolean useMultipleBuffers) in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 474
--- End of stack trace from previous location ---
@ghost
Copy link

ghost commented Apr 19, 2021

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Run: runtime-libraries-coreclr outerloop 20210418.2

Failed tests:

net6.0-OSX-Release-arm64-CoreCLR_release-OSX.1100.ARM64.Open
 -System.Net.Sockets.Tests.SendReceive_MemoryArrayTask.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_MemoryArrayTask.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.SendReceive_MemoryArrayTask.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_MemoryArrayTask.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.SendReceive_MemoryNativeTask.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_MemoryNativeTask.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.SendReceive_MemoryNativeTask.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_MemoryNativeTask.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.SendReceiveUdpClient.SendToRecvFromAsync_Datagram_UDP_UdpClient(loopbackAddress: ::1)
 -System.Net.Sockets.Tests.SendReceiveUdpClient.SendToRecvFromAsync_Datagram_UDP_UdpClient(loopbackAddress: 127.0.0.1)
 -System.Net.Sockets.Tests.SendReceive_Task.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_Task.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.SendReceive_Task.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_Task.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.AcceptSyncForceNonBlocking.Accept_Success(listenAt: ::1)
 -System.Net.Sockets.Tests.AcceptSyncForceNonBlocking.Accept_Success(listenAt: 127.0.0.1)
 -System.Net.Sockets.Tests.AcceptApm.Accept_Success(listenAt: ::1)
 -System.Net.Sockets.Tests.AcceptApm.Accept_Success(listenAt: 127.0.0.1)
 -System.Net.Sockets.Tests.AcceptEap.Accept_Success(listenAt: ::1)
 -System.Net.Sockets.Tests.AcceptEap.Accept_Success(listenAt: 127.0.0.1)
 -System.Net.Sockets.Tests.SendReceive_Apm.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_Apm.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.SendReceive_Apm.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_Apm.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.SendReceive_Eap.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_Eap.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: 127.0.0.1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.SendReceive_Eap.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: False)
 -System.Net.Sockets.Tests.SendReceive_Eap.SendRecv_Stream_TCP_MultipleConcurrentSends(listenAt: ::1, useMultipleBuffers: True)
 -System.Net.Sockets.Tests.AcceptTask.Accept_Success(listenAt: ::1)
 -System.Net.Sockets.Tests.AcceptTask.Accept_Success(listenAt: 127.0.0.1)
 -System.Net.Sockets.Tests.AcceptSync.Accept_Success(listenAt: ::1)
 -System.Net.Sockets.Tests.AcceptSync.Accept_Success(listenAt: 127.0.0.1)

One of Error messages:

System.NullReferenceException : Object reference not set to an instance of an object.


Stack trace
   at System.Linq.Enumerable.WhereEnumerableIterator`1.Select[TResult](Func`2 selector) in /_/src/libraries/System.Linq/src/System/Linq/Where.cs:line 135
   at System.Linq.Enumerable.Select[TSource,TResult](IEnumerable`1 source, Func`2 selector) in /_/src/libraries/System.Linq/src/System/Linq/Select.cs:line 23
   at System.Net.Sockets.Tests.SendReceive`1.SendRecv_Stream_TCP_MultipleConcurrentSends(IPAddress listenAt, Boolean useMultipleBuffers) in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs:line 474
--- End of stack trace from previous location ---
Author: VincentBu
Assignees: -
Labels:

arch-arm64, area-System.Net.Sockets, os-mac-os-x

Milestone: -

@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Apr 19, 2021
@antonfirsov
Copy link
Member

Very odd, there is no user code explicitly utilizing LINQ in those tests. The stack trace of SendToRecvFromAsync_Datagram_UDP_UdpClient makes me think it might originate from async state machine enumerators:

   at System.Linq.Enumerable.WhereEnumerableIterator`1.Select[TResult](Func`2 selector) in /_/src/libraries/System.Linq/src/System/Linq/Where.cs:line 135
   at System.Linq.Enumerable.Select[TSource,TResult](IEnumerable`1 source, Func`2 selector) in /_/src/libraries/System.Linq/src/System/Linq/Select.cs:line 16
   at System.Net.Sockets.Tests.SendReceiveUdpClient.<>c__DisplayClass0_0.<<SendToRecvFromAsync_Datagram_UDP_UdpClient>b__0>d.MoveNext() in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceiveUdpClient.cs:line 47
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 82
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 49
   at System.Net.Sockets.Tests.SendReceiveUdpClient.SendToRecvFromAsync_Datagram_UDP_UdpClient(IPAddress loopbackAddress) in /_/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceiveUdpClient.cs:line 80
--- End of stack trace from previous location ---

So far happened only once, on one particular machine, but multiple tests failing simultaneously.

@mangod9
Copy link
Member

mangod9 commented Apr 19, 2021

if this is only on macOS arm64 there are some known stability issues. we are gradually fixing stress bugs.

@wfurt
Copy link
Member

wfurt commented Apr 20, 2021

The trace points to Equality assert. (if we can try it)
I have the hardware so I can try to reproduce it locally.

@antonfirsov
Copy link
Member

antonfirsov commented Apr 20, 2021

The trace points to Equality assert.

Honestly, I doubt this can come from within that assert. Line 474 is:

sendData and receiveBuffer are never null in this code, and I don't see why would Xunit use Select and Where for this comparison. The other stack trace #51480 (comment) is very suspicious.

@karelz karelz removed the untriaged New issue has not been triaged by the area owner label Apr 20, 2021
@karelz karelz added this to the 6.0.0 milestone Apr 20, 2021
@wfurt
Copy link
Member

wfurt commented Apr 21, 2021

XUnit uses Linq for IEnumerable https://github.com/xunit/assert.xunit/blob/main/CollectionAsserts.cs comparion.
effectively Enumerable.SequenceEqual.

Linq uses lot of trickery and possibly parallelism. We would need to probably look at the full core dump to see what is actually NULL. It may not be the buffers.

cc: @sdmaclea

@tmds
Copy link
Member

tmds commented Apr 29, 2021

Root cause is probably the same as for the NRE we're seeing on RHEL8 arm64: #43349.

@geoffkizer
Copy link
Contributor

These tests should all use AssertExtensions.SequenceEqual instead of Assert.Equal anyway.

As mentioned above, Assert.Equal does an IEnumerable-based sequence comparison here, which is much much slower than a span-based sequence comparsion (which AssertExtensions.SequenceEqual does).

@antonfirsov
Copy link
Member

Putting perf concerns aside, we are comparing two byte[]-s, which are never null, so NullReferenceException is very unexpected here. Assuming Xunit is stable doing that job, this can't be explained by anything but a runtime issue, #43349 confirms this with it's weird stack traces.

FYI there were no new occurrences since 18/04.

@sdmaclea
Copy link
Contributor

I have seen a significant Apple Silicon stability improvement with macOS 11.3. I have requested the M1 queues be updated to macOS 11.3. I would not be surprised if this resolves this issue completely.

@karelz
Copy link
Member

karelz commented Apr 30, 2021

@sdmaclea in that case we should close this issue until it happens again on the new version ...

@sdmaclea
Copy link
Contributor

We could do that or disable the test until the queues are updated. This PR could hold the disabled-test until we re-enable when https://github.com/dotnet/core-eng/issues/12776 is completed.

I will close now per your recommendation.

@ghost ghost locked as resolved and limited conversation to collaborators May 30, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

8 participants