-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Some JSON test suites crash in ConsumeStringAndValidate on Red Hat CI machines #41582
Comments
The code paths cited in the description have not been modified since 3.1, so I don't believe this is a regression in 5.0. Will triage as a 6.0 issue. @tmds when did you start running these tests? You mention Aug 6th, but I don't see any commits in System.Text.Json around that date that would have triggered new failures. Also, are the stack traces identical in each failure case? Are you able to point me to any actual CI logs/artifacts? |
I debugged the issue and traced it down to this function returning an invalid index: runtime/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs Lines 1275 to 1279 in 85190a3
This is performing a vectorized search, and @mikem8361 to debug this I have configured out CI machine to capture coredumps and the testhost. When I pass the testhost
On the CI machine, the testhost lives at Can I make it work without putting the testhost at the same spot? |
Is the testhost called “dotnet”? Just checking. As far as I know putting the full path to the host program on the command should work no matter where it is. It shouldn’t have to be in the same location as the target machine. The other reason for the SOS error is that lldb needs the matching version of the libcoreclr.so file. Putting in the same directory as the dump or the host should work.
The way to tell (you may already know this), if you got the right host in the right location is that `target modules list` displays the full set of modules (including libcoreclr.so). The way you can tell that lldb picked up libcoreclr.so in that module list it doesn’t have the pointer after the name:
`/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.12/libcoreclr.so(0x00007fea7157a000)`
I hope that helps.
|
@tmds it looks like this is caused by the other issue you opened - #41584. Can we close this issue and reopen it if the solution there doesn't fix this issue? cc @tannergooding |
Also cc @benaadams who has been working in this space recently: |
Yes, this is the command line that gets executed:
That doesn't work for me:
This doesn't work for me either:
I can only make it work when I put the testhost folder at the exact same location on my development machine as where it was on the CI machine (
This is the output of
|
#41097 changes the method |
I don’t understand why giving the full path to the host program isn’t working. I’ll have to investigate a little more. I’ve always put the host (dotnet) in the same directory as the dump (I use dotnet-symbol to get it usually).
The reason clrstack isn’t working is that lldb also needs the libcoreclr.so binary:
```
[ 12] A217988E /home/tester/runtime/artifacts/bin/testhost/net5.0-Linux-Debug-x64/shared/Microsoft.NETCore.App/6.0.0/libcoreclr.so `(0x00007f1dc795d000)`
```
Notice the (0x00007f1dc795d000) at the end. That is lldb’s way of saying it needs the actual binary. Even though in is in the module list, SOS can’t get the info from it to find/download the DAC.
Putting dotnet and libcoreclr.so in the same directory as the dump seems to work for me in those cases. Overall, lldb on core dumps is really fussy and you have to bend over backwards to get everything right. I tried to doc some of these issues [here]( https://github.com/dotnet/diagnostics/blob/master/documentation/FAQ.md), but it may not be complete. You can always use `dotnet-dump analyze`. It is easier to get working in these cases.
On top of all that you will probably need to copy the DAC (libmscordaccore.so) somewhere (the same directory as the dump) and use the `setclrpath` command to point to that directory.
|
@mikem8361 thanks for the additional info and FAQ link.
I'll close this now. |
We do a daily build+test run of dotnet/runtime on Fedora 32 and RHEL8. Since Aug 6th, a number of tests are crashing in
ConsumeStringAndValidate
Example stack trace:
When I run on my development machine, the tests pass without crashing.
cc @omajid @RheaAyase
The text was updated successfully, but these errors were encountered: