[jitstress] HardwareIntrinsics_ro fails with "process cannot access the file" error #83298
Comments
dotnet-issue-labeler
bot
added
the
area-CodeGen-coreclr
|
Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak Issue DetailsThis seems to be failing widely in jitsress runs now: https://dev.azure.com/dnceng-public/public/_build/results?buildId=201133&view=ms.vss-test-web.build-test-results-tab
|
|
Tagging subscribers to this area: @hoyosjs Issue DetailsSeems to be failing widely. Build InformationBuild: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=201133 Error MessageFill the error message using known issues guidance. {
"ErrorMessage": "HardwareIntrinsics_",
"BuildRetry": false,
"ErrorPattern": "",
"ExcludeConsoleLog": false
}01:45:37.661 Passed test: global::JIT.HardwareIntrinsics.X86._Ssse3.Program.SignInt32()
01:45:37.661 Running test: JIT/HardwareIntrinsics/General/Vector256_1/Vector256_1_ro/Vector256_1_ro.cmd
App Exit Code: 259
Expected: 100
Actual: 259
END EXECUTION - FAILED
FAILED
Unhandled exception. System.IO.IOException: The process cannot access the file 'C:\h\w\AE5609E1\w\B7AD09EA\e\JIT\HardwareIntrinsics\HardwareIntrinsics_ro\HardwareIntrinsics_ro.testStats.csv' because it is being used by another process.
at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
at System.IO.StreamReader.ValidateArgsAndOpenPath(String path, Encoding encoding, Int32 bufferSize)
at System.IO.File.ReadLines(String path)
at XUnitLogChecker.Main(String[] args) in D:\a\_work\1\s\src\tests\Common\XUnitLogChecker\XUnitLogChecker.cs:line 77@ivdiazsa Could this be related to #78742?
|
jakobbotsch
added
blocking-clean-ci-optional
jakobbotsch
changed the title
|
@ivdiazsa could you please take a look if that was your PR, this broke many outerloop pipelines |
|
I just checked the failures and the Linux ones seem to be because of the test taking too long. Do we expect these tests to last more than 5 minutes? If yes, then that's an adjustment to make to the watcher. If not, then we can confirm the watcher is doing its job of preventing potential freezes. As for the Windows ones, I have no idea. Seems like potentially a problem with Windows liking to hold on to resources for whatever reason. However, it seems to be working on my machine so far. I will continue trying to repro it. |
|
I think that we have two different issues here:
Hope that helps Tomas |
|
I've been trying to repro this all day, but to no avail. I think @trylek's explanation makes total sense so what I will do next is to take his suggestion of making the watcher use that environment variable to determine the timeouts if set. As for the problem with actual timeouts:
Could you give more details regarding this suggestion Tomas? It sounds a reasonable way of fixing it. Do you mean like, have the log checker wait a couple seconds before executing on Windows? |
|
In the merged tests case, like HardwareIntrinsics_ro, all the tests run under a single invocation of corerun.exe, so it's not surprising at all that the total time would exceed 5 minutes.
That make sense to me: if the log read fails, then wait a few seconds, try again, wait a few seconds, try again. Maybe wait as much as a minute or more? |
Got it! Yes, I also think 1 minute of retrying every couple seconds or so to read the log is reasonable. If we still can't read it after that then maybe it's safe to assume something went terribly wrong that would require a developer to look at. |
|
@ivdiazsa - for debugging the locked csv file issue I bet it should suffice to set the watcher timeout to something ridiculous like 1 second, then you should be surely able to hit the timeout and forced app termination and I assume you'll immediately also see the locked file issue, I'm hitting it locally all the time on my Windows machine whenever I forcibly kill an app, all files it held on to remain locked for something like10~30 more seconds (subjective guess, I haven't actually measured it). |
|
For the directory deletion example I mentioned, this is how I implemented it in R2RTest (by porting my previous fix for .NET Native where it also used to crash lab runs about 8 years back): |
|
(It's a tad overcomplicated due to using the async logic to delete subtrees concurrently, that's something you thankfully don't need.) |
jakobbotsch
added
the
Known Build Error
|
Revert PR is up - it doesn't make sense to block others. |
|
I updated the issue - it was running against a pretty broad string. |
|
Closing after being addressed by PR #83427. |
ghost
locked as resolved and limited conversation to collaborators
Seems to be failing widely.
@ivdiazsa Could this be related to #78742? Could also likely be #83261.
Build Information
Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=201133
Build error leg or test failing: HardwareIntrinsics_ro.WorkItemExecution
Pull request: #83005
Error Message
Fill the error message using known issues guidance.
{ "ErrorMessage": "csv' because it is being used by another process", "ExcludeConsoleLog": false }Report
Summary
The text was updated successfully, but these errors were encountered: