Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Race condition in FileBodyStore #393

Open
gingters opened this issue Nov 16, 2022 · 1 comment
Open

Race condition in FileBodyStore #393

gingters opened this issue Nov 16, 2022 · 1 comment
Assignees

Comments

@gingters
Copy link
Member

We get an unhandled(!) exception, when the file body store tries to remove a response body file, while this file / stream is still in use (i.e. framework did not yet dispose the stream after sending out the content to the client).

Seems to be a race condition.

Proposed solution:
a) handle the exception. We should not have this unhandled.
b) if we get the IOException, log this as INFO only. Then try to Task.Delay by a few seconds and retry the deletion. If that also fails, then log as ERROR, but still have it handled. In this case we end up with a file that might clog the file storage, but at least we have logged this.

Here the exception with stack trace:

System.IO.IOException: The process cannot access the file 'C:\Users\SebastianGingter\AppData\Local\Temp\res_c6cc96b9-864c-46f8-837d-4ecc105b4886' because it is being used by another process.
   at System.IO.FileSystem.DeleteFile(String fullPath)
   at System.IO.File.Delete(String path)
   at Thinktecture.Relay.Server.Transport.FileBodyStore.RemoveResponseBodyAsync(Guid requestId, CancellationToken cancellationToken) in C:\Dev\tt\relayserver3\src\Thinktecture.Relay.Server\Transport\FileBodyStore.cs:line 142
   at Thinktecture.Relay.Server.Transport.FileBodyStore.<>c__DisplayClass13_0.<GetResponseBodyRemoveDisposable>b__0() in C:\Dev\tt\relayserver3\src\Thinktecture.Relay.Server\Transport\FileBodyStore.cs:line 158
   at Thinktecture.Relay.Server.DisposeAction.DisposeAsync() in C:\Dev\tt\relayserver3\src\Thinktecture.Relay.Server.Abstractions\DisposeAction.cs:line 22
   at Thinktecture.Relay.Server.Middleware.RelayContext`2.DisposeAsync() in C:\Dev\tt\relayserver3\src\Thinktecture.Relay.Server\Middleware\RelayContext.cs:line 68
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.<DisposeAsync>g__Await|22_0(Int32 i, ValueTask vt, List`1 toDispose)
   at Microsoft.AspNetCore.Http.Features.RequestServicesFeature.<DisposeAsync>g__Awaited|9_0(RequestServicesFeature servicesFeature, ValueTask vt)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.<FireOnCompleted>g__ProcessEvents|227_0(HttpProtocol protocol, Stack`1 events)
@gingters gingters added the bug label Nov 16, 2022
@gingters gingters added this to the 3.0.0-alpha.1 milestone Nov 16, 2022
@gingters gingters self-assigned this Nov 16, 2022
@gingters
Copy link
Member Author

Okay, further investigaten / debugging showed:
This does not happen when running in docker (default and suggested way of running RelayServer v3), but only when running on Windows natively (not recommended). So this is a low-prio problem, if at all.

Also, on Windows, the file is locked by the hosting process for up to several minutes after the request is long finished.
Due to the long locking time, a simple retry after a few seconds doesn't do the trick.

Further investigation required: Why is the file locked so long on Windows?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants