-
Notifications
You must be signed in to change notification settings - Fork 118
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support atomic append on .NET 7+ #221
Comments
I have read everything in the links above. I see the pull request was open then closed then a new "Draft" style PR created and exists in the Done column of the System.IO - FileStream project the dotnet/runtime. @adamsitnik I don't understand what done means if no code is merged, could you shed some light on the progress, I just can't tell if atomic append is moving forward or not. This is a feature I am hoping for. I have a need to write to a single log file using Serilog's simple file sink on Linux. All files are rolled by a Linux utility called Logrotate by convention in deployed environment. Logrotate truncates the original file and as Serilog continues to write the file is typically prefixed with NUL bytes. Originally when I first noticed this I was not setting shared to true on the file sink. Mind you, I did not know about this current issue or Adam’s work on atomic appends. I just had this post from @nblumhardt and it was from 2016. So I set the flag and started experimenting by running two instances of an app appending to the same file for experiment. Looked good and was hoping it would play well with Logrotate on Linux. So, I deployed to my Linux machine and put a load on the service to produce logs that would roll on schedule. Ran for a few hours and things looked pretty good, no more NUL characters. But after 12 plus hours I did encounter the problem again and this time Serilog file sync just stopped logging and the last file rolled ended with NUL characters prefixed in the rolled file. So, no it did not work, My conclusion is to stop using the Linux Logrotate utility in the meantime. Just start using rolling file sink. But our convention is, all applications such as Java applications are to append to their own single file and as a convention let Logrotate handle rotation. .NET Core apps deployed to Linux are the new kids on the block and these behavioral differences make for a slow adoption. I am really looking forward to real atomic append solution. Thank you to @nblumhardt and @adamsitnik for all the hard work you put in and sharing. |
Another solution I've had suggested for Linux is to use a syslog sink and log to syslog server and let that handle the log files as appropriate. |
I have lots of logging duplication/overlap between the host (linux + logrotate) and the server app (aspnet + serilog). But this is a long-standing issue, so I assume it's complicated. In the meantime, does anyone have a workaround (i.e. to get serilog and logrotate to work together)? |
I'm using syslog now to handle writing to files and this syslog sink - https://github.com/IonxSolutions/serilog-sinks-syslog .NET then is not clashing with logrotate as it doesn't write direct to log files. Though there are limitations, such as requiring a syslog service or server running. Platform I'm using only has the basic syslog service running which doesn't support structured logging and the syslog sink doesn't support RenderedCompactJsonFormatter to format in NDJON messages for structured text logs (not sure if that would confuse serilog anyway). |
That's a pretty clever solution! Sadly in my environment we don't use it. Thanks for the good workaround. |
I am redirecting ">>" all output from the dotnet app to a file. |
Not out of the box, but formatters are pluggable! Just create your own implementation of |
@JoeShook That's quite smart. But when logging to console (and thus indirectly to file via bash redirection), it's human readable - don't you lose the structured logging? |
@lonix1, in my case I am not using structured logging. I am not sure how structured logging issues are different between using a file sink vs console sink. Using a console sink and redirecting to file via bash just solved the linux "logrotate" utility issue by removing dotnet non-atomic writes from the flow. |
@JoeShook If I remember correctly, the console logger doesn't do structured logging. So although it's an excellent workaround, there is a tradeoff - native log rotation vs structured logging. Thanks for your advice. |
I have failed to convince others on the .NET Team that dotnet/runtime#55465 (comment) @JoeShook please send a new proposal to dotnet/runtime (you can see my old proposal here: dotnet/runtime#53432) |
@adamsitnik That is an interesting problem. Maybe it could be solved with a new option, e.g. Also, I am now confused :) ... are you saying this feature in serilog is still impossible? |
I've tried the solution using private static FileStream CreateFile(string path, FileMode mode, FileSystemRights rights, FileShare share, int bufferSize, FileOptions options)
{
// FileSystemRights.AppendData sets the Win32 FILE_APPEND_DATA flag. On Linux this is O_APPEND
#if NET48
_fileOutput = new FileStream(path, mode, rights, share, bufferSize, options);
#else
// In .NET 7 for Windows it's exposed with FileSystemAclExtensions.Create API
if (System.Runtime.InteropServices.RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
var _fileOutput = FileSystemAclExtensions.Create(new FileInfo(path), mode, rights, share, bufferSize, options, new FileSecurity());
// Inherit ACL from container
var security = new FileSecurity();
security.SetAccessRuleProtection(false, false);
FileSystemAclExtensions.SetAccessControl(new FileInfo(path), security);
return _fileOutput;
}
else
{
throw new NotSupportedException();
}
#endif
} |
Thanks for the investigation, @bocca! It would be great to be able to support this 😎 |
This is available now via
FileSystemAclExtensions.Create()
, added in .NET Core 3.Some context in an earlier post: https://nblumhardt.com/2016/08/atomic-shared-log-file-writes/
Current status: dotnet/runtime#53432 (comment)
Thanks @adamsitnik for the nudge! :-)
The text was updated successfully, but these errors were encountered: