Improve --log-size-max handling

[rhatdan]

Currently conmon just truncates the log file when max is reached, this means we loose all of the data.

I think we should allow for a backup file. So when log-size-max is reached we rename the existing file to a .1 version. Now when the user reads the full log file they would read the .1 file first and then the main log file. This would mean we would always have log-size-max data available, after we fill it up. If .1 already existed, we would loose that data.

Since user asked for log-size-max, we should probably do the rename at log-size-max/2 to grant their wishes.

@haircommander @giuseppe @vrothberg @mheon WDYT?

[rhatdan]

Docker also handles an addition field to indicate the number of backup files, I believe.

[vrothberg]

Docker also handles an addition field to indicate the number of backup files, I believe.

That answers the immediate question I had in mind: How do we set a limit to protect from running out of disk space?

SGTM

[haircommander]

the approach sounds good, as long as the behaviour is gated by a flag. In the k8s world, kubelet handles the renaming of the files, so we should preserve existing behaviour unless otherwise specified

[haircommander]

another thing to think about is how to handle the metadata. does docker truncate based on file size or based on size of log line? regardless, we'll have differing behaviour until conmon actually supports json-file

[exekias]

++ on this issue, we (Filebeat dev here) are seeing more issues coming from k8s users using alternative runtimes, like CRI-o. Having truncate as the mechanism for rotation makes it impossible to guarantee delivery of all log messages. A rename + create approach like the one describe in this issue allows agents to keep reading both the old and the new file after a rotation event.

does docker truncate based on file size or based on size of log line?

Docker does this based on size, which probably makes sense, as it allows system administrators to allocate disk for logs in a predictable way, regardless of log line lengths.

Also a clarification about Docker rotation strategy: Docker doesn't truncate files, this is important. It will rename the existing log file and create a new one for the following logs.