A solution for podman containers max log size

[PavelKuzub]

Problem description

Today I ran out of disk space on UDM Pro's 12.2G storage area that negatively impacted UDM's original functionality.
Upon investigation, the disk was consumed by custom container logs (homebridge, hoobs) that reached 8GB+:

# Get container logs
ls -lah /mnt/data/podman/storage/overlay-containers/*/userdata/ctr.log

The catalyst of the log file to grow was an error printed by one of the Homebridge addons that went unnoticed for a while.

Our custom containers are set up without any logs rotation and the default setting is to grow unlimited.
It is a matter of time when each one of us using custom containers will get a disk full unless a solution is added to limit the max log size a container can have and thus maintain a stable disk usage footprint.

Research

As per documentation of podman and this issue, an ability to configure max log size per container was added in podman 2.2.0, while as of today, UDM Pro runs podman 1.6.1.

There was a reference to changing the setting for all containers via containers.conf file. This documentation covers log_size_max property.

However, podman 1.6.1 does not yet support containers.conf file - it was first mentioned in release notes of podman 1.9.0

I have investigated the source code of podman 1.6.1 and traced down the default setting to be coming from config file libpod.conf:

/etc/containers/libpod.conf

property:

max_log_size = -1

The content of this config file is reset on reboot, so an extra early on_boot.d script allowed to change default before the custom containers start.

# cat /mnt/data/on_boot.d/05-max_log_size.sh
#!/bin/sh
# Set a limit for container logs. 104857600 Bytes = 100 Megabytes
sed -i 's/max_log_size = -1/max_log_size = 104857600/g' /etc/containers/libpod.conf;

You can verify that the setting has been applied by looking for conmon parameters

ps -ef | grep conmon | grep log-size-max

I have verified that the log is getting truncated by setting up a limit of 10 kilobytes. It is not a rotation, but a truncation.
unifi-os container does not write anything in the log, so UniFi is not suffering from the lack of log size limits. Only custom additional containers.

Proposal

Set a limit to the containers log size.

Would the above be a strategic approach worth a Pull Request?
If so, what would be the right place in this repo to place a generic script like this that impacts all containers?

[boostchicken]

Hey,

Thanks and excellent write up. I would put this in own folder called container-common or something like DNS common. Also, make sure to update the README.md to describe it and maybe even highlight it in red and update other README's if needed. If you can get the script in, I can help with that as well.

Thanks!

[PavelKuzub]

Hello John,

I have raised a PR #102 with proposed changes. Let me know if any additional commits are required or wanted. Thanks

PK

[boostchicken]

Merged!@