OVS leak log file when rotating logs

[antoninbas]

Describe the bug
One of our users reported an issue with log rotation in Antrea. I tracked it down to an issue in Open vSwitch. When using log rotation with OVS (OVS ships with configuration files for logrotate for multiple distributions), if monitoring is used for the ovs-vswitchd and ovsdb-server daemons, the first log file is "leaked". Antrea relies on logrotate to rotate the OVS log files and is affected by this issue (#1329).

To Reproduce
Steps to reproduce the behavior. The steps below were executed by exec'ing into an Antrea Agent Pod (antrea-ovs container). We force a log rotation by increasing the log file size artificially and running logrotate manually, but the same situation will arise after a while in Antrea (OVS generates logs when a Pod is created / deleted, and Antrea runs logrotate itself every hour).

1. Increase the size of the /var/log/openvswitch/ovs-vswitchd.log log file artificially

root@k8s-node-worker-1:/# dd if=/dev/zero of=/var/log/openvswitch/ovs-vswitchd.log bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.30647 s, 822 MB/s
root@k8s-node-worker-1:/# ls /var/log/openvswitch/ovs-vswitchd.log -l
-rw-r----- 1 root root 1073741824 Mar 27 01:46 /var/log/openvswitch/ovs-vswitchd.log

2. Run logrotate

logrotate /etc/logrotate.d/openvswitch-switch

3. Check that the log file was rotated

root@k8s-node-worker-1:/# ls -l /var/log/openvswitch/ovs-vswitchd*
-rw-r----- 1 root adm       95 Mar 27 01:47 /var/log/openvswitch/ovs-vswitchd.log
-rw-r----- 1 root root 1042069 Mar 27 01:46 /var/log/openvswitch/ovs-vswitchd.log.1.gz

4. Check for the file leak

root@k8s-node-worker-1:/# lsof | grep ovs-vswitchd.log
monitor     96               root    7w      REG                8,1 1073741824 521155 /var/log/openvswitch/ovs-vswitchd.log.1 (deleted)
ovs-vswit   97               root   48w      REG                8,1         95 521206 /var/log/openvswitch/ovs-vswitchd.log
ovs-vswit   97 109 handler1  root   48w      REG                8,1         95 521206 /var/log/openvswitch/ovs-vswitchd.log
ovs-vswit   97 110 revalidat root   48w      REG                8,1         95 521206 /var/log/openvswitch/ovs-vswitchd.log
ovs-vswit   97 111 urcu2     root   48w      REG                8,1         95 521206 /var/log/openvswitch/ovs-vswitchd.log

You can see that the 1GB file is still present on disk, because the monitor process for ovs-vswitchd still has a reference to the file.

root          96  0.0  0.1  13516  2640 ?        S<s  01:43   0:00 ovs-vswitchd: monitoring pid 97 (healthy)

Actual behavior
logrotate moves the log file from ovs-vswitchd.log to ovs-vswitchd.log.1, then compresses ovs-vswitchd.log.1 and tells ovs-vswitch to re-open its log file (using ovs-appctl vlog/reopen). However, only the ovs-vswitchd process is notified over the control socket, the monitoring process is NOT notified. Therefore the monitoring process keeps referring to the old file (the file was moved but the inode is the same), which can never be deleted.

Expected
The monitoring daemon should be notified by ovs-appctl and should re-open its log file.

Versions:
Antrea: v0.13.1, v0.14.0-dev

[jianjuns]

So, this should be a fix in OVS monitoring script?

[antoninbas]

@jianjuns yes, this needs to be fixed in OVS. I cannot think of any workaround we can do in Antrea: the monitor processes need to re-open their log files (like the main OVS daemons), otherwise the file can never be deleted by the OS. Fortunately that's only a 100MB "leak" with the default log rotation settings. It also means that the logging for the monitor processes is broken (they don't log to the right file after the first rotation), but that part doesn't really matter. I believe the OVS folks are looking into this already.

[github-actions]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment, or this will be closed in 180 days

[alex-vmw]

/remove-lifecycle stale

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

[antoninbas]

This has been addressed on the master branch of OVS: openvswitch/ovs@78ff396. We will include the patch in the OVS version that we ship with Antrea v1.7.0.