l2arc_feed is constantly writing to the cache device

[ChromaxS]

System information

Type	Version/Name
Distribution Name	archlinux
Distribution Version	2021.01.01
Linux Kernel	5.4.63 and 5.10.10
Architecture	x86_64
ZFS Version	6fffc88 and fd95af8
SPL Version	n/a

Describe the problem you're observing

I'm seeing a behavior that I didn't notice until I rebooted for a kernel update. I switched between 5.4 to the 5.10 branch and updated to the latest ZFS HEAD: fd95af8

The cache device in my pool was being written to once per second at a single page: 4k. I booted up a VM and recreated this by just creating a new pool with a cache and observing it write once per second as well.

I dug through the commits from the build I was running from early March 2020, to the build I upgraded from at September 2020 and then to today, 2021-01-25, as they pertained to moudle/zfs/arc.c. I ripped out several commits that were parts of larger efforts, but was unable to get the constant writing to stop. From my test VM, the pool is created and then I immediately watch "zpool iostat" and have used other tools like iotop and iostat which confirm there is indeed a write occurring every second.

I'm sure this is undesirable behavior, if not a bug.

Describe how to reproduce the problem

NOTE: the a.raw, etc, are off by one due to the VM having a root volume for the archlinux install, so the raw files map to vdb+.

Formatting '/vol/kvm/user/archlinux-zfs/a.raw', fmt=raw size=107374182400
Formatting '/vol/kvm/user/archlinux-zfs/b.raw', fmt=raw size=107374182400
Formatting '/vol/kvm/user/archlinux-zfs/c.raw', fmt=raw size=107374182400
Formatting '/vol/kvm/user/archlinux-zfs/d.raw', fmt=raw size=107374182400
Formatting '/vol/kvm/user/archlinux-zfs/e.raw', fmt=raw size=107374182400
$ /opt/kvm.pl --kernel=/boot/linux-5.10.10-host+ archlinux

[root@zfs51010test ~]# uname -a
Linux zfs51010test 5.10.10-host+ #213 SMP Sun Jan 24 17:53:15 CST 2021 x86_64 GNU/Linux
[root@zfs5463test ~]# ls -la /usr/local/zfs
lrwxrwxrwx 1 root root 13 Jan 25 12:00 /usr/local/zfs -> zfs-fd95af8d
[root@zfs51010test ~]# zpool create -f vol /dev/vd[bcde] cache /dev/vdf
[  431.296776]  vdb: vdb1 vdb9
[  431.440321]  vdc: vdc1 vdc9
[  431.585677]  vdd: vdd1 vdd9
[  431.715964]  vde: vde1 vde9
[  431.857756]  vdf: vdf1 vdf9
[root@zfs51010test ~]# zpool iostat -v 1 | grep vdf
  vdf           0   100G      1      1  38.5K  7.35K
  vdf           0   100G      0      0      0    503
  vdf           0   100G      0      0      0    511
  vdf           0   100G      0      0      0    511
  vdf           0   100G      0      0      0    511
  vdf           0   100G      0      0      0    511
  vdf           0   100G      0      0      0    511
  vdf           0   100G      0      0      0    511
  vdf           0   100G      0      0      0    511
  vdf           0   100G      0      0      0    511
  vdf           0   100G      0      0      0    511
  vdf           0   100G      0      0      0    511
  vdf           0   100G      0      0      0    511
^C

The writes are occurring once every second, indefinitely. I tried this on my older kernel, 5.4.63, and the behavior is identical.

Include any warning/errors/backtraces from the system logs

None. This isn't a crash but instead runtime behavior.

[IvanVolosyuk]

I see similar writes to cache device, zfs 2.0.[01]. In my case zfs iostat reports 510 bytes written consistently every second interleaved into l2arc devices in two pool. Linux iostat confirms the writes actually happening.

[gamanakis]

I see what you mean. It may be because the header of the L2ARC device is updated constantly each time l2arc_write_buffers() runs, independently of whether we actually write buffers to L2ARC. This is by design because we need to keep track of the evict hand on the device, in case the user activates or uses trimming of L2ARC devices.

Let me try with a conditional checking if the evict hand has been updated in the header and update the device header only in this case.

[gamanakis]

I can confirm updating the header is the cause of this behavior. The linked PR resolves it.