Enable CONFIG_DM_WRITECACHE module in kernel

[szasza576]

Describe the bug

Please compile the kernel with enabled CONFIG_DM_WRITECACHE module.
There are already several Device Mapper enabled in the kernel but his would be very helpful to speed up slow HDDs.
Enabling the module doesn't impact the default behaviour of the system hence it doesn't impact others.

Thank you in advance.

Steps to reproduce the behaviour

Try to load the module:
sudo modprobe dm-writecache

or

Try to create a cached LV:

homepi:~ $ sudo lvconvert --type writecache --cachevol ssd-cache data/lv-data
Erase all existing data on data/ssd-cache? [y/n]: y
  Failed to initialize logical volume data/ssd-cache at position 0 and size 4096.
  LV data/ssd-cache could not be zeroed.

Device (s)

Raspberry Pi 4 Mod. B

System

Raspberry Pi reference 2022-01-28
version c073992a577d160d8c30bfcf746d2208951c0537 (clean) (release) (start_cd)
5.15.36-v8+

Logs

No response

Additional context

https://github.com/raspberrypi/linux/blob/rpi-5.15.y/drivers/md/Kconfig#L332

[pelwell]

Am I write in thinking that the writes are cached on an SSD (since Pis have no persistent memory) until they are committed to the slow device?
Do you have any performance figures showing the speed-up that can be expected?

[szasza576]

Correct. There is an SSD in USB caddy. PI4 can use it up to ~350 MB/s write.
I also have some HDDs (also on the USB port).
3.5 inch PMR disk can go up to ~150 MB/s.
2.5 inch DM-SMR (which is very common) can go ~70 MB/s for the first 10GB then it falls to 2 MB/s (yeap 2 megabyte per sec).

I would use a 300GB SSD partition in front of the HDD drives. In this way I get a performant speed but still high capacity (and cheap). The writecache can do the flush during the night.
I tried the DM_CACHE with writeback policy (which is available BTW in RPIOS) but this doesn't speed all writes once the cache is full. The two cache solutions have different intention. The DM_CACHE is a hot cache based on promotions while the DM_WRITECACHE just focuses on caching writes and could nicely complete the SMR drives.

The PI's USB 3 speed matches the SSD speed and the extra CPU load isn't really noticable. So overall I see this a good combination to increase speed and eliminate IO bottlneck.

ps.: yes, I know this wears out the SSD but this is acceptable for those who uses cache to speed up.

[pelwell]

I've enabled DM_WRITECACHE as a module on Pi 4s - see 9bc1ec5.

[szasza576]

@pelwell Thank you for enabling the feature. The updated kernel arrived and I can serve some test data.

TL/DR: DM_WRITECACHE can give back the native speed of the SSD with the space of the HDD.

The feature in nutshell: DM_WRITECACHE as its name says helps in the writing the data. If you have a fast SSD and a slow HDD then you can bring the fast SSD on front of the HDD and all the writes will happen on the SSD first and then flushed to the slower HDD in the background. It has pros and cons so please do your research before using the feature. Here, I focus on how does it work on RPI 4. More details here: https://man7.org/linux/man-pages/man7/lvmcache.7.html

The cache works in this way:

• It is transparent so it looks you're writing to your slow device
• First your data is written to the SSD
• If the SSD utilization hits the "high_watermark" level then it starts the flushing to the HDD. (This also means that your data might be lost if it isn't flushed while the SSD dies. This is the disadvantage of this type of cache.)
• If the cache is full then all writes are done on your HDD directly (so the speed falls back to your HDD's speed). In this case the cache stops flushing to give all bandwidth to the user to write the data.
• Once there is no additional writes (the system in idle) then the cache starts flushing again until the utilization hits the "low_watermark" level.

In the attachment you can see the dstat printout where the sda is the SSD (Samsung SSD 870 EVO 500GB) on USB 3.0 port and the sdc is the HDD (Western Digital WD20SPZX) on USB 2.0* port.
dm-writecache_dstat.md

*The WD HDD is connected to the USB 2.0 port because this is a DM-SMR drive which has ~10GB PMR area and the rest is SMR. While it is writing to the PMR then it can write ~70-90 MB/s but once that area is full then the speed falls to 2-5 MB/s. This is how SMR disks work so the USB 2.0 port is fine for it.
*Speedtesting the SSD alone performs ~325 MB/s. (It is a bit behind of the USB-3 and the PCI theoretical bandwidth but I'm ok with this and it is in-line with other measurements on the internet.)

Note that the CPU utilization is mainly generated by the 2 dd copy threads.

So as you can see from the attachment DM_WRITECACHE can combine the raw performance of the different storage arrays smartly. We can reach 325 MB/s read bandwidth while there is free space in the cache.
Now you say that this is too nice to be true and you're right unfortunately :(
The speed is there, the CPU utilization is low but this cache is very memory demanding. The other cache solution, the DM_CACHE (which is a hot cache) requires minimal extra resource but the DM_WRITECACHE requires much more.
For 10GB cache it needs ~300MB RAM
For 100GB cache it needs ~2GB RAM

This is the weak point of the solution which is already noticed by the people of the internet but no solution yet:
https://access.redhat.com/solutions/6771021
https://bugzilla.redhat.com/show_bug.cgi?id=1855038

IMPORTANT, at the first try I created a 300GB cache and I attached it to my HDD's volume. Then the system failed with the activation due to short of memory. This setup required ~4,2 GB of free RAM which was more than challenging on a 4GB RPI model (SWAP doesn't play). This was the error message: device-mapper: reload ioctl on (253:0) failed: Cannot allocate memory

Ok, it didn't work so I tried to decommission the setup and tried to uncache the volume. Here comes the twist: to detach the cache first it tries to activate the volume to flush the cache first to preserve the data. Of course this fails because short of memory so the detachment fails as well. (Yeah, it was never used so the cache is empty but this is the process :) )
The only way of decommissioning was to delete the cached volume which deletes the cache AND the original data volume.
So before you start trying this feature do some pre-testing with empty volumes.

Nevertheless the solution works as it is documented (besides being a memory beast) and still can boost databases and similar IO intensive applications which require only few tens of GB cache.

[popcornmix]

@szasza576 okay to close?

[szasza576]

@popcornmix yes, it is working. Please close the ticket. Thank you.