DietPi-Software | Samba: Move logging to RAM

[WolfganP]

Required Information:

• DietPi version | 159 (yes, I know, old version... planning to update soon)
• Distro version | 9.6
• Kernel version | Linux DietPi 4.9.62-v7+ Fix for dumb or unset TERM #2 SMP Fri Nov 17 23:52:26 GMT 2017 armv7l GNU/Linux
• SBC device | RPi2
• Power supply used | 5V 2A
• SDcard used | (EG: SanDisk ultra)

Steps to reproduce:

So, as my RasPi acts as a 24/7 media / file / IoT server, I started a mission to reduce power usage (ie disconnect HDD while media/file server is not needed) and avoid unnecessary write ops to the SD card to improve the component life and system stability.

So, using fnotifystat I was relocating files to tmpfs / ext HDD and dynamically taking down unneeded processes, but there are a couple of functions that still keep writing to the SD card unnecessarily:

pi@DietPi:~$ sudo fnotifystat -vnT -x /mnt,/proc,/dev,/run,/tmp,/var/log | grep W\)

05/01/19 12:10:47   0.0  (-C-W)  1330 nmbd            /var/cache/samba/browse.dat.
05/01/19 12:11:07   0.0  (-C-W)  1330 nmbd            /var/cache/samba/browse.dat.
05/01/19 12:17:03   0.0  (---W)  4380 date            /etc/fake-hwclock.data
05/01/19 12:17:03   0.0  (-C-W)  4378 fake-hwclock    /etc/fake-hwclock.data
05/01/19 12:22:50   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat
05/01/19 12:23:10   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat
05/01/19 12:27:36   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat (deleted)
05/01/19 12:27:36   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat
05/01/19 12:34:52   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat (deleted)
05/01/19 12:34:52   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat
05/01/19 12:39:34   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat
05/01/19 12:39:34   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat
05/01/19 12:46:52   0.0  (-C-W)  1330 nmbd            /var/cache/samba/browse.dat.
05/01/19 12:46:52   0.0  (-C-W)  1330 nmbd            /var/cache/samba/browse.dat.
05/01/19 12:51:32   0.0  (-C-W)  1330 nmbd            /var/cache/samba/browse.dat.
05/01/19 12:51:32   0.0  (-C-W)  1330 nmbd            /var/cache/samba/browse.dat.
05/01/19 12:58:52   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat (deleted)
05/01/19 12:58:52   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat
05/01/19 13:03:33   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat
05/01/19 13:03:53   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat
05/01/19 13:10:49   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat
05/01/19 13:11:09   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat
05/01/19 13:15:32   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat
05/01/19 13:15:52   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat
05/01/19 13:17:03   0.0  (---W)  6060 date            /etc/fake-hwclock.data
05/01/19 13:17:03   0.0  (-C-W)  6058 fake-hwclock    /etc/fake-hwclock.data
05/01/19 13:22:51   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat (deleted)
05/01/19 13:22:51   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat
05/01/19 13:34:53   0.0  (OC-W)  1330 nmbd            /var/cache/samba/browse.dat
05/01/19 13:35:05   0.0  (-C-W)  1330 nmbd            /var/cache/samba/browse.dat.

For /var/cache/samba/browse.dat (rewritten every 5' or so) I found this patch at https://lists.samba.org/archive/samba-technical/2015-May/107144.html and I'm still investigating the issue (I'll report back if found more info / a solution)

For /etc/fake-hwclock.data being re-written regularly (every hour), I found https://github.com/Fourdee/DietPi/issues/2041 relevant, but all the discussion centers on needing it at boot time because the clock is not yet updated by ntp.
So, is there any way to avoid that file write after the system is completely up and running?

Thx as always for any tip.

[WolfganP]

Just for the sake of documentation in one single place, the write ops to the sdcard can be checked via awk '/mmc/ {print $3"\t"$10}' /proc/diskstats as per https://www.kernel.org/doc/Documentation/iostats.txt

For the /var/cache/samba/browse.dat case, I found this https://ubuntuforums.org/showthread.php?t=2166829 relevant and tested it succesfully. Converted to systemd specs, this can be easily automated as:

Just needed Once in system prep:

sudo mv /var/cache/samba /var/cache/samba.bak
sudo mkdir /var/run/samba-cache
sudo ln -s /var/run/samba-cache /var/cache/samba

Then in every boot, the symlink creation is automated via:

$ sudo systemctl edit nmbd

[Service]
ExecStartPre=/bin/sh -c "/usr/bin/test -x /var/run/samba-cache || /bin/mkdir /var/run/samba-cache"

In order to allow smbd and nmbd services to be started in any order later on, I repeated the folder creation (if needed) for smbd unit:

$ sudo systemctl edit smbd

[Service]
ExecStartPre=/bin/sh -c "/usr/bin/test -x /var/run/samba-cache || /bin/mkdir /var/run/samba-cache"

I'll keep adding info if any additional findings.

[MichaIng]

@WolfganP
Thanks for opening this issue.

fnotifystat looks like a very interesting tool to improve any system according to disk I/O.

/var/cache/samba/browse.dat

• As long as there is no fix (only write that file if actual values have changed) available, you can surely link that file to RAM, e.g.:

mv /var/cache/samba/browse.dat /tmp/samba-browse.dat
ln -s /var/cache/samba/browse.dat /tmp/samba-browse.dat

• Just assure (although I am pretty sure) that:
  • a) It is really no issue that the content is lost on reboot
  • b) The file itself is re-created, if non-existent
• In case do a backup of it first, if you face any issues.

€: Ah you already found a similar solution, about this I would recommend:

• Use /var/run/nmbd for symlink target, since this should be created/available automatically, when nmbd service starts. So you don't need to pre-create some additional dir. Although: The dir might be used for other things, not just browse.dat? In this case, since /var/cache generally is meant to be boot-persistent, to symlink only browse.dat itself, not the whole dir, as my suggestion above: ln -s /var/cache/samba/browse.dat /var/run/nmbd/ to link it to run dir of the service that writes it.

/etc/fake-hwclock.data

• Indeed this one should stay in place, otherwise on early boot (before network time sync) you will have 1970th system time+timestamps, which might cause further issues down the road, especially if then network time sync fails for some reason. RPi has no own hardware clock 😉.
• But: fake-hwclock updates the file very late on shutdown to have the most current time on startup. The hourly update via /etc/cron.hourly/fake-hwclock is just a failsafe, if the system crashes (e.g. power loss), so no ordinary shutdown with fake-hwclock update is done. In theory you can remove the cron job or have it run e.g. just once a day.

[WolfganP]

Thx @MichaIng , we were writing at the same time with the findings. See my previous reply with a solution for /var/cache/samba/browse.dat

Regarding /etc/fake-hwclock.data I'll adopt your suggestion, and reschedule it to run more spaced, ie once a day, or every 6 hours, or ...

[MichaIng]

@WolfganP
Jap, I updated my post accordingly, see my suggestion about linking only the file, not the whole folder, and link target. But both solutions should work fine.

To have fake-hwclock running daily is pretty simply:
mv /etc/cron.hourly/fake-hwclock /etc/cron.daily/fake-hwclock

To have it run every 6 hours you need to create a custom crontab entry or systemd timer.

[WolfganP]

Jap, I updated my post accordingly, see my suggestion about linking only the file, not the whole folder, and link target. But both solutions should work fine.

I tested linking just the /var/cache/samba/browse.dat file but it didn't work. nmbd deletes the symlink and recreates the file on the folder at start; no issues with linking the folder thou. I'll keep investigating.

[Fourdee]

@WolfganP

Many thanks for the info and resolution! 👍

Ok, we could move the whole sub directory to RAM:

root@Fuzon:~# ls -lha /var/cache/samba/
total 536K
drwxr-xr-x 3 root root 4.0K Jan  5 10:06 .
drwxr-xr-x 9 root root 4.0K Dec  9 07:40 ..
-rw-r--r-- 1 root root  154 Jan  5 10:06 browse.dat
-rw-r--r-- 1 root root 520K Jan  5 17:45 gencache.tdb
drwxr-xr-x 2 root root 4.0K Jul 21 22:09 printing

/var/cache is intended for cached data from applications. Such data is locally generated as a result of time-consuming I/O or calculation. The application must be able to regenerate or restore the data. Unlike /var/spool , the cached files can be deleted without data loss.
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s05.html

Actually, might even be able to move everything in /var/cache to RAM, if we can exclude APT?

root@Fuzon:~# G_TREESIZE /var/cache
[  OK  ] Root access verified.
116.1 MB /var/cache
88.9 MB /var/cache/apt
21.4 MB /var/cache/netdata
4.9 MB /var/cache/debconf
560.0 KB /var/cache/samba
152.0 KB /var/cache/fontconfig
72.0 KB /var/cache/lighttpd
32.0 KB /var/cache/ldconfig

[MichaIng]

Actually, might even be able to move everything in /var/cache to RAM, if we can exclude APT?

The cache dir of APT can be moved without symlink:

2019-01-05 19:22:32 root@micha:/tmp# cat /etc/apt/apt.conf.d/99-micha-apt_cache
#Dir "/";
Dir::Cache "tmp/apt";
#Dir::Cache::archives "archives/";
#Dir::Cache::srcpkgcache "";
#Dir::Cache::pkgcache "pkgcache.bin";

• Source package cache can be disabled completely, if not required, but I am not sure if this is then re-downloaded on every APT update, so I left it in place in my case.

But yeah the possible issue is the size, especially when many downloaded archives are cached:

2019-01-05 20:25:58 root@micha:/tmp# G_TREESIZE apt
59.1 MB apt
0.0 KB apt/archives

So this can be only done, if it's verified that enough free RAM is available and still the risk remains, that downloading some very large APT package(s) might fail, when installing large packages or such, that require much dependencies.

In general:

• Even that packages should be able to recreate /var/cache content, during my research some time ago, there are many, which do no handle this properly.
• And we do not know, if there are other packages that store large amount of data there, like APT does.
• So I think we should leave /var/cache where it is and only allow/by default link sub dirs to RAM, where we know it works and the size will always be small.
• What we could offer instead, is linking it to dietpi_userdata, which should be moved to external drive, if available anyway, thus reduces SDcard I/O. But should be optional since users might complain external HDD constantly spinning or reviving from idle too regularly.

[Fourdee]

🈯️ Fourdee@bff17ab

In /etc/default/fake-hwclock you can set:

FILE=/tmp/fake-hwclock.dat

root@DietPi:~# ls -lha /tmp
total 8.0K
drwxrwxrwt  7 root root    180 Jan  5 19:42 .
drwxr-xr-x 22 root root   4.0K Oct 30 15:26 ..
-rw-r--r--  1 root root     20 Jan  5 19:42 fake-hwclock.dat

G_CONFIG_INJECT 'FILE=' 'FILE=/tmp/fake-hwclock.dat' /etc/default/fake-hwclock
systemctl restart fake-hwclock

[MichaIng]

@Fourdee

FILE=/tmp/fake-hwclock.dat

This does not work, since the file must be reboot persistent to do it's job. Even external drive is dangerous, if it is required before external drives are fully mounted, and of course, if the drive got detached. So this must stay on RootFS and I would also not touch the hourly safe to better prevent issues in case of power loss. Leave this to end user custom change, if really desired. But this file is tiny, and hourly disk touch should be really no issue, besides you are 105% enthusiast 😉.

[WolfganP]

I agree on leaving /etc/fake-hwclock.dat as it is, due it's persistance nature being its reason to be.
OK with leaving the "manual optimization" trade off decision to the advance user regarding it's update frequency.

As I said earlier, my motivation was to avoid SDcard excessive wearing for 24/7 servers when it can be avoided, and /var/cache/samba/browse.dat was the biggest offender as it updates every 5' or so unnecessarily. The relocation of folder mods above tested safe in my setup.

[Fourdee]

@MichaIng

This does not work, since the file must be reboot persistent to do it's job.

Ah good point 👍 we can do a service to save this to disk during shutdown and restore during boot.

[MichaIng]

@Fourdee

Ah good point 👍 we can do a service to save this to disk during shutdown and restore during boot.

Simply removing the cron job /etc/cron.hourly/fake-hwclock has same result: 1970th system clock, if system crashes due to power loss or other critical issue.
fake-hwclock creates/writes to this file only on late shutdown stage itself, while the cron job is to prevent urgently outdated system clock in case of unordered shutdown.

[Fourdee]

@MichaIng

/etc/cron.hourly/fake-hwclock

(Fourdee@3c34712) Yep, so only issue now is if power loss, time will be reset to last boot time saved and/or from last shutdown save.

Shame /etc/cron.hourly/fake-hwclock doesnt simply run systemctl restart fake-hwclock

[MichaIng]

@Fourdee

Shame /etc/cron.hourly/fake-hwclock doesnt simply run systemctl restart fake-hwclock

? Would be just overload, since the service does nothing else (on stop) then what the cron job does?

As said, the current solution is a workaround for a result that we can achieve by simply removing the cron job, which will be (due to our apt.conf.d settings) not automatically re-created on APT upgrade.

But I still do not like the workaround/solution. This is a very tiny file touch once an hour that prevents potentially large issues when the system starts with outdated clock due to crash/power loss or simply because SSH connection does not work on HDMI-less SBCs and users are forced to unplug power cable.

I vote to revert all changes about fake-hwclock here. This is really no issue, but, as it is by default, a meaningful implementation.

[WolfganP]

Totally agreed with @MichaIng on /etc/fake-hwclock.dat case . Leave it as it is and if some advanced user wants to manually change the balance on wearing vs reboot stability, it's all documented in this issue.

The samba solution however, is transparent to the user and bigger offender in terms of SDcard writing, so it may be worthy to tackle for all users.

[Fourdee]

Hmm, fake-hwclock starts before tmp.mount. Changing order of this will give incorrect times during boot.

@WolfganP @MichaIng

Agree 👍, will revert.

[MichaIng]

@Fourdee
Yes, it makes sense that it starts as early as ever possible, also to have all following boot logs with correct time stamps and of course if something else depends on moreless correct system time.

Having another look onto the FORCE option: https://manpages.debian.org/stretch/fake-hwclock/fake-hwclock.8.en.html#COMMANDS
Actually from my point of view, besides for testing reasons, it doesn't make much sense to set this true? E.g. systemd already does an earlier check and sets the system clock to it's own release date (in case no real hwclock available):
[ 1.443606] systemd[1]: System time before build time, advancing clock.
If we now ship an image, which was created before systemd release, with FORCE=force, fake-hwclock would set the time further into the past.
And of course if somehow before shutdown (and after last cron job update) the system clock got out of order, it would safe an older time.
All this should be relevant only in very rare cases, but at least I see no reason to actively set the FORCE option, but simply leave it as it is by default.
Or did some reason for this appear in the past?

[MichaIng]

Okay about the samba cache:

• So this cache dir is used by different processes/services right? So we cannot bin the tmp dir creation to only one service.

Me by chance found the tmpfile systemd feature: https://manpages.debian.org/stretch/systemd/tmpfiles.d.5.en.html

• Used by systemd/packages: /usr/lib/tmpfiles.d
• To be used by admin: /etc/tmpfiles.d

So the following should work well, at best BEFORE samba APT install:

# Remove disk cache dir or previous symlink
[[ -e /var/cache/samba ]] && rm -R /var/cache/samba
# Pre-create RAM cache dir
mkdir -p /var/run/samba-cache
# Link disk cache to RAM
ln -s /var/run/samba-cache /var/cache/samba
# Create RAM cache dir automatically on boot
echo 'd /var/run/samba-cache - - - - -' > /etc/tmpfiles.d/dietpi-samba_cache.conf

Actually on Debian, /var/run is just a symlink to /run, but for whatever old-fashioned reason, nearly all services still define /var/run for their runtime directories and tmp files, also the systemd ones, that should know better. So I think we should follow this, who knows when such might change in the feature.

---

PR: https://github.com/Fourdee/DietPi/pull/2397

@WolfganP
Okay PR is done and tested throughout, although not with active samba mount. But the result is exactly the same as in your case, only with a different tmp dir creation method.

So, would be great if you could report back, if there are any issues, since samba looses it's cache on every reboot. It seems to be really just a "cache", but a real-world test is always great to be sure 🙂.

[Fourdee]

• Existing install patch + test
• 🈯️ New install on v6.20 + test