Rspamd stats are sometimes reset across reboots

[hardware]

Classification :

• Data loss
• WebUI/Usability
• Serious bug
• Other bug
• Feature
• Enhancement

Reproducibility :

• Always
• Sometimes
• Rarely
• Unable
• I didn’t try
• Not applicable

Rspamd version:

1.6.3

Operation system, CPU, memory and environment:

OS: Debian GNU/Linux 9 (stretch)
Kernel: 4.9.0-3-amd64
Total Memory: 866.2MiB
Docker: 17.06.1-ce

Description :

There is a bug in Rspamd which causes the WebUI counters are not saved on the filesystem when rspamd stop (maybe when it terminated too quickly or unexpectedly). It seems that stats.ucl content is never updating during rspamd execution.

root@mail:/var/mail/rspamd# ls -la
total 792
drwxr-x--- 2 _rspamd _rspamd   4096 Sep  3 20:19 .
drwxr-xr-x 9 vmail   vmail     4096 Aug 21 22:55 ..
-rw-r--r-- 1 _rspamd _rspamd 797696 Sep  4 06:51 rspamd.rrd   <-  4 sep = last start
srw------- 1 root    root         0 Sep  3 20:19 rspamd.sock
-rw-r--r-- 1 _rspamd _rspamd    171 Aug 29 16:07 stats.ucl    <- 29 aug = previous start

In my setup, DDBDIR=/var/mail/rspamd : https://github.com/hardware/debian-mail-overlay/blob/7765c687c342224c3a3789265c13523cb58bb487/Dockerfile#L91

Steps to reproduce:

Restart rspamd.

Expected results:

Have an up to date version of the stats.ucl on the filesystem.

Actual results:

Sometimes, WebUI counters are reset with old values.

Configuration:

/etc/rspamd : https://github.com/hardware/mailserver/tree/master/rootfs/etc/rspamd

[hardware]

Maybe related to @vthriller's issue : #1773

[ChessSpider]

+1
I experience the same thing.. I use mailcow but i checked and they seem to persist everything required.

[ksylvan]

Is this changed at all in 1.6.4?

[ksylvan]

Answering my own question above: No, it's not fixed in 1.6.4. ;-) Anything we can do to help pinpoint the issue?

[hardware]

This issue has been fixed with hardware/mailserver#181 pull request, thanks to @pjeby

The container main process exited without first terminating rspamd properly, which in rspamd's case meant it sometimes didn't saved its stats. Now the finish script shutdown rspamd and waits for the service to stop completely before docker stops the container.

[moisseev]

@vstakhov It would be nice if rspamd could save stats periodically to avoid stats loss when rspamd terminated unexpectedly.

[vstakhov]

But it will still loose stats in this case. So what's the purpose of such a periodic saving?

[moisseev]

It will lose just stats since last saving. Depending on saving interval it could be insignificant (or far less significant) amount of data. Especially for very long uptimes.

[pjeby]

Probably saving should only happen when the stats have changed since the last save; otherwise saving every minute is just needless writing to disk. For that matter, if they're only saved when changed, then the interval can be much more frequent than once per minute without wasting disk writes.

[vstakhov]

What problem are you trying to solve? Saving ~100 bytes of disk writes per minute?

[pjeby]

No, of decreasing the stats save interval. If it's only 100 bytes, might as well write it every time the stats change, then, or at least at a more frequent interval, conditional on the stats having changed.

(TBH, I know next to nothing about rspamd or the specifics of this issue; I was just fixing the bug in hardware/mailserver that caused the stats to be lost, but I'm new to the specific significance of the stats themselves and what it means to have or not have them up to date. So I will stop adding my ignorance now. 😄)