state.db-wal grown too large

[VladoPortos]

Any idea what is in state.db-wal ?
It grown to 8+GB within a month or so... is it something that can be cleaned up, or how can I find out why it is filling up so much ?

[clook]

I had the same issue (16 GB file, leading to pod eviction).
I could vacuum the following way without problem (not sure how safe it is though):

systemctl stop k3s
sqlite3 /var/lib/rancher/k3s/server/db/state.db
sqlite> vacuum;
sqlite> .quit
# The file  /var/lib/rancher/k3s/server/db/state.db-wal is not there anymore, k3s kine sqlite driver will recreate it.
systemctl start k3s

If the vacuum has to be implemented, a specific issue should imo go there: https://github.com/rancher/kine

[Locus99]

I had a similar situation where the file got to an amazing 177GB in size. Running those commands got rid of that file. Will follow up if there are any issues from running the commands.

[brandond]

We should probably evaluate either enabling auto-vacuum, or manually vacuuming at intervals or at startup since auto-vacuum cannot be enabled on existing databases.

https://sqlite.org/lang_vacuum.html

[brandond]

For anyone encountering this in the future - could you comment here? We'd like to know more about the workload and collect some statistics from your database.

In particular, using the Precompiled Binaries for Linux from https://www.sqlite.org/download.html, run:

./sqlite3_analyzer /var/lib/rancher/k3s/server/db/state.db
./sqlite3 /var/lib/rancher/k3s/server/db/state.db << EOF
.header on
.mode column
.dbinfo

SELECT 
  COUNT(*) AS rows, 
  (SELECT prev_revision FROM kine WHERE name = "compact_rev_key" LIMIT 1) AS compact_rev,
  MAX(id) AS current_rev
FROM kine;

SELECT COUNT(*), name FROM kine GROUP BY name ORDER BY COUNT(*) DESC LIMIT 50;

EOF

[brandond]

Closing this issue as it does not appear to be affecting anyone any longer. If someone does run into this, please collect information on the sqlite database as described above and open a new issue.

[andrewwebber]

Unfortunately I experience this issue on K3OS on AWS. Due to the minimal OS it is difficult to execute sqlite3 on the host.

I have made the following workaround for anyone on K3OS or a minimum / cloud OS

https://github.com/andrewwebber/k3os-db-vacuum