Standard fsck isn't run on VM startup

[nrgaway]

/rw is not mounted in the standard way from /etc/fstab, because of for example DispVM, which does not have /rw mounted at all. This is also the reason why standard fsck isn't run on VM startup.

Possible solutions:
Add feature to qubes-manager to:

1. Show if there is disk error
   2 .Check disk
   3 .Repair disk

[marmarek]

Currently if one want to run fsck on /rw, it requires:

1. Adding single parameter to kernel command line (qvm-prefs -s VMNAME kernelopts single)
2. Starting VM - it would timeout on qrexec connection, but that's ok.
3. Access VM console using sudo xl console VMNAME
4. Get shell access (just press enter when prompted for root password)
5. Run fsck on /dev/xvdb (/rw): fsck -f /dev/xvdb
6. Shutdown the VM - poweroff from that shell
7. Restore default kernel command line: qvm-prefs -s VMNAME kernelopts default

[marmarek]

We should go with running standard fsck on VM startup. If it isn't desirable for some VM (which doesn't have /rw at all - DispVM), this could be addressed in .mount unit file with some condition.

[cfcs]

Alternatively you can run fsck on private.img from dom0 (if you trust fsck to not have security flaws, which is grantedly a risky assumption). Either way it's a required step for shrinking/compacting appvm filesystems to free up space.

[Rudd-O]

The agent really should detect if there is an error in the kernel ringbuffer or journald log, and submit that error for dom0 to display it in qubes-manager or with a notification.

But this will not cover the case of a system entirely failing to boot. In that case, it's possible that monitoring the console log in dom0 can provide a mechanism that lets the user know (qubes-manager or notification) that the VM is not booting properly, MUCH, much faster than just waiting for a timeout and then just dying.

[v6ak]

Hello,
I am not sure if we should reopen this issue or create a new one. This issue does not seem to be fully fixed. There is some interesting part of journalctl output:

Jul 30 07:48:53 localhost mount-dirs.sh[294]: Private device management: checking /dev/xvdb
Jul 30 07:48:53 localhost mount-dirs.sh[294]: Private device management: checking /dev/xvdb
Jul 30 07:48:53 localhost systemd[1]: Started udev Kernel Device Manager.
Jul 30 07:48:53 localhost mount-dirs.sh[294]: Private device management: fsck.ext4 /dev/xvdb failed:
Jul 30 07:48:53 localhost mount-dirs.sh[294]: /dev/xvdb contains a file system with errors, check forced.
Jul 30 07:48:53 localhost mount-dirs.sh[294]: /dev/xvdb: Inodes that were part of a corrupted orphan linked list found.
Jul 30 07:48:53 localhost mount-dirs.sh[294]: /dev/xvdb: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
Jul 30 07:48:53 localhost mount-dirs.sh[294]:         (i.e., without -a or -p options)
Jul 30 07:48:53 localhost resize-rootfs-if-needed.sh[293]: dumpe2fs 1.44.5 (15-Dec-2018)
Jul 30 07:48:53 localhost kernel: EXT4-fs (xvdb): warning: mounting fs with errors, running e2fsck is recommended
Jul 30 07:48:53 localhost kernel: EXT4-fs (xvdb): mounted filesystem with ordered data mode. Opts: discard
Jul 30 07:48:53 localhost resize-rootfs-if-needed.sh[293]: root filesystem already at 31041496 blocks
Jul 30 07:48:53 localhost systemd[1]: Started Adjust root filesystem size.
Jul 30 07:48:53 localhost mount-dirs.sh[294]: Checking /rw
Jul 30 07:48:53 localhost mount-dirs.sh[294]: Private device size management: enlarging /dev/xvdb
Jul 30 07:48:53 localhost mount-dirs.sh[294]: Private device size management: resize2fs /dev/xvdb failed:
Jul 30 07:48:53 localhost mount-dirs.sh[294]: resize2fs 1.44.5 (15-Dec-2018)
Jul 30 07:48:53 localhost mount-dirs.sh[294]: resize2fs: Permission denied to resize filesystem
Jul 30 07:48:53 localhost mount-dirs.sh[294]: Filesystem at /dev/xvdb is mounted on /rw; on-line resizing required
Jul 30 07:48:53 localhost mount-dirs.sh[294]: old_desc_blocks = 2, new_desc_blocks = 2
Jul 30 07:48:53 localhost kernel: EXT4-fs warning (device xvdb): ext4_resize_begin:46: There are errors in the filesystem, so online resizing is not allowed

So, the automatic way has failed. I am not notified (unless I read journalctl after every boot) and I am not sure how to do it manually:

$ qvm-block attach disp2095 dom0:/dev/qubes_dom0/vm-XXXXX-private
qvm-block: error: backend vm 'dom0' doesn't expose device '/dev/qubes_dom0/vm-XXXXX-private'

How has this happened? I guess this is related to having a full LVM thin provisioned pool in the past.

[andrewdavidwong]

@v6ak, @nrgaway: Is this still a problem on 4.1?

[github-actions]

This issue is being closed because:

• This issue is believed to affect only Qubes OS 4.1 (and possibly earlier).
• Qubes OS 4.1 has reached end-of-life (EOL).

If anyone believes that this issue should be reopened, please leave a comment saying so.
(For example, if a bug still affects Qubes OS 4.2, then the comment "Affects 4.2" will suffice.)