Delayed failure with minimal config following official docs: /boot/efi: mount(2) system call failed: No space left on device #214871
Labels
0.kind: bug
Something is broken
Comments
|
I'm not sure what's going on with the To me, the more concerning issue right now would be the extreme slowdowns / freezes / timeouts / 100% CPU usage you are experiencing.
Was this Either way, here's a few troubleshooting steps I would try:
Finally, I would also check whether the SSDs have available firmware updates, e.g. by setting Note: Since you mentioned that you cannot complete a |
Yes, when writing the Issue, 10mins before reboot.
Fresh out of the box, bought 3 months ago, would surprise me, but will do anyways, thx! Just to reiterate in case of bad wording from me: No space is actually full... I can still write files, upload to Nextcloud, etc.
After reboot, I have zero problems completing
Thx, will do! It's a 24/7 running server, so it doesn't matter if it's an hours long test...
Awesome tip, thx. Will do and report back the next time it happens.
I'm a complete Nix noob and went with the most official documentation I could find. If there is another way to configure ZFS Mirror boot drives (including EFI being mirrored by a cp command) with encrypted root, then pls link it and I'll try that instead. I'm in love with the idea of declarative way of configuring a system and really want to make NixOS work. edit: I recall during install the docs instructing the user to disable disk cache... I wonder if that has a hand in this, since this is something that felt particularly weird when compared to installing a normal Linux distro, gotta find that quote real quick... Here it is, probably unrelated though... from https://openzfs.github.io/openzfs-docs/Getting%20Started/NixOS/Root%20on%20ZFS/3-system-configuration.html
|
BTW, I would also check whether the motherboard has any BIOS updates, just in case. Also, don't forget the
For reference, this is the configuration I've been using for ZFS mirrored boot drives with encrypted root. It uses LUKS for encryption (rather than ZFS encryption) and keeps the GRUB EFI loader installed on the UEFI partitions of both SSDs (with the remaining boot files, such as kernels, initrds and grub config in the { config, pkgs, ... }:
{
boot = {
loader = {
grub = {
enable = true;
version = 2;
efiSupport = true;
efiInstallAsRemovable = true;
zfsSupport = true;
mirroredBoots = [
{ devices = [ "nodev" ]; path = "/boot"; efiSysMountPoint = "/uefi0"; }
{ devices = [ "nodev" ]; path = "/boot"; efiSysMountPoint = "/uefi1"; }
];
};
efi = {
canTouchEfiVariables = false;
efiSysMountPoint = "/uefi0";
};
};
initrd = {
#(...)
luks.devices = {
# These are the UUIDs of the LUKS partitions where the root ZFS pool is stored
dec_rpool0 = { device = "/dev/disk/by-uuid/e4dd6bcf-01d7-4ace-b8ff-490bfb678ca5"; allowDiscards = true; bypassWorkqueues = true; };
dec_rpool1 = { device = "/dev/disk/by-uuid/dbf90988-93f1-4f73-8757-cace0bb377eb"; allowDiscards = true; bypassWorkqueues = true; };
# (...)
};
};
};
fileSystems = {
"/" = {
device = "rpool";
fsType = "zfs";
options = [ "noatime" "nodiratime" ];
};
"/uefi0" = {
device = "/dev/disk/by-uuid/39D6-67CE";
fsType = "vfat";
options = [ "noatime" "nodiratime" "umask=077" ];
};
"/uefi1" = {
device = "/dev/disk/by-uuid/3A15-C7E2";
fsType = "vfat";
options = [ "noatime" "nodiratime" "umask=077" ];
};
"/boot" = {
device = "bpool";
fsType = "zfs";
options = [ "noatime" "nodiratime" ];
};
# This one is optional, it's just an example of how to add a ZFS filesystem to your config
"/home" = {
device = "rpool/home";
fsType = "zfs";
options = [ "noatime" "nodiratime" ];
};
# (...)
};
services.zfs.autoScrub = {
enable = true;
interval = "Sun, 03:00";
pools = [ "bpool" "rpool" ];
};
}Note that this configuration is for a machine which exclusively uses NixOS, so with this config I don't have to worry about adding/removing EFI entries for NixOS, I simply configure the BIOS to boot from the default EFI boot entries for the hard disks, which boots from the The Also, to use the above config without any issues I also made sure to do |
Yes... I am not sure, but this might indeed be an issue. It is not necessary to do that at all, and the In case this is a problem, I would suggest simply doing What was really necessary to do after the installation to make the system boot is to do |
FrostKiwi
added a commit
to FrostKiwi/openzfs-docs
that referenced
this issue
Feb 7, 2023
By my understanding, unnecessary due to unmounting at the end of the instruction. Has lead to unstable and error-spewing ZFS setups, as discussed in this issue: NixOS/nixpkgs#214871
It's a bit early to judge, but I think that was it! 🥳 Careful celebration, server seems to be in remission. I not only The main early onset symptom seems gone! Each time I would
As such, the step is completely unnecessary, whether it caused my issues or not, because as per documentation,
Complete server is very fresh OOTB, so it returned with no updates. Devices with no available firmware updates:
• MG07ACA14TE
• MG07ACA14TE
• MG07ACA14TE
• MG07ACA14TE
• SSD 980 PRO 1TB
• SSD 980 PRO 1TB
No updatable devices
I specifically updated everything before deploying that Supermicro H12SSL. The Motherboard bios, BMC firmware the whole shebang.
Issued those commands, returned the next morning and copied the results of smartctl -a of NVME SSD1smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.9] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: Samsung SSD 980 PRO 1TB
Serial Number: S5GXNX0T<redacted>
Firmware Version: 5B2QGXA7
PCI Vendor/Subsystem ID: 0x144d
IEEE OUI Identifier: 0x002538
Total NVM Capacity: 1,000,204,886,016 [1.00 TB]
Unallocated NVM Capacity: 0
Controller ID: 6
NVMe Version: 1.3
Number of Namespaces: 1
Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB]
Namespace 1 Utilization: 17,607,667,712 [17.6 GB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 002538 b921be1d9e
Local Time is: Tue Feb 7 02:15:52 2023 CET
Firmware Updates (0x16): 3 Slots, no Reset required
Optional Admin Commands (0x0017): Security Format Frmw_DL Self_Test
Optional NVM Commands (0x0057): Comp Wr_Unc DS_Mngmt Sav/Sel_Feat Timestmp
Log Page Attributes (0x0f): S/H_per_NS Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
Maximum Data Transfer Size: 128 Pages
Warning Comp. Temp. Threshold: 82 Celsius
Critical Comp. Temp. Threshold: 85 Celsius
Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 8.49W - - 0 0 0 0 0 0
1 + 4.48W - - 1 1 1 1 0 200
2 + 3.18W - - 2 2 2 2 0 1000
3 - 0.0400W - - 3 3 3 3 2000 1200
4 - 0.0050W - - 4 4 4 4 500 9500
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 0
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 30 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 0%
Data Units Read: 13,986 [7.16 GB]
Data Units Written: 2,565,130 [1.31 TB]
Host Read Commands: 530,336
Host Write Commands: 28,001,809
Controller Busy Time: 264
Power Cycles: 30
Power On Hours: 15
Unsafe Shutdowns: 16
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 30 Celsius
Temperature Sensor 2: 38 Celsius
Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Loggedsmartctl -a of NVME SSD2smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.9] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: Samsung SSD 980 PRO 1TB
Serial Number: S5GXN<redacted>
Firmware Version: 5B2QGXA7
PCI Vendor/Subsystem ID: 0x144d
IEEE OUI Identifier: 0x002538
Total NVM Capacity: 1,000,204,886,016 [1.00 TB]
Unallocated NVM Capacity: 0
Controller ID: 6
NVMe Version: 1.3
Number of Namespaces: 1
Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB]
Namespace 1 Utilization: 17,564,291,072 [17.5 GB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 002538 b921be1d9f
Local Time is: Tue Feb 7 02:16:55 2023 CET
Firmware Updates (0x16): 3 Slots, no Reset required
Optional Admin Commands (0x0017): Security Format Frmw_DL Self_Test
Optional NVM Commands (0x0057): Comp Wr_Unc DS_Mngmt Sav/Sel_Feat Timestmp
Log Page Attributes (0x0f): S/H_per_NS Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
Maximum Data Transfer Size: 128 Pages
Warning Comp. Temp. Threshold: 82 Celsius
Critical Comp. Temp. Threshold: 85 Celsius
Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 8.49W - - 0 0 0 0 0 0
1 + 4.48W - - 1 1 1 1 0 200
2 + 3.18W - - 2 2 2 2 0 1000
3 - 0.0400W - - 3 3 3 3 2000 1200
4 - 0.0050W - - 4 4 4 4 500 9500
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 0
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 30 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 0%
Data Units Read: 13,138 [6.72 GB]
Data Units Written: 2,565,051 [1.31 TB]
Host Read Commands: 541,733
Host Write Commands: 28,036,790
Controller Busy Time: 262
Power Cycles: 30
Power On Hours: 15
Unsafe Shutdowns: 16
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 30 Celsius
Temperature Sensor 2: 37 Celsius
Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged |
|
Indeed, the SMART data looks good to me. I'm glad your issues seem to be resolved! 🥳 |
Will wait couple of days, if it's fine write down all the other symptoms the system had (maybe useful to others who stumble across this) and close the issue. |
|
@wizeman Still no disk full errors, but I managed to get nginx stuck in an infinite loop again. Also strace -pstrace: Process 1801584 attached
read(4, "unt-ro\n142659 142654 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142660 142659 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142665 142660 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,co"..., 1024) = 1024
read(4, "aster:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142677 142672 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142678 142677 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142683 14"..., 1024) = 1024
read(4, "rset=iso8859-1,shortname=mixed,errors=remount-ro\n142690 142689 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142695 142690 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142696 142695 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat "..., 1024) = 1024
read(4, "n/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142708 142707 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142713 142708 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,s"..., 1024) = 1024
read(4, "w,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142725 142720 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142726 142725 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142731 142726 259:6 / /run/systemd/unit-r"..., 1024) = 1024
read(4, "rrors=remount-ro\n142738 142737 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142743 142738 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142744 142743 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dma"..., 1024) = 1024
read(4, "relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142756 142755 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142761 142756 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro"..., 1024) = 1024
read(4, "=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142773 142768 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142774 142773 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142779 142774 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:"..., 1024) = 1024
read(4, "59:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142791 142786 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142792 142791 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=i"..., 1024) = 1024
read(4, "vme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142804 142803 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142809 142804 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142810 142809 259:6 / /run/syst"..., 1024) = 1024
read(4, "me=mixed,errors=remount-ro\n142821 142816 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142822 142821 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142827 142822 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmas"..., 1024) = 1024
read(4, "ot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142839 142834 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142840 142839 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors="..., 1024) = 1024
read(4, "2,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142852 142851 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142857 142852 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142858 142857 259:6 / /run/systemd/unit-root/boot/efi rw,relati"..., 1024) = 1024
read(4, "9 142864 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142870 142869 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142875 142870 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,i"..., 1024) = 1024
read(4, "fat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142887 142882 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142888 142887 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142893 142888 259:6 /"..., 1024) = 1024
read(4, "-1,shortname=mixed,errors=remount-ro\n142900 142899 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142905 142900 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142906 142905 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1"..., 1024) = 1024
read(4, "it-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142918 142917 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142923 142918 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mix"..., 1024) = 1024
read(4, ",dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142935 142930 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142936 142935 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142941 142936 259:6 / /run/systemd/unit-root/boot/efi"..., 1024) = 1024
read(4, "t-ro\n142948 142947 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142953 142948 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142954 142953 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,code"..., 1024) = 1024
read(4, "ter:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142966 142965 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142971 142966 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142972 1429"..., 1024) = 1024
read(4, "et=iso8859-1,shortname=mixed,errors=remount-ro\n142983 142978 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142984 142983 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142989 142984 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /d"..., 1024) = 1024
read(4, "systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143001 142996 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143002 143001 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,sho"..., 1024) = 1024
read(4, "fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143014 143013 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143019 143014 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143020 143019 259:6 / /run/systemd/unit-roo"..., 1024) = 1024
read(4, "ors=remount-ro\n143031 143026 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143032 143031 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143037 143032 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask"..., 1024) = 1024
read(4, "latime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143049 143044 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143050 143049 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n1"..., 1024) = 1024
read(4, "37,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143062 143061 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143067 143062 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143068 143067 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54"..., 1024) = 1024
read(4, ":6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143080 143079 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143085 143080 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso"..., 1024) = 1024
read(4, "e1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143097 143092 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143098 143097 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143103 143098 259:6 / /run/system"..., 1024) = 1024
read(4, "=mixed,errors=remount-ro\n143110 143109 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143115 143110 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143116 143115 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask="..., 1024) = 1024
read(4, "/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143128 143127 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143133 143128 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=re"..., 1024) = 1024
read(4, "codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143145 143140 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143146 143145 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143151 143146 259:6 / /run/systemd/unit-root/boot/efi rw,relatime"..., 1024) = 1024
read(4, "143157 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143163 143158 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143164 143163 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,ioc"..., 1024) = 1024
read(4, "t /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143176 143175 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143181 143176 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143182 143181 259:6 / /"..., 1024) = 1024
read(4, ",shortname=mixed,errors=remount-ro\n143193 143188 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143194 143193 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143199 143194 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1"..., 1024) = 1024
read(4, "-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143211 143206 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143212 143211 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed"..., 1024) = 1024
read(4, "mask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143224 143223 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143229 143224 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143230 143229 259:6 / /run/systemd/unit-root/boot/efi r"..., 1024) = 1024
read(4, "ro\n143241 143236 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143242 143241 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143247 143242 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepa"..., 1024) = 1024
read(4, "r:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143259 143254 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143260 143259 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143265 143260"..., 1024) = 1024
read(4, "=iso8859-1,shortname=mixed,errors=remount-ro\n143272 143271 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143277 143272 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143278 143277 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev"..., 1024) = 1024
read(4, "stemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143290 143289 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143295 143290 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,short"..., 1024) = 1024
read(4, "ask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143307 143302 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143308 143307 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143313 143308 259:6 / /run/systemd/unit-root/"..., 1024) = 1024
read(4, "s=remount-ro\n143320 143319 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143325 143320 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143326 143325 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0"..., 1024) = 1024
read(4, "time master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143338 143337 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143343 143338 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143"..., 1024) = 1024
read(4, ",iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143355 143350 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143356 143355 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143361 143356 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 -"..., 1024) = 1024
read(4, " / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143373 143368 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143374 143373 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso88"..., 1024) = 1024
read(4, "n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143386 143385 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143391 143386 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143392 143391 259:6 / /run/systemd/"..., 1024) = 1024
read(4, "ixed,errors=remount-ro\n143403 143398 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143404 143403 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143409 143404 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=00"..., 1024) = 1024
read(4, "fi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143421 143416 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143422 143421 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remo"..., 1024) = 1024
read(4, "depage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143434 143433 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143439 143434 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143440 143439 259:6 / /run/systemd/unit-root/boot/efi rw,relatime m"..., 1024) = 1024
read(4, "3446 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143452 143451 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143457 143452 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocha"..., 1024) = 1024
read(4, "/dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143469 143464 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143470 143469 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143475 143470 259:6 / /ru"..., 1024) = 1024
read(4, "hortname=mixed,errors=remount-ro\n143482 143481 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143487 143482 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143488 143487 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 r"..., 1024) = 1024
read(4, "oot/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143500 143499 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143505 143500 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,e"..., 1024) = 1024
read(4, "sk=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143517 143512 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143518 143517 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143523 143518 259:6 / /run/systemd/unit-root/boot/efi rw,"..., 1024) = 1024
read(4, "\n143530 143529 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143535 143530 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143536 143535 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage"..., 1024) = 1024
read(4, "54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143548 143547 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143553 143548 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143554 143553 2"..., 1024) = 1024This process used to be called nginx, but after another edit: Here is a video of it all. The silent failing, the lack of errors by The silent fail is after 2:20 of log.mp4edit2: followup I presume its all connected to that mirroredBoots = [
{ devices = [ "nodev" ]; path = "/boot"; efiSysMountPoint = "/uefi0"; }
{ devices = [ "nodev" ]; path = "/boot"; efiSysMountPoint = "/uefi1"; }
];
};
efi = {
canTouchEfiVariables = false;
efiSysMountPoint = "/uefi0";
};Would that config replicate the efi partition twice across both SSDs? |
gmelikov
pushed a commit
to openzfs/openzfs-docs
that referenced
this issue
Feb 8, 2023
By my understanding, unnecessary due to unmounting at the end of the instruction. Has lead to unstable and error-spewing ZFS setups, as discussed in this issue: NixOS/nixpkgs#214871
FYI, these are not system call errors, this is just a very long sequence of (successful) read() system calls. It seems that this process is reading from a file (or socket, or pipe, etc) that is outputting a very large sequence of the contents you mentioned. Part of those contents appear to be somewhat similar to In this case, the file descriptor corresponding to this "file" is file descriptor number 4 (although this number could be different on different runs). To figure out which "file" corresponds to that file descriptor, you'd have to repeat what you did with ... where This will show you what each open file descriptor corresponds to, in that process. If it turns out that the file descriptor corresponds to a pipe or a socket, a further step will be needed to figure out what it is connected to.
My guess is that you are running into what appears to be a few different problems. It may turn out that they have the same root cause, but I wouldn't be surprised if they are totally separate issues. Unfortunately I have no idea what's going on yet, so we'll need to do more debugging to figure out what's going on.
Yes, but you will also need to add the Also note that with my configuration, you will need to configure your BIOS to boot from the boot entries corresponding to the SSDs, rather than the "NixOS" boot entries (once you activate my configuration, you should be able to delete these NixOS entries if they still exist). |
|
Ah, I think I figured it out what file is being read. It's not Assuming that is the case, I believe your system is mounting an absolutely huge number of Can you show me the output of FYI, this is what my system shows: As you can see, I only have 28 entries, which is normal. But I suspect that if your system has been running for a few days, it will have thousands of entries. If so, I think this could be the underlying problem that you're experiencing, although we'll have to figure out what part of your configuration is causing this issue. |
|
I am now suspecting that the problematic part of your configuration is this one: As you can see, this is calling The Either way, as I mentioned in my last comment, it would be great if you could confirm that the output of But I suspect your issues will disappear if you get rid of these extra |
Thank you thank you so much for the great analysis! This has indeed struck me as odd, when following the NixOS Root on ZFS docs, but as a newcomer to Nix and NixOS I thought I wouldn't screw around and stick to what is documented.
You are right! Each invocation of 3 days 16 hours of almost idle run-time:20 29 0:5 / /dev rw,nosuid shared:10 - devtmpfs devtmpfs rw,size=6589200k,nr_inodes=16470489,mode=755
21 20 0:19 / /dev/pts rw,nosuid,noexec,relatime shared:11 - devpts devpts rw,gid=3,mode=620,ptmxmode=666
22 20 0:20 / /dev/shm rw,nosuid,nodev shared:12 - tmpfs tmpfs rw,size=65892000k
23 29 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:4 - proc proc rw
24 29 0:22 / /run rw,nosuid,nodev shared:13 - tmpfs tmpfs rw,size=32946000k,mode=755
25 24 0:23 / /run/keys rw,nosuid,nodev,relatime shared:14 - ramfs ramfs rw,mode=750
26 24 0:24 / /run/wrappers rw,nodev,relatime shared:15 - tmpfs tmpfs rw,size=65892000k,mode=755
27 29 0:25 / /sys rw,nosuid,nodev,noexec,relatime shared:5 - sysfs sysfs rw
29 1 0:27 / / rw,relatime shared:1 - zfs rpool/nixos/root rw,xattr,posixacl
30 29 0:28 / /var/lib rw,relatime shared:2 - zfs rpool/nixos/var/lib rw,xattr,posixacl
31 29 0:29 / /var/log rw,relatime shared:3 - zfs rpool/nixos/var/log rw,xattr,posixacl
32 29 0:27 /nix/store /nix/store ro,relatime shared:16 - zfs rpool/nixos/root rw,xattr,posixacl
33 27 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:6 - securityfs securityfs rw
34 27 0:30 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:7 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot
35 27 0:31 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:8 - efivarfs efivarfs rw
36 27 0:32 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:9 - bpf bpf rw,mode=700
37 20 0:17 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:17 - mqueue mqueue rw
39 20 0:33 / /dev/hugepages rw,relatime shared:18 - hugetlbfs hugetlbfs rw,pagesize=2M
38 27 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:19 - debugfs debugfs rw
40 27 0:34 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:20 - pstore pstore rw
41 27 0:35 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:21 - fusectl fusectl rw
42 27 0:36 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:22 - configfs configfs rw
43 29 0:37 / /home rw,relatime shared:23 - zfs rpool/nixos/home rw,xattr,posixacl
96 29 0:39 / /boot rw,nodev,relatime shared:48 - zfs bpool/nixos/root rw,xattr,posixacl
99 96 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
102 96 259:1 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966732E-part1 rw,relatime shared:54 - vfat /dev/nvme0n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
105 96 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
108 29 0:40 / /tank1 rw,relatime shared:57 - zfs tank1 rw,xattr,posixacl
576 24 0:50 / /run/user/78 rw,nosuid,nodev,relatime shared:297 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=78,gid=78
593 576 0:51 / /run/user/78/gvfs rw,nosuid,nodev,relatime shared:324 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=78,group_id=78
603 24 0:53 / /run/user/1000 rw,nosuid,nodev,relatime shared:209 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=1000,gid=100running nixos-rebuild switch once20 29 0:5 / /dev rw,nosuid shared:10 - devtmpfs devtmpfs rw,size=6589200k,nr_inodes=16470489,mode=755
21 20 0:19 / /dev/pts rw,nosuid,noexec,relatime shared:11 - devpts devpts rw,gid=3,mode=620,ptmxmode=666
22 20 0:20 / /dev/shm rw,nosuid,nodev shared:12 - tmpfs tmpfs rw,size=65892000k
23 29 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:4 - proc proc rw
24 29 0:22 / /run rw,nosuid,nodev shared:13 - tmpfs tmpfs rw,size=32946000k,mode=755
25 24 0:23 / /run/keys rw,nosuid,nodev,relatime shared:14 - ramfs ramfs rw,mode=750
26 24 0:24 / /run/wrappers rw,nodev,relatime shared:15 - tmpfs tmpfs rw,size=65892000k,mode=755
27 29 0:25 / /sys rw,nosuid,nodev,noexec,relatime shared:5 - sysfs sysfs rw
29 1 0:27 / / rw,relatime shared:1 - zfs rpool/nixos/root rw,xattr,posixacl
30 29 0:28 / /var/lib rw,relatime shared:2 - zfs rpool/nixos/var/lib rw,xattr,posixacl
31 29 0:29 / /var/log rw,relatime shared:3 - zfs rpool/nixos/var/log rw,xattr,posixacl
32 29 0:27 /nix/store /nix/store ro,relatime shared:16 - zfs rpool/nixos/root rw,xattr,posixacl
33 27 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:6 - securityfs securityfs rw
34 27 0:30 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:7 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot
35 27 0:31 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:8 - efivarfs efivarfs rw
36 27 0:32 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:9 - bpf bpf rw,mode=700
37 20 0:17 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:17 - mqueue mqueue rw
39 20 0:33 / /dev/hugepages rw,relatime shared:18 - hugetlbfs hugetlbfs rw,pagesize=2M
38 27 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:19 - debugfs debugfs rw
40 27 0:34 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:20 - pstore pstore rw
41 27 0:35 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:21 - fusectl fusectl rw
42 27 0:36 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:22 - configfs configfs rw
43 29 0:37 / /home rw,relatime shared:23 - zfs rpool/nixos/home rw,xattr,posixacl
96 29 0:39 / /boot rw,nodev,relatime shared:48 - zfs bpool/nixos/root rw,xattr,posixacl
99 96 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
102 96 259:1 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966732E-part1 rw,relatime shared:54 - vfat /dev/nvme0n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
105 96 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
108 29 0:40 / /tank1 rw,relatime shared:57 - zfs tank1 rw,xattr,posixacl
576 24 0:50 / /run/user/78 rw,nosuid,nodev,relatime shared:297 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=78,gid=78
593 576 0:51 / /run/user/78/gvfs rw,nosuid,nodev,relatime shared:324 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=78,group_id=78
603 24 0:53 / /run/user/1000 rw,nosuid,nodev,relatime shared:209 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=1000,gid=100
584 105 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
700 99 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-rorunning nixos-rebuild switch twice[20 29 0:5 / /dev rw,nosuid shared:10 - devtmpfs devtmpfs rw,size=6589200k,nr_inodes=16470489,mode=755
21 20 0:19 / /dev/pts rw,nosuid,noexec,relatime shared:11 - devpts devpts rw,gid=3,mode=620,ptmxmode=666
22 20 0:20 / /dev/shm rw,nosuid,nodev shared:12 - tmpfs tmpfs rw,size=65892000k
23 29 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:4 - proc proc rw
24 29 0:22 / /run rw,nosuid,nodev shared:13 - tmpfs tmpfs rw,size=32946000k,mode=755
25 24 0:23 / /run/keys rw,nosuid,nodev,relatime shared:14 - ramfs ramfs rw,mode=750
26 24 0:24 / /run/wrappers rw,nodev,relatime shared:15 - tmpfs tmpfs rw,size=65892000k,mode=755
27 29 0:25 / /sys rw,nosuid,nodev,noexec,relatime shared:5 - sysfs sysfs rw
29 1 0:27 / / rw,relatime shared:1 - zfs rpool/nixos/root rw,xattr,posixacl
30 29 0:28 / /var/lib rw,relatime shared:2 - zfs rpool/nixos/var/lib rw,xattr,posixacl
31 29 0:29 / /var/log rw,relatime shared:3 - zfs rpool/nixos/var/log rw,xattr,posixacl
32 29 0:27 /nix/store /nix/store ro,relatime shared:16 - zfs rpool/nixos/root rw,xattr,posixacl
33 27 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:6 - securityfs securityfs rw
34 27 0:30 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:7 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot
35 27 0:31 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:8 - efivarfs efivarfs rw
36 27 0:32 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:9 - bpf bpf rw,mode=700
37 20 0:17 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:17 - mqueue mqueue rw
39 20 0:33 / /dev/hugepages rw,relatime shared:18 - hugetlbfs hugetlbfs rw,pagesize=2M
38 27 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:19 - debugfs debugfs rw
40 27 0:34 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:20 - pstore pstore rw
41 27 0:35 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:21 - fusectl fusectl rw
42 27 0:36 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:22 - configfs configfs rw
43 29 0:37 / /home rw,relatime shared:23 - zfs rpool/nixos/home rw,xattr,posixacl
96 29 0:39 / /boot rw,nodev,relatime shared:48 - zfs bpool/nixos/root rw,xattr,posixacl
99 96 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
102 96 259:1 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966732E-part1 rw,relatime shared:54 - vfat /dev/nvme0n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
105 96 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
108 29 0:40 / /tank1 rw,relatime shared:57 - zfs tank1 rw,xattr,posixacl
576 24 0:50 / /run/user/78 rw,nosuid,nodev,relatime shared:297 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=78,gid=78
593 576 0:51 / /run/user/78/gvfs rw,nosuid,nodev,relatime shared:324 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=78,group_id=78
603 24 0:53 / /run/user/1000 rw,nosuid,nodev,relatime shared:209 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=1000,gid=100
584 105 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
700 99 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro](https://dpaste.com/EYMPLSU57.txt)running nixos-rebuild switch three times20 29 0:5 / /dev rw,nosuid shared:10 - devtmpfs devtmpfs rw,size=6589200k,nr_inodes=16470489,mode=755
21 20 0:19 / /dev/pts rw,nosuid,noexec,relatime shared:11 - devpts devpts rw,gid=3,mode=620,ptmxmode=666
22 20 0:20 / /dev/shm rw,nosuid,nodev shared:12 - tmpfs tmpfs rw,size=65892000k
23 29 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:4 - proc proc rw
24 29 0:22 / /run rw,nosuid,nodev shared:13 - tmpfs tmpfs rw,size=32946000k,mode=755
25 24 0:23 / /run/keys rw,nosuid,nodev,relatime shared:14 - ramfs ramfs rw,mode=750
26 24 0:24 / /run/wrappers rw,nodev,relatime shared:15 - tmpfs tmpfs rw,size=65892000k,mode=755
27 29 0:25 / /sys rw,nosuid,nodev,noexec,relatime shared:5 - sysfs sysfs rw
29 1 0:27 / / rw,relatime shared:1 - zfs rpool/nixos/root rw,xattr,posixacl
30 29 0:28 / /var/lib rw,relatime shared:2 - zfs rpool/nixos/var/lib rw,xattr,posixacl
31 29 0:29 / /var/log rw,relatime shared:3 - zfs rpool/nixos/var/log rw,xattr,posixacl
32 29 0:27 /nix/store /nix/store ro,relatime shared:16 - zfs rpool/nixos/root rw,xattr,posixacl
33 27 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:6 - securityfs securityfs rw
34 27 0:30 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:7 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot
35 27 0:31 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:8 - efivarfs efivarfs rw
36 27 0:32 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:9 - bpf bpf rw,mode=700
37 20 0:17 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:17 - mqueue mqueue rw
39 20 0:33 / /dev/hugepages rw,relatime shared:18 - hugetlbfs hugetlbfs rw,pagesize=2M
38 27 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:19 - debugfs debugfs rw
40 27 0:34 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:20 - pstore pstore rw
41 27 0:35 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:21 - fusectl fusectl rw
42 27 0:36 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:22 - configfs configfs rw
43 29 0:37 / /home rw,relatime shared:23 - zfs rpool/nixos/home rw,xattr,posixacl
96 29 0:39 / /boot rw,nodev,relatime shared:48 - zfs bpool/nixos/root rw,xattr,posixacl
99 96 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
102 96 259:1 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966732E-part1 rw,relatime shared:54 - vfat /dev/nvme0n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
105 96 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
108 29 0:40 / /tank1 rw,relatime shared:57 - zfs tank1 rw,xattr,posixacl
576 24 0:50 / /run/user/78 rw,nosuid,nodev,relatime shared:297 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=78,gid=78
593 576 0:51 / /run/user/78/gvfs rw,nosuid,nodev,relatime shared:324 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=78,group_id=78
603 24 0:53 / /run/user/1000 rw,nosuid,nodev,relatime shared:209 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=1000,gid=100
584 802 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
700 804 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
467 811 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
732 809 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
473 807 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
469 813 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
798 467 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
813 700 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
811 584 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
809 99 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
807 105 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
804 732 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
802 473 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
800 469 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-rorunning nixos-rebuild switch four times20 29 0:5 / /dev rw,nosuid shared:10 - devtmpfs devtmpfs rw,size=6589200k,nr_inodes=16470489,mode=755
21 20 0:19 / /dev/pts rw,nosuid,noexec,relatime shared:11 - devpts devpts rw,gid=3,mode=620,ptmxmode=666
22 20 0:20 / /dev/shm rw,nosuid,nodev shared:12 - tmpfs tmpfs rw,size=65892000k
23 29 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:4 - proc proc rw
24 29 0:22 / /run rw,nosuid,nodev shared:13 - tmpfs tmpfs rw,size=32946000k,mode=755
25 24 0:23 / /run/keys rw,nosuid,nodev,relatime shared:14 - ramfs ramfs rw,mode=750
26 24 0:24 / /run/wrappers rw,nodev,relatime shared:15 - tmpfs tmpfs rw,size=65892000k,mode=755
27 29 0:25 / /sys rw,nosuid,nodev,noexec,relatime shared:5 - sysfs sysfs rw
29 1 0:27 / / rw,relatime shared:1 - zfs rpool/nixos/root rw,xattr,posixacl
30 29 0:28 / /var/lib rw,relatime shared:2 - zfs rpool/nixos/var/lib rw,xattr,posixacl
31 29 0:29 / /var/log rw,relatime shared:3 - zfs rpool/nixos/var/log rw,xattr,posixacl
32 29 0:27 /nix/store /nix/store ro,relatime shared:16 - zfs rpool/nixos/root rw,xattr,posixacl
33 27 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:6 - securityfs securityfs rw
34 27 0:30 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:7 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot
35 27 0:31 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:8 - efivarfs efivarfs rw
36 27 0:32 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:9 - bpf bpf rw,mode=700
37 20 0:17 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:17 - mqueue mqueue rw
39 20 0:33 / /dev/hugepages rw,relatime shared:18 - hugetlbfs hugetlbfs rw,pagesize=2M
38 27 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:19 - debugfs debugfs rw
40 27 0:34 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:20 - pstore pstore rw
41 27 0:35 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:21 - fusectl fusectl rw
42 27 0:36 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:22 - configfs configfs rw
43 29 0:37 / /home rw,relatime shared:23 - zfs rpool/nixos/home rw,xattr,posixacl
96 29 0:39 / /boot rw,nodev,relatime shared:48 - zfs bpool/nixos/root rw,xattr,posixacl
99 96 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
102 96 259:1 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966732E-part1 rw,relatime shared:54 - vfat /dev/nvme0n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
105 96 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
108 29 0:40 / /tank1 rw,relatime shared:57 - zfs tank1 rw,xattr,posixacl
576 24 0:50 / /run/user/78 rw,nosuid,nodev,relatime shared:297 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=78,gid=78
593 576 0:51 / /run/user/78/gvfs rw,nosuid,nodev,relatime shared:324 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=78,group_id=78
603 24 0:53 / /run/user/1000 rw,nosuid,nodev,relatime shared:209 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=1000,gid=100
584 946 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
700 948 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
467 955 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
732 953 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
473 951 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
469 957 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
798 971 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
813 970 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
811 968 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
809 966 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
807 964 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
804 961 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
802 959 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
800 973 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
942 798 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
973 469 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
971 467 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
970 700 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
968 584 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
966 99 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
964 105 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
961 732 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
959 473 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
957 813 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
955 811 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
953 809 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
951 807 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
948 804 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
946 802 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
944 800 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-roOhh, funnily enough, this code block was removed two days ago: openzfs/openzfs-docs@1211e98 edit: Fresh restart20 29 0:5 / /dev rw,nosuid shared:10 - devtmpfs devtmpfs rw,size=6589200k,nr_inodes=16470488,mode=755
21 20 0:19 / /dev/pts rw,nosuid,noexec,relatime shared:11 - devpts devpts rw,gid=3,mode=620,ptmxmode=666
22 20 0:20 / /dev/shm rw,nosuid,nodev shared:12 - tmpfs tmpfs rw
23 29 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:4 - proc proc rw
24 29 0:22 / /run rw,nosuid,nodev shared:13 - tmpfs tmpfs rw,size=32946000k,mode=755
25 24 0:23 / /run/keys rw,nosuid,nodev,relatime shared:14 - ramfs ramfs rw,mode=750
26 24 0:24 / /run/wrappers rw,nodev,relatime shared:15 - tmpfs tmpfs rw,mode=755
27 29 0:25 / /sys rw,nosuid,nodev,noexec,relatime shared:5 - sysfs sysfs rw
29 1 0:27 / / rw,relatime shared:1 - zfs rpool/nixos/root rw,xattr,posixacl
30 29 0:28 / /var/lib rw,relatime shared:2 - zfs rpool/nixos/var/lib rw,xattr,posixacl
31 29 0:29 / /var/log rw,relatime shared:3 - zfs rpool/nixos/var/log rw,xattr,posixacl
32 29 0:27 /nix/store /nix/store ro,relatime shared:16 - zfs rpool/nixos/root rw,xattr,posixacl
33 27 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:6 - securityfs securityfs rw
34 27 0:30 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:7 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot
35 27 0:31 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:8 - efivarfs efivarfs rw
36 27 0:32 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:9 - bpf bpf rw,mode=700
37 20 0:17 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:17 - mqueue mqueue rw
38 27 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:18 - debugfs debugfs rw
39 20 0:33 / /dev/hugepages rw,relatime shared:19 - hugetlbfs hugetlbfs rw,pagesize=2M
40 27 0:34 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:20 - pstore pstore rw
41 27 0:35 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:21 - fusectl fusectl rw
42 27 0:36 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:22 - configfs configfs rw
92 29 0:37 / /home rw,relatime shared:47 - zfs rpool/nixos/home rw,xattr,posixacl
96 29 0:39 / /boot rw,nodev,relatime shared:50 - zfs bpool/nixos/root rw,xattr,posixacl
99 96 0:40 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - autofs systemd-1 rw,fd=188,pgrp=1,timeout=60,minproto=5,maxproto=5,direct,pipe_ino=14523
102 96 0:41 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966732E-part1 rw,relatime shared:54 - autofs systemd-1 rw,fd=189,pgrp=1,timeout=60,minproto=5,maxproto=5,direct,pipe_ino=17598
105 99 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:56 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
108 96 259:6 / /boot/efi rw,relatime shared:56 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
111 29 0:42 / /tank1 rw,relatime shared:59 - zfs tank1 rw,xattr,posixacl
592 24 0:52 / /run/user/78 rw,nosuid,nodev,relatime shared:304 - tmpfs tmpfs rw,size=13178400k,nr_inodes=3294600,mode=700,uid=78,gid=78
609 592 0:53 / /run/user/78/gvfs rw,nosuid,nodev,relatime shared:332 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=78,group_id=78
699 24 0:55 / /run/user/1000 rw,nosuid,nodev,relatime shared:214 - tmpfs tmpfs rw,size=13178400k,nr_inodes=3294600,mode=700,uid=1000,gid=100Running nixos-rebuild switch five times20 29 0:5 / /dev rw,nosuid shared:10 - devtmpfs devtmpfs rw,size=6589200k,nr_inodes=16470488,mode=755
21 20 0:19 / /dev/pts rw,nosuid,noexec,relatime shared:11 - devpts devpts rw,gid=3,mode=620,ptmxmode=666
22 20 0:20 / /dev/shm rw,nosuid,nodev shared:12 - tmpfs tmpfs rw
23 29 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:4 - proc proc rw
24 29 0:22 / /run rw,nosuid,nodev shared:13 - tmpfs tmpfs rw,size=32946000k,mode=755
25 24 0:23 / /run/keys rw,nosuid,nodev,relatime shared:14 - ramfs ramfs rw,mode=750
26 24 0:24 / /run/wrappers rw,nodev,relatime shared:15 - tmpfs tmpfs rw,mode=755
27 29 0:25 / /sys rw,nosuid,nodev,noexec,relatime shared:5 - sysfs sysfs rw
29 1 0:27 / / rw,relatime shared:1 - zfs rpool/nixos/root rw,xattr,posixacl
30 29 0:28 / /var/lib rw,relatime shared:2 - zfs rpool/nixos/var/lib rw,xattr,posixacl
31 29 0:29 / /var/log rw,relatime shared:3 - zfs rpool/nixos/var/log rw,xattr,posixacl
32 29 0:27 /nix/store /nix/store ro,relatime shared:16 - zfs rpool/nixos/root rw,xattr,posixacl
33 27 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:6 - securityfs securityfs rw
34 27 0:30 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:7 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot
35 27 0:31 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:8 - efivarfs efivarfs rw
36 27 0:32 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:9 - bpf bpf rw,mode=700
37 20 0:17 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:17 - mqueue mqueue rw
38 27 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:18 - debugfs debugfs rw
39 20 0:33 / /dev/hugepages rw,relatime shared:19 - hugetlbfs hugetlbfs rw,pagesize=2M
40 27 0:34 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:20 - pstore pstore rw
41 27 0:35 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:21 - fusectl fusectl rw
42 27 0:36 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:22 - configfs configfs rw
92 29 0:37 / /home rw,relatime shared:47 - zfs rpool/nixos/home rw,xattr,posixacl
96 29 0:39 / /boot rw,nodev,relatime shared:50 - zfs bpool/nixos/root rw,xattr,posixacl
99 96 0:40 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - autofs systemd-1 rw,fd=188,pgrp=1,timeout=60,minproto=5,maxproto=5,direct,pipe_ino=14523
102 96 0:41 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966732E-part1 rw,relatime shared:54 - autofs systemd-1 rw,fd=189,pgrp=1,timeout=60,minproto=5,maxproto=5,direct,pipe_ino=17598
105 99 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:56 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
108 96 259:6 / /boot/efi rw,relatime shared:56 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
111 29 0:42 / /tank1 rw,relatime shared:59 - zfs tank1 rw,xattr,posixacl
592 24 0:52 / /run/user/78 rw,nosuid,nodev,relatime shared:304 - tmpfs tmpfs rw,size=13178400k,nr_inodes=3294600,mode=700,uid=78,gid=78
609 592 0:53 / /run/user/78/gvfs rw,nosuid,nodev,relatime shared:332 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=78,group_id=78
699 24 0:55 / /run/user/1000 rw,nosuid,nodev,relatime shared:214 - tmpfs tmpfs rw,size=13178400k,nr_inodes=3294600,mode=700,uid=1000,gid=100
619 102 259:2 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966732E-part1 rw,relatime shared:388 - vfat /dev/nvme0n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro |
FrostKiwi
referenced
this issue
in openzfs/openzfs-docs
Feb 12, 2023
Previously we used a bind mount from /boot/efis/*-part1 to /boot/efi to facilitate bootloader configuration. Recent reports indicate that this bind mount prevents the system from booting. This pull request removes the bind mount. Closes #383. Signed-off-by: Maurice Zhou <[email protected]>
|
Assuming that this is resolved. Let me know otherwise! |
Yes, forgot to close this. Thanks. This issue's resultFollowing the docs on OpenZFS to setup NixOS with Root on ZFS (at time of issue creation) created a broken NixOS setup which killed itself with accumulated up-time and openzfs/openzfs-docs@1211e98 solved it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
I have a server with dual NVME SSDs as my boot and root drive, which are setup in a mirror config in accordance with https://openzfs.github.io/openzfs-docs/Getting%20Started/NixOS/Root%20on%20ZFS.html in UEFI mode.
After roughly 48 hours or so, the server running this config starts to break down.
nixos-rebuild switchstops working, never completing and getting stuck without errors or anything.dfstops working, never returning.HTOPshows (namely nscd) processes stuck at 100%, putting full load on one core and never exiting, see screenshot below. The mentioned efi and boot partitions are 1gb and 4gb as per doc. Never full, always around 2%.Log of nixos-rebuild switch showing the 'No space left on device.' error (Config identical to last time it was completed successfully, yet does not complete and just gets stuck forever)
This log took roughly 12 minutes to generate. Every step, every output line gets stuck for roughly a minute or so.
This has been happening since it was a completely fresh install with a minimal config.
the used zfs.nix
Only thing changed from stock is this, in an attempt to fix this issue. But either way it's the same result:
Full DMESG for sake of complettness
systemd-gpt-auto-generatorError at the very bottomI did not have such a problem with FreeBSD. After switching to NixOS, I have not been able to run the server for longer than 3 days without requiring a reboot. I cannot complete
nixos-rebuild switch, even though the config is indentical to the last time it was ran.This is similar to #27821 , but with the exception, that following @wizeman 's comment #27821 (comment) does not work, because no
/sys/firmware/efi/efivars/dump-*exist.I am forced to reboot. After reboot everything is fine again, for roughly 2-3 days.
Steps To Reproduce
Steps to reproduce the behavior:
Expected behavior
Things not blowing up after 48 hours.
Screenshots
Additional context
Add any other context about the problem here.
Notify maintainers
@jcumming @jonringer
Metadata
Please run
nix-shell -p nix-info --run "nix-info -m"and paste the result.The text was updated successfully, but these errors were encountered: