Use ZFS features (but simple!)

[probonopd]

ZFS is this incredibly powerful tool that 99% of "mere mortals" desktop users will never entirely wrap their head around.

Hence, we should pre-configure it in a way most suitable to desktop users, and provide some really easy tools (ideally with zero configuration) to do the most common jobs, like scheduled local snapshots, rollbacks from within the boot manager, and network backups.

ZFS features I want to make use of

• Snapshots: Taking a snapshot should make a "cheap" copy of the whole system one can easily revert back to, should things go wrong. It should also be possible to go back and forth between different snapshots.
• Boot environments: I would want them to work like "I can select every snapshot directly in the boot manager to decide which one to boot into at boot time". Ideally, every snapshot is a boot environment and vice versa. (I don't need subtle differences to exist between the two.)
• Replication (network backups): I want to backup (image) the whole system to a network device, which I can restore from ("bare metal") like with a Time Machine network backup. Doing subsequent backups should copy only what has changed ("delta backups"). PC-BSD had a graphical installer that could restore systems from network servers (e.g., FreeNAS) using ZFS replication over SSH. I would want to re-create something like that.

Ideally I want to configure as little as possible as I am not a system administrator but a mere end user (zero configuration).

ZFS complexities to be understood/handled

While ZFS seems incredibly powerful, turns out it is also a lot more complex than needed, at least for desktop use.

Especially when plugging in and out external disks all the time, which I tend to do a lot. For example, I want to take the internal disk, put it on a SATA-to-USB3 adapter, and attach it to another target system, then copy stuff from /home on that disk to the internal disk of the target system.

Datasets

There is a concept of datasets which is some weird mix between partitions (Mac "volumes") and directories but not quite. Why is there yet another concept that we need to deal with? I would probably like to have just one dataset per pool so that things become as simple as possible.

This stuff reminds me of the old times when people set up different partitions for /home, /var, /usr and so on. I never liked having to deal with more than one partition on a disk, because I never knew which one would need how much space over time. With ZFS datasets, I no longer need to define sizes for those, but for some strange reasons, it seems that people still are dealing with different paths individually. I am used to systems where everything is on one partition, which makes everything much simpler for me.

So, is there a way to make it so that there is just one "/" dataset and that is it? Especially I want the whole disk behave the same and not run into unintended surprises like the ones outlined below.

Mounting external disks

When attaching an external disk with ZFS on it, then it doesn't just get mounted as an external disk. Instead, one has to "import the pool" (whatever that really means) and, suprise!, this mounts stuff over the booted system. For example, /home on the external disk doesn't get mounted under /media/external/home or something like it, but under /home.

This is referred to as "clobbering or replacing the current file system.

The behavior can be different for different datasets, depending on on whether some esoteric ZFS options like "canmount" have been set.

What is this good for? I find this dangerous, because it is very unexpected. How do I copy stuff from /home on the external disk to /home on the internal disk?

https://unix.stackexchange.com/questions/296972/how-to-mount-external-zfs-file-system-without-clobbering-altering-current-or-ext

Boot environments

When I played with boot environments, I thought "it does not work" until I found out by watching a YouTube video that actually /home is not covered by boot environments by default. What good is that for? To me, this violates POLA. Why can't I just have the whole system, without any exceptions, covered by boot environments? What good are "backups" when they leave out the most important type of data, the user's own? There should be at least a BIG RED WARNING that boot environments do NOT protect user data. (Probably the inventors thought of boot environments of something that covers only system changes while leaving user data as-is; whereas I experiment with user-side settings a lot, like changing to different desktop environments and playing with their settings, and want to use boot environments for that.)

[probonopd]

Here is a prototype for Boot Environments:

It is a GUI for the beadm tool by @vermaden.

[probonopd]

Why can't I just have the whole system, without any exceptions, covered by boot environments?

To get this, follow https://twitter.com/vermaden/status/1321952264735674368:

After default #FreeBSD on #ZFS install do these:

# zfs rename -u zroot/usr zroot/ROOT/default/usr
# zfs rename -u zroot/var zroot/ROOT/default/var

The /home is by default at /usr/home so it will also be moved to BE.

[probonopd]

Automounting existing ZFS pools from hot-plugged external disks

https://docs.oracle.com/cd/E19253-01/819-5461/gbcgl/index.html

Pools can also be imported using an alternate root. This feature allows for recovery situations, where the mount points should not be interpreted in context of the current root, but under some temporary directory where repairs can be performed. This feature also can be used when you are mounting removable media as described in the preceding section.

zpool import -R ...

Probably we should use this for automouting ZFS pools. Or is there an automounter for FreeBSD that can already do this?

Creating ZFS pools on external disks

Previously known as "formatting a disk".

https://docs.oracle.com/cd/E19253-01/819-5461/gaztn/index.html

The most common reason for creating an alternate root pool is for use with removable media. In these circumstances, users typically want a single file system, and they want it to be mounted wherever they choose on the target system.

zpool create -R ...

Managing ZFS pools the traditional way

https://docs.oracle.com/cd/E19253-01/819-5461/gaztn/index.html

You can manage ZFS file systems with legacy tools by setting the mountpoint property to legacy. Legacy file systems must be managed through the mount and umount commands

Isn't that what we want anyway, for everything but the pool from which the system boots?

Automatic fast and frequent incremental ZFS backups to a backup server

https://www.grendelman.net/wp/fast-frequent-incremental-zfs-backups-with-zrep/

Zrep is a shell script (written in Ksh) that was originally designed as a solution for asynchronous (but continuous) replication of file systems for the purpose of high availability (using a push mechanism).

• It seems that TrueNAS is a commonly used server system for on-site backups, and hence we should check whether it offers this or another similar solution

• PC-BSD/TrueOS had(?) Life-Preserver which could send backups to TrueNAS and restore to bare metal from those

• Can https://zrepl.github.io/ also do this?

• Automatically triggered local snapshots

• Encrypted partial differential cloud backup (likely of selected directories only), "like Dropbox"

• Automatically triggered (optionally encrypted) differential backup to external disk

• Automatically triggered (optionally encrypted) differential backup to local server e.g., over ssh

[vermaden]

From the useful tools that may but also may not) be useful here is rclone which can clone backups between various types, for example it can clone tar.gz file into S3 bucket on Minio or Amazon S3.

[probonopd]

https://twitter.com/vermaden/status/1326097314281353217

I can add ZFS pool attach to sysutils/automount if that is desired?

This would be super useful. My use case: Take FreeBSD ZFS boot disk out of a computer, attach it to another running FreeBSD system using a USB3 adapter. The partitions should be auto-mounted under /media "as usual", like if we were dealing with, e.g., NTFS

[probonopd]

For LAN backups:

According to @darkain, FreeNAS/TrueNAS is a very commonly installed thing, and is FreeBSD+ZFS based.
Whatever ZFS scripts are made, all that would be needed is something to automatically put SSH keys on the destination, otherwise, it'll "just work".

So we need to:

1. Discover FreeNAS/TrueNAS on the network using Zeroconf
2. Copy our SSH keys there (how)?

For WAN backups:

https://www.tarsnap.com/ --> See helloSystem/Utilities#155.

[shilch]

Want I would find very user friendly is if the operating system was a separate ZFS dataset mounted read-only at /System or so. Updates could then be shipped as ZFS snapshots and only need a proper boot manager that can handle this. Thereby updates would be atomic in nature and can easily be rolled back. Additionally, you have a clear distinction between the operating system and own changes.
Is someone aware of a system (some Linux distribution?) that does it that way?

[darkain]

FreeBSD standard distribution already partially does this. The "root" file system is actually zroot/ROOT/default with a mountpoint of /

If this were made read-only, with other datasets made read-write with other mount points, it could easily solve this situation.

Also, check out how iocage does jails, they have the concept of base jails that other jails can inherit from. They've already done a lot of the work on that front that can probably be replicated for the host OS as well.

[probonopd]

Very interesting!

[vermaden]

Want I would find very user friendly is if the operating system was a separate ZFS dataset mounted read-only at /System or so. Updates could then be shipped as ZFS snapshots and only need a proper boot manager that can handle this. Thereby updates would be atomic in nature and can easily be rolled back. Additionally, you have a clear distinction between the operating system and own changes.
Is someone aware of a system (some Linux distribution?) that does it that way?

FreeBSD already has this in the default ZFS install. Its called ZFS Boot Environments and its managed by beadm(8) from sysutils/beadm or bectl(8) from base. If you do not know this concept then please read this - https://is.gd/BECTL - should contain all needed information about ZFS Boot Environments.

[shilch]

I would prefer a read-only mount since even root can not accidentally modify system data. It requires the user to really understand what he's doing to modify the system.
But I don't know if it breaks POSIX if directories like /bin or /usr/bin are not writable to.

Edit: To clarify, have the system mounted read-only at /System and have a bind-mount / nullfs mount for directories at /bin or /usr/bin point into directories inside the /System.

[shilch]

Want I would find very user friendly is if the operating system was a separate ZFS dataset mounted read-only at /System or so. Updates could then be shipped as ZFS snapshots and only need a proper boot manager that can handle this. Thereby updates would be atomic in nature and can easily be rolled back. Additionally, you have a clear distinction between the operating system and own changes.
Is someone aware of a system (some Linux distribution?) that does it that way?

FreeBSD already has this in the default ZFS install. Its called ZFS Boot Environments and its managed by beadm(8) from sysutils/beadm or bectl(8) from base. If you do not know this concept then please read this - https://is.gd/BECTL - should contain all needed information about ZFS Boot Environments.

Thanks, I already read about it but I'm not sure if it is what I mean. I will test it out.

It looks pretty much like it. The question is how flexible the tools are.

[vermaden]

The question is how flexible the tools are.

What do you mean?

[shilch]

The question is how flexible the tools are.

What do you mean?

Whether they support different mount points and mount options such as read-only.

[vermaden]

Whether they support different mount points and mount options such as read-only.

Different mount points are supported. You can set in ZFS the readonly property to ON of OFF for any dataset.

[probonopd]

So far, I was unable to use zfs+nullfs+unionfs ro construct a root filesystem that appears to be r/w even though it resides on a r/o ISO.

https://github.com/helloSystem/ISO/blob/pure-zfs-test/overlays/ramdisk/init.sh#L63-L97

The result is unfortunately a kernel panic.

[grahamperrin]

Avoiding clobber

I guess:

1. include OpenZFS in stable/12 ISOs/distributions of helloSystem
2. zpool import with option -N to not mount any file system
3. use temporary mount points to avoid clobber.

@ahrens point (3), is that sane/practical?

IMHO for 13.0-CURRENT, manual page wording is potentially confusing in that temporary mount point properties do not include the mount point mountpoint.

References:

• https://www.freebsd.org/cgi/man.cgi?query=zpool-import
• https://www.freebsd.org/cgi/man.cgi?query=zfs-mount&sektion=8&manpath=FreeBSD+12.2-RELEASE+and+Ports
• https://www.freebsd.org/cgi/man.cgi?query=zfs-mount&sektion=8&manpath=FreeBSD+13.0-current
• https://www.freebsd.org/cgi/man.cgi?query=zfsprops&sektion=8&manpath=FreeBSD+13.0-current
• RFC: Temporary property and mount handling design changes needed to address multiple issues. · Discussion #7452 · openzfs/zfs

Historical:

• Add temporary mount options openzfs/zfs#3722 ▶ Add temporary mount options · openzfs/zfs@0282c41 committed 2015-09-03
• Feature #6780: zfs(8) temporary mount point properties should include mountpoint - illumos gate - illumos (2016-03-22, before OpenZFS came to FreeBSD; I should edit this)
• …

[grahamperrin]

3. use temporary mount points to avoid clobber.

I had an opportunity to try this a few days ago. Either:

• I made a mistake with command syntax; or
• the feature is not yet implemented and mountpoint is silently ignored as a temporary mount point property.

Clobber occurred, so I quietly dug myself out of my hole.