Default filesystem choices for FCOS

[ajeddeloh]

Container Llinux uses ext4 by default, Fedora Workstation uses ext4, Fedora Server uses xfs. We should pick one for / and /var (if /var is a seperate partition). Some people like btrfs, so maybe consider that too?

I personally don't have any strong opinion.

Assuming we can setup ostree between the disks and files stage, users would be able to change this via Ignition as well, so this would only affect the default image we ship.

[cgwalters]

In https://pagure.io/pungi-fedora/pull-request/257 one reason to use XFS is that it doesn't have fixed inode limits - and that's highly relevant for containers.

Another good reason to use XFS is it will soon have reflinks and that's also quite useful for overlayfs (and in general).

The other reason to use XFS is that it's the default for Red Hat Enterprise Linux 7 today.

[cgwalters]

Fedora Atomic host uses XFS for these reasons today.

[cgwalters]

(If one is comparing ext4 vs XFS, the main thing ext4 has today that XFS doesn't is encryption...that's a whole big discussion)

[dustymabe]

I don't see a good reason to go against what Fedora Server (and RHEL) are doing today with XFS. The only gripe I have with XFS is the inability to shrink a filesystem (which is annoying, but in practice you don't really need it that often).

[mskarbek]

btrfs is and will be laden with bad PR (fully deserved IMO) and hard to sell to certain groups of people. Personally, will never use it again, I value my data and don't want to go through btrfs hell again.
XFS is also a base file system for stratis which is supposed to land in Fedora 29 as 1.0 release and may be interesting in terms of future storage solutions in FCOS.

As soon as there will be something to work with I will start to experiment with combining ZFS and FCOS but that is a very limited use case, I know that. (Yes, I'm throwing that here as a statement, there are people using ZFS on Linux in production ;))

[dustymabe]

added meeting label so we can grab some consensus on this one.

[jasonbrooks]

I'm +1 to xfs

[mscherer]

So, what would be the plan for upgrading the on disk format if needed ?

I do remember that we did face a bug with RHOSP 12 or 13 with the introduction of container on XFS with a FS that was created with RHEL 7.0 and later upgraded to 7.5 (using every intermediate step). Openstack folks gave me those 2 links: https://bugzilla.redhat.com/show_bug.cgi?id=1564671 and https://bugzilla.redhat.com/show_bug.cgi?id=1309498 for people who want more details, I can give links to the internal discussion if needed.

Since the issue did only appear on a upgraded setup who was installed on 7.0, since the goal is to have continuous upgrade, is this something that need to be taken in account in the XFS vs ext4 (especially since Colin point out there is new features coming that we may need) ?

[dustymabe]

is this something that need to be taken in account in the XFS vs ext4

is the question the stability of XFS vs ext4 ?

[mscherer]

No, more on the question of on disk upgrade if needed. I do guess all suffer from the same issue anyway, but I kinda feel this is something to look for a continuous upgrade system, and something that would influence the choice.

Like would we need to change something, and if so "how", and can it be done safely in a cloud world (aka, with server that can reboot at the wrong moment, etc, etc)

[cgwalters]

No, more on the question of on disk upgrade if needed.

https://marc.info/?l=linux-ext4&m=149520766520774&w=2

is a bit related here (of course Ignition does do filesystem stuff from the initramfs). But this starts to get quite filesystem specific as far as what transitions can be made.

In the big picture, no upgrade system is really going to get you out of a need to reprovision at some point - at least for "long-lived" servers/desktops etc.

For Red Hat CoreOS, we are building technology that will do in-place updates where we can, or ensure nodes are reprovisioned as necessary. Particularly in virtualized infrastructure, reprovisioning isn't that expensive. For bare metal, one can amortize it.

The RHEL XFS thing was quite unfortunate. On the other hand, if you used LVM and didn't allocate all the disk space, you can easily make a new filesystem.

There's no magical solutions to this - there's really no getting around trying to avoid having long-lived pet filesystems. You want backups anyways, and if you have those you can reprovision and restore from backup.

[ajeddeloh]

Historically this hasn't been an issue with CL, at least in my time working on it (which is ext4 by default, not sure if XFS will cause more concern). We should not try to update the FS itself automatically (not sure how possible that even is), because it's just far too risky. Number one rule of automatic updates is "don't do things that you think might break other people's systems" and number zero rule is "Don't delete their data". If you want an updated filesystem, reprovision.

This does mean that we need to make sure that the kernel/userspace tools continue to work with older version filesystems.

[bgilbert]

Historically this hasn't been an issue with CL, at least in my time working on it (which is ext4 by default, not sure if XFS will cause more concern).

Also, we originally defaulted to btrfs for the root FS and there are still systems out there that use it.

[dustymabe]

@mscherer
Like would we need to change something, and if so "how", and can it be done safely in a cloud world (aka, with server that can reboot at the wrong moment, etc, etc)

we've talked a bit about this as well and have considered something like a 'forced reprovision' every so often, where if you're system has been up for more than X period of time we won't guarantee not to break backwards compatibility. i.e. some sort of EPOCH so that we don't have to carry workarounds for old issues forever. The idea is that these machines should be automatically provisioned (via ignition) so re-provisioning shouldn't be that big of a deal.

[dustymabe]

i.e. some sort of EPOCH so that we don't have to carry workarounds for old issues forever.

either way we're not going to solve that problem in this thread. We've got enough +1 for XFS. I'll open a PR for the Design doc and close this one out as decided.

[dustymabe]

We've got enough +1 for XFS. I'll open a PR for the Design doc and close this one out as decided.

#51

[dustymabe]

closing this since #51 was merged.

[Conan-Kudo]

I know I'm late to the party on this, but I want to rebut this particular thing in case this topic comes up again.

btrfs is and will be laden with bad PR (fully deserved IMO) and hard to sell to certain groups of people. Personally, will never use it again, I value my data and don't want to go through btrfs hell again.

It is not fully deserved. The most recent stress test of Btrfs indicates it performs well for data protection, just slower than ZFS.

I value my data too, which is why I've been using Btrfs on nearly all my systems (minus one that predates my switch to Btrfs for new systems) since mid-2015.

I've watched Btrfs improve in quality, correctness, and even performance over the past few years. It is nothing like the filesystem that was when RHEL 7 was branched from Fedora, or even when CentOS 6 was released. The Btrfs feature status page indicates most features are in good working order now, and even SUSE has enabled and supports most features in Btrfs in SLE 15 SP1.

The fact that so much of Facebook's own infrastructure runs on Btrfs also speaks to its quality. It also speaks to how perfect Btrfs is for container use-cases (as Fedora CoreOS is targeted at).

XFS is also a base file system for stratis which is supposed to land in Fedora 29 as 1.0 release and may be interesting in terms of future storage solutions in FCOS.

Stratis is moribund with the developer driving it leaving Red Hat. The concept is fundamentally flawed, and is not an improvement over LVM+XFS.

(Yes, I'm throwing that here as a statement, there are people using ZFS on Linux in production ;))

Unless this issue in ZoL is addressed, it doesn't matter.

[ajeddeloh]

@Conan-Kudo you can use Ignition to reformat /var (where your data should be anyway) to btrfs or ext4 or whatever you want. Long term I want to support the same for the root filesystem but that's trickier since it requires recreating the rootfs from a remote ostree repo in the initramfs. It would take longer to boot on first boot, true, but hopefully that's not a huge issue since it's only the first boot.