Enable opt-usrlocal-overlays by default

[travier]

Describe the enhancement

opt-usrlocal-overlays: boolean, optional: Defaults to false. By default, /opt and /usr/local are symlinks to subdirectories in / var. This prevents the ability to compose with packages that install in those directories. If enabled, RPMs with /opt and /usr/local content are allowed; client-side, both paths are writable overlay directories on. Requires libostree v2023.9+.

See:

• Determine support path for RPMs that install into /opt, /usr/local etc rpm-ostree#233
• Support RPMs installing in /opt and /usr/local rpm-ostree#4728
• ci: Test opt-usrlocal-overlays end-to-end in Prow CI rpm-ostree#4810
• https://coreos.github.io/rpm-ostree/treefile/#experimental-options

System details

N/A

Additional information

For Atomic Desktops, see: https://gitlab.com/fedora/ostree/sig/-/issues/20

[cgwalters]

Enabling by default would definitely escalate the divergence in things.

[cgwalters]

I think we need to start with telling anyone who wants to use package layering client side with opt to enable the service.

[jlebon]

Yeah, I don't think it's ready to enable by default yet. We really need more testing on this in real-world use cases to see how it fares. We can definitely document how to enable it for now client-side or in a derived container build (working on that right now).

[xynydev]

We can definitely document how to enable it for now client-side or in a derived container build (working on that right now).

If that is possible, I would appreciate the docs before this issue is completed, as I (/we) currently resort to a pretty ugly workaround for installing packages into /opt/ when building a derived container image.

[jlebon]

Just circling back here. Currently, we are no longer considering turning on state overlays by default. The ideal fix for /opt packages is to add symlinks to /var subdirs for the subpaths of /opt that need to be writable. That gives you the most immutability where possible, while poking holes only as needed. But it of course requires knowing what to symlink, which is software-dependent.

Both state overlays and transient root are easier alternatives which will make things Just Work, but with the tradeoff of allowing more mutability than necessary. This is documented in https://containers.github.io/bootc/filesystem.html#opt, in which I'm working to add the state overlay option (containers/bootc#668).

Note that in FCOS, all of this is gated on #1718.