compose: Support layered-refs

[cgwalters]

A long while ago we added add-files to the manifest and people complained about various issues. We ended up implementing split compose steps and while that isn't wrong, I think it would be very natural for us to support something like this:

packages:
  - bash
  - podman
  ...

layered-refs:
 - examplecorp/overlay/somebinary
 - examplecorp/overlay/otherbinaries

And those layered-refs come from the target ostree repo; they can be built however the builder wants. This would work very nicely for rpm-ostree to be wrapped by a higher level build system, and further encourage those high level build systems to use ostree.

It'd interact nicely with our change detection; commit hashes on ostree refs do exactly what we need.

Specifically for cosa I am thinking it should support the config repo having an files directory with noarch (script/text) content that gets injected as a layered-refs.

[cgwalters]

One thing I'd love to support also is doing this on the client side - imagine spinning up a toolbox container with build tools, and doing a kernel/systemd/ostree/whatever build directly from git master, then make install DESTDIR=, commit it to an ostree ref and tell rpm-ostree to overlay that. No RPM involved (it'd be silly to compress then decompress the binary and unpack it to an ostree commit...). On the other hand this would mean the rpm version's old files wouldn't be removed.

[jlebon]

Oh man, I'm literally playing with something kinda similar to this right at this very moment. Though I'm thinking more of the config repo files going into an RPM that gets injected into the manifest. See coreos/ignition-dracut#47 (comment) and coreos/ignition-dracut#47 (comment). The advantages of using an RPM is that we reuse rpm-ostree's existing change detection logic and also the files are tracked by the rpmdb (so e.g. rpm -qf actually shows something useful).

[cgwalters]

The advantages of using an RPM is that we reuse rpm-ostree's existing change detection logic

Yeah but that'd be trivial to extend, just rev-parse each ref and add its checksum into our input checksum.

and also the files are tracked by the rpmdb (so e.g. rpm -qf actually shows something useful).

Yeah...though I have a patch I haven't submitted that supports us intercepting commands like rpm...we can easily make this happen:

$ rpm -qf /usr/share/rpm-ostree/treefile.json
<rpm-ostree internal>
$ rpm -qf /usr/lib/systemd/system/ignition-firstboot-complete.service
<ostree ref cosa/files>

[lucab]

What would be expected behavior when two sources are trying to own the same path?

[cgwalters]

What would be expected behavior when two sources are trying to own the same path?

See ostree checkout --union-identical.

[jlebon]

OK, just because I was already more halfway through this, I pushed it out as a WIP. :) Seems like a good opportunity anyway to help in the discussions of the two approaches.

coreos/coreos-assembler#414
coreos/fedora-coreos-config#66

(BTW, totally not against the approach here. I haven't thought about it a lot yet. Either way ISTM like the RPM approach in cosa can easily be a stopgap until we implement something like here if we end up going that way.)

[dustymabe]

Does this issue overlap at all with server side package layering?

[cgwalters]

This was actually done, treefile now has ostree-layers and ostree-override-layers https://github.com/coreos/rpm-ostree/blob/main/docs/treefile.md