Unencapsulating OCI with extra layers (or: speeding up building child commits with couple RPM difference)

[pmtk]

In MicroShift CI we're building several ostree commits using osbuild.
However, because we need to build a parent image (clean el92), and then build commits with different versions of MicroShift, it takes quite a lot of time (using osbuild).

Some time ago, I had success with encapsulating the commits and adding RPMs using podman build.
But OCI images cannot be used with kickstart (for RHEL at least), so I wanted to turn that OCI back into ostree commit by unencapsulating it.
Unfortunately, the process was failing with either: Image has N non-ostree layers (with and without --squash) or Missing ostree.final-diffid (using --squash-all).

AFAIK you cannot edit image labels (to update final-diffid after squashing all layers), so maybe there's another tool that could help with that (without writing own tool that would use API directly to construct the OCI)?
I tried using buildah, but sudo buildah run $ctr -- rpm-ostree install /tmp/rpms/RPMS/noarch/* fails, whereas same command succeeds in podman build.

So, the question is: is there a way to turn this kind of OCI back into ostree commit without hacking the process too much? Maybe there's another way to achieve the goal with osbuild? I also tried rpm-ostree compose install but it wanted to resolve all the RPM dependencies even though the base-commit-turned-OCI has all the dependencies.

Here are the commands I'm using

sudo ostree container encapsulate --repo "${REPO}" "${BASE_REF}" "containers-storage:${BASE_OCI}"

sudo podman build --squash --tag "${NEW_OCI}" -f- "/home/microshift/microshift-rpm-build/_output/rpmbuild/" <<EOF
FROM ${BASE_OCI}
COPY RPMS /tmp/rpms
RUN rpm-ostree install /tmp/rpms/noarch/* /tmp/rpms/${BUILD_ARCH}/* && \
    systemctl enable microshift crio microshift-test-agent && \
    rm -rf /tmp/* && \
    ostree container commit
EOF

sudo ostree container unencapsulate --repo "${REPO}" --write-ref "${NEW_REF}" "ostree-unverified-image:containers-storage:${NEW_OCI}"

And outputs

$ sudo podman build --squash --tag localhost/microshift:src -f- /home/microshift/microshift-rpm-build/_output/rpmbuild/
STEP 1/3: FROM localhost/microshift:el92-deps
STEP 2/3: COPY RPMS /tmp/rpms
STEP 3/3: RUN rpm-ostree install /tmp/rpms/noarch/* /tmp/rpms/x86_64/* &&     systemctl enable microshift crio microshift-test-agent &&     rm -rf /tmp/* &&     ostree container commit

(rpm-ostree install:7): librhsm-WARNING **: 03:45:24.933: Found 0 entitlement certificates

(rpm-ostree install:7): librhsm-WARNING **: 03:45:24.934: Found 0 entitlement certificates
No enabled rpm-md repositories.
Importing rpm-md...done
Resolving dependencies...done
Installing 6 packages:
  microshift-4.14.0_0.nightly_2023_08_10_072041_20230818075653_3897d5086_dirty-1.el9.x86_64 (@commandline)
  microshift-greenboot-4.14.0_0.nightly_2023_08_10_072041_20230818075653_3897d5086_dirty-1.el9.noarch (@commandline)
  microshift-networking-4.14.0_0.nightly_2023_08_10_072041_20230818075653_3897d5086_dirty-1.el9.x86_64 (@commandline)
  microshift-release-info-4.14.0_0.nightly_2023_08_10_072041_20230818075653_3897d5086_dirty-1.el9.noarch (@commandline)
  microshift-selinux-4.14.0_0.nightly_2023_08_10_072041_20230818075653_3897d5086_dirty-1.el9.noarch (@commandline)
  microshift-test-agent-0.0.1-1.noarch (@commandline)
Installing: microshift-greenboot-4.14.0_0.nightly_2023_08_10_072041_20230818075653_3897d5086_dirty-1.el9.noarch (local)
Installing: microshift-networking-4.14.0_0.nightly_2023_08_10_072041_20230818075653_3897d5086_dirty-1.el9.x86_64 (local)
rpm-ostree-systemctl: Ignored non-preset command: is-active --quiet NetworkManager
rpm-ostree-systemctl: Ignored non-preset command: restart --quiet NetworkManager
rpm-ostree-systemctl: Ignored non-preset command: enable --now --quiet openvswitch
Installing: microshift-selinux-4.14.0_0.nightly_2023_08_10_072041_20230818075653_3897d5086_dirty-1.el9.noarch (local)
Installing: microshift-4.14.0_0.nightly_2023_08_10_072041_20230818075653_3897d5086_dirty-1.el9.x86_64 (local)
rpm-ostree-systemctl: Ignored non-preset command: is-active --quiet crio
rpm-ostree-systemctl: Ignored non-preset command: restart --quiet crio
Installing: microshift-release-info-4.14.0_0.nightly_2023_08_10_072041_20230818075653_3897d5086_dirty-1.el9.noarch (local)
Installing: microshift-test-agent-0.0.1-1.noarch (local)
Created symlink /etc/systemd/system/multi-user.target.wants/microshift.service → /usr/lib/systemd/system/microshift.service.
Created symlink /etc/systemd/system/cri-o.service → /usr/lib/systemd/system/crio.service.
Created symlink /etc/systemd/system/multi-user.target.wants/crio.service → /usr/lib/systemd/system/crio.service.
Created symlink /etc/systemd/system/multi-user.target.wants/microshift-test-agent.service → /usr/lib/systemd/system/microshift-test-agent.service.
COMMIT localhost/microshift:src
Getting image source signatures
Copying blob 18998f155fcc skipped: already exists  
Copying blob 375f76e8ae57 done  
Copying config ea266c7804 done  
Writing manifest to image destination
Storing signatures
--> ea266c78046
Successfully tagged localhost/microshift:src
ea266c7804611c5fa901c80097d27f6cd54235d77cbb78e1db98e25612118673

$ sudo ostree container unencapsulate --repo /home/microshift/test-rpm-ostree-compose/repo --write-ref xd/2 ostree-unverified-image:containers-storage:localhost/microshift:src
   Downloading...                                                                                                                                                                          error: Image has 1 non-ostree layers

[cgwalters]

But OCI images cannot be used with kickstart (for RHEL at least), so I wanted to turn that OCI back into ostree commit by unencapsulating it.
...
So, the question is: is there a way to turn this kind of OCI back into ostree commit without hacking the process too much?

This is similar to ostreedev/ostree-rs-ext#11 - I just added a comment there, but basically as soon as you have derived layers, you can't "unencapsulate" anymore. Basically ostree container image pull should work for this use case, does it for you?

[pmtk]

Yes, I think it does.
But it cannot pull into archive mode repo directly right? I'd have to pull into bare/bare-user and then do a pull-local from bare to archive, is this correct?

[cgwalters]

Yes, that's ostreedev/ostree-rs-ext#495 and indeed the double pull is the workaround today.

[pmtk]

Got it, thanks!