pull_with_options uses /var/tmp as tmp space, preventing pulling big files during kickstart installs

[champtar]

Using libostree/2021.3 (Alma 8.5)
I'm trying to install a tree over https containing big files (~1.5G) using anaconda/kickstart
While anaconda is running /var/tmp only has 800 Mo free (/dev/mapper/live-rw), and once pull_with_options pulls a big file, running df -h, I can see / fill up until I get a No space left on device.
lsof /var/tmp shows a deleted file:

file:// install are working but not https:// (so either libcurl or the code around it need love)
If possible tmp space should be in the ostree repo dir by default IMO.

As a workaround we can just mount a tmpfs on top of /var/tmp:

%pre
mount -t tmpfs -o size=100% tmpfs /var/tmp/
%end

[cgwalters]

If possible tmp space should be in the ostree repo dir by default IMO.

Yes; hmm, AFAICS this should be the case right now. This needs a bit more debugging; there's various places where we pass the tempdir down, and it should point to the repository tmpdir i.e. /ostree/repo/tmp.

[champtar]

Do you know if it shipped with 8.7, IE should I try again ?

[NeilHanlon]

Heya - Just a note as I am experiencing this as well in the current 8.8 releases with the following ostree/rpm-ostree.

ostree-2022.2-6.el8
rpm-ostree-2022.10.115.g15eba7b1-2.el8_8

More than happy to try and debug this -- I was looking into the code which performs the download and found my way toward the libGObject interfaces for ostree and rpm-ostree, but would appreciate any pointers you have to where I can look further, @cgwalters.

FYI the versions above are from Rocky Linux 8.8, but I did confirm it occurs w/ RHEL versions, too.

[dbnicholson]

I'm pretty sure this is from the fetcher. Unless OSTREE_FETCHER_REQUEST_LINKABLE is set (which is not the common case AFAICT), _ostree_fetcher_tmpf_from_flags uses glnx_open_anonymous_tmpfile, which uses TMPDIR or /var/tmp as a fallback.

[dbnicholson]

So, I believe as a workaround you can set TMPDIR to somewhere with lots of space (potentially /ostree/repo/tmp) and see if that avoids the problem.

[cgwalters]

Right, the _LINKABLE flag is set if

  if (!is_meta && pull_data->trusted_http_direct)
    flags |= OSTREE_FETCHER_REQUEST_LINKABLE;

That logic dates to 2e3889a

Which directly writes to the repo only if we're not verifying checksums.

But that's a bug; first, we should always be opening the tempfile in the repo location because that's where the space will be allocated.

Second, it'd actually be a lot more efficient to just verify the checksum as we're fetching and then if that passes, directly use that fd we allocated for the final file.

[cgwalters]

PR in #2875

[cgwalters]

xref #2900 (comment) - when we aim to re-land this it'll need to be done in a way that avoids breaking flatpak (even if we find a flatpak fix, we should be conservative and still work with older versions).