Build 16.1.0 for linux and macOS

[jdblischak]

Checklist

• Used a personal fork of the feedstock to propose changes
• Reset the build number to 0 (if the version changed)
• Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
• Ensured the license file is being packaged.

---

• The most recent version of r-arrow available from conda-forge is 14.0.1
• More recent releases have been blocked for about 6 months due to failing Windows builds (r-arrow v14.0.2 #71 was opened in late Dec 2023)
• While we continue to troubleshoot the Windows build (eg Use old msys2-conda-epoch #79), I think we should distribute more recent versions for linux and macOS
• To create this PR, I started with the bot's PR r-arrow v15.0.1 #76 for 15.0.1, rebased onto main, skipped Windows, added {{ stdlib("c") }}, and rerendered

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[jdblischak]

First I tried 15.0.1, but the URL https://dist.apache.org/repos/dist/release/arrow/arrow-15.0.1/apache-arrow-15.0.1.tar.gz no longer exists. Looking at https://dist.apache.org/repos/dist/release/arrow/, it appears that links to old releases are removed. Is there a less ephemeral link that we could use as the source?

Then I tried 16.0.0, but that failed to load during the test section. It was looking for libarrow_substrait.so.1600, but libarrow 16.1.0 was installed at runtime, which provides libarrow_substrait.so.1601. I'm a little confused how that was possible since the libarrow_substrait run exports max pin is "x.x", which I assumed would require libarrow_substrait 16.0 to be installed at runtime.

> library('arrow'); stopifnot(arrow_with_dataset(), arrow_with_parquet(), arrow_with_s3())
Error: package or namespace load failed for ‘arrow’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/home/conda/feedstock_root/build_artifacts/r-arrow_1717529301336/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/R/library/arrow/libs/arrow.so':
  libarrow_substrait.so.1600: cannot open shared object file: No such file or directory
Execution halted

Anyways, 16.1.0 was able to build. Can we release this newer version for linux and macOS?

[jdblischak]

@h-vetinari @xhochy thoughts on temporarily skipping the Windows build in order to create Linux and macOS binaries for r-arrow 16.1.0?

[h-vetinari]

I'll leave this for @xhochy to decide, who has been battling the windows builds here.

[jdblischak]

I'm a little confused how that was possible since the libarrow_substrait run exports max pin is "x.x", which I assumed would require libarrow_substrait 16.0 to be installed at runtime.

Tip

tl;dr Skip to the end to jump to the conclusion. Essentially the tightened run exports pin was only applied to 16.1.0 and not 16.0.0, so I don't believe we can build r-arrow 16.0.0 without backporting this to libarrow* 16.0.0.

I am still unclear why the run exports of libarrow-substrait isn't being enforced for r-arrow. It is listed in the host requirements, so it's run exports should be enforced at runtime.

r-arrow-feedstock/recipe/meta.yaml

Line 44 in 51b3145

- libarrow-substrait {{ version }}

Here's the link to the 16.0.0 build. I confirmed again that libarrow-substrait 16.0.0 was installed in the host env but libarrow-substrait 16.1.0 in the test section.

But it's not like there is no run exports. It clearly pins on the major version:

mamba create --dry-run -n test \
  -c conda-forge --override-channels \
  r-arrow=14 | grep substrait
##   + libarrow-substrait               14.0.2  he4f5ca8_25_cpu      conda-forge      524kB

I also looked at pyarrow. It pins correctly, but in an odd way that doesn't utilize run exports. Instead it manually pins the runtime dependency. I assume there is some reason why this strategy was chosen instead of relying on run exports

https://github.com/conda-forge/pyarrow-feedstock/blob/a0ad9428696da1954eb787e0cfd0eaeded3910bc/recipe/meta.yaml#L163-L169

mamba create --dry-run -n test \
  -c conda-forge --override-channels \
  pyarrow=15 | grep substrait
##   + libarrow-substrait             15.0.2  he4f5ca8_12_cpu       conda-forge      523kB

mamba create --dry-run -n test \
  -c conda-forge --override-channels \
  pyarrow=16.0 | grep substrait
##   + libarrow-substrait             16.0.0  h7e0c224_1_cpu       conda-forge      549kB

mamba create --dry-run -n test \
  -c conda-forge --override-channels \
  pyarrow=16.1 | grep substrait
##   + libarrow-substrait             16.1.0  h7e0c224_7_cpu       conda-forge      549kB

Ah, I think I found it. The run exports of libarrow-substrait (and the other libarrow libraries) was only increased from max_pin="x" to max_pin="x.x" 3 weeks ago in conda-forge/arrow-cpp-feedstock#1409. There was a repodata patch applied to existing libraries in conda-forge/conda-forge-repodata-patches-feedstock#731, but r-arrow hadn't been built again libarrow 16.0 at that point.

So that must mean my run above installed a libarrow-substrait 16.0 build prior to the updated pin. Unclear why the solver would favor this older version.

     libarrow-substrait:        16.0.0-h7e0c224_1_cpu       conda-forge

Nope. That is the latest version. It was uploaded on 2024-05-08, so prior to the updated run exports pins that was merged on 2024-05-17.

Oh wait. That PR was merged into main, so it only affected 16.1.0, not 16.0.0. And because the repodata patch can't be applied between the build and the test section, then IIUC there is no way to build r-arrow 16.0.0 without backporting the tightened run exports to 16.0.0.

[jdblischak]

I'll leave this for @xhochy to decide, who has been battling the windows builds here.

@xhochy any thoughts on temporarily skipping the Windows build in order to ship r-arrow 16.1.0 for Linux and macOS?

[xhochy]

Fine with me. I found a bit of the root of the windows errors, but for fully solving, it will me some more hours of attention.