julia=1.7.1 currently needs openlibm<0.8.0 therefore latest conda env builds with deps from conda-forge fail to solve

[valeriupredoi]

Hey guys, Happy New Year! Just a heads up that the latest Julia can not be installed in a conda env with deps from conda-forge unless one pins openlibm=0.7.0 (presumably a pin like openlibm<0.8 should work too) since the latest openlibm is now 0.8.0 as of 12 hours ago; see our issue on this matter ESMValGroup/ESMValTool#2450 Cheers 🍺

[Keno]

The Conda package is not an official distribution, so there's nothing we can do. Please file this issue with Conda.

[valeriupredoi]

in the process of doing that, but that's something you guys should probably be aware of in case other users report the same, I'd have kept the issue open a wee longer if I were you, at least until the problem gets fixed 👍 Who maintains the conda forge package, isn't you after all? Cheers 🍺

[Keno]

Anything that's not the official distribution is usually in various states of broken ;) - the Julia issue tracker is for issues with Julia, which this is not. We do not have the resource to support unsupported distributions (by definition) and do not track those issues here.

[valeriupredoi]

gotcha! Cheers, mate! I'm just about to get that issue over to the Julia conda-feedstock people 😁

[ngam]

the Julia issue tracker is for issues with Julia, which this is not

Well... It kinda is. The openlibm ABI has changed and so anyone using julia with system utilities may, in fact, be affected by this.

[mkitti]

Julia 1.7.1 currently uses openlibm 0.7.4. That was not available in conda-forge until today. 0.7.5 was also made available today.

I'm one of the main drivers of the conda-forge builds, especially over the past 6 months. While I am a community contributor to Julia, I am not acting in any official capacity in my conda-forge contributions because I have none.

We will get the < 0.8 pin in today. We will see if openlibm 0.7.5 works as well. That said, since all the patches and pins from the official Julia builds are not in conda-forge, I consider the conda-forge builds to be highly experimental.

[Keno]

The openlibm ABI has changed and so anyone using julia with system utilities may, in fact, be affected by this.

That is also not a supported configuration at this time. Each official julia version pins all of its dependencies to specific versions. People are free to change those versions (and people frequently do as experiments or in preparation for upgrades), but those are again unsupported configurations.

[ngam]

its dependencies to specific versions

I understand --- and maybe that's why you just vendor everything. But, I would hope that you start providing an actual range against which julia should work flawlessly. This is especially the case as there is a significant push to advertise julia --- this julia --- as a mature software. A mature software wouldn't be pinning deps to the sub-minor. This also goes against the spirit --- again as advertised by the appointed spokesperson of julia --- that julia is truly open source. What use is it if it is open source, yet has exacting and crazy deps and so you have to just get the binaries?

[ngam]

I am not trying to start a fight. I want you to open up to third-party providers, so that we --- all of us --- together can make julia what is premised to be.

[Keno]

A mature software wouldn't be pinning deps to the sub-minor

Well, I would disagree on that. Our first and foremost objective is to deliver our users working software. As I mentioned elsewhere that includes an enormous amount of validation work to validate even just this one particular set of dependencies works. We used to be a lot more loose with the dependencies and got stricter as the software matured. We do not have the resources available to extend these validation efforts to more configurations and even if we did, spending those resources on that would not be the correct allocation of those resources.

[Keno]

Also, just to be clear, building from source is definitely supported. If you download the official julia distribution and run make you should get a working distribution.

[ngam]

Also, just to be clear, building from source is definitely supported. If you download the official julia distribution and run make you should get a working distribution.

Essentially that's all we do, except we build against "system" utilities rather than repackage them through the make run.

Our first and foremost objective is to deliver our users working software.

Anyway, that's a good objective and if the cost to slow the embrace of third-party stuff, that's fine.

Well, I would disagree on that.

The idea is that not many breaking changes would be happening between minor releases from all random deps, but you may have encountered difficulties and had to adjust course --- who knows? Anyway, expect to close more issues and PRs from me going forward 😜 (not to spam you, but to at least alert you of potential problems down the line)

[ngam]

Thanks for taking the time to engage and explain @Keno. Appreciate it.

[oscardssmith]

One thing that might help is that we will hopefully no longer rely on openlibm by Julia 1.8 (possibly 1.9).

[mkitti]

As Keno pointed out earlier, it consumes a lot of resources to fully test the builds. The pinned versions are the only ones where the full battery of tests have been done, including those checking for performance regressions. There usually is a PR floating around to advance the version number of the dependencies.

When I got involved with the julia-feedstock, we were only doing very few tests, and it was recommended to me by core to keep the tests limited so as to not overwhelm conda-forge's infrastructure. We'll need to check with conda-forge core before expanding testing significantly. That said we do not have the resources to come close to the testing at https://build.julialang.org/

Also just to be clear, Keno Fischer is one the co-founders of Julia and is CTO of Julia Computing.

[Keno]

except we build against "system" utilities rather than repackage them through the make run.

Yes, messing with the versions is the unsupported part ;)

The idea is that not many breaking changes would be happening between minor releases from all random deps,

Yeah, in our experience the likelihood that upgrading a dependency (even a minor version bump) breaks nothing is much closer to 0 than to 100%. Not to blame upstream generally. Julia release validation is much more intensive than any CI those projects could or should reasonably run.

[mkitti]

Also a version bump from 0.7.x to 0.8 signifies a breaking change, although my understanding is that the change may not actually be breaking since the ABI only expanded. Also since it's sub-1.0, openlibm should not be considered stable from version to version at all per https://semver.org/

[Keno]

to the testing at https://build.julialang.org/

And don't forget the PkgEval runs, nanosoldier and everything repeated again across the proprietary code bases at the various companies that are built on Julia. As I said, the validation effort is significant.

[ngam]

although my understanding is that the change may not actually be breaking since the ABI only expanded

It was actually just a mistake: JuliaMath/openlibm#248 (comment). The variable shouldn't have been bumped up...

[ngam]

Also just to be clear, Keno Fischer is one the co-founders of Julia and is CTO of Julia Computing

Glad @Keno (manager) is directly involved with unruly users* (me) 🤣

That said we do not have the resources

Fair, but there are options. For example, for complex packages like tensorflow and qt, users volunteer to build locally and upload to anaconda from their own devices. The hard-coded limit is 6 hours, I believe, for ci (mainly azure) builds (we can split the tests to get around that). Currently our julia build + test takes only one hour (for linux 64 and osx-64; it would likely take significantly longer if/when we cross-compile for aarch64, etc., though we usually disable the tests completely for cross-compilation), so we have room to expand. In other words, from the perspective of conda-forge, I wouldn't consider testing the hindrance --- for now.

@Keno, in all seriousness though, the ultimate goal is to open the crazy infrastructure available through conda-forge (15000 packages or so) to the julia ecosystem. Wouldn't that be simply great? Much easier said than done though...

(*funny thing: I'm not actually a user of julia --- yet --- but I find this building effort really fun and interesting and as a benefit I get familiar with julia and the associated infrastructure, which I've been meaning to do for years)

[Keno]

Currently our julia build + test takes only one hour

Ok, so let's say that's currently what 10 core-hours (not sure how many platforms you build)? Just for comparison, I'd estimate that one validation run of julia (CI+PkgEval+Nanosoldier) takes about 300 core-days and we run a number of those for every release, plus the time for humans to go through the reports (which are unfortunately fairly noisy) to figure out if there's an issue. Does conda-forge have the resources to increase its resource investment in Julia by more than 1000x plus the human time? And even if it does, is the benefit really worth it? I really don't want to be dismissive here, packaging is thankless work, but I just want to put things in perspective.

[ngam]

Does conda-forge have the resources to increase its resource investment in Julia by more than 1000x plus the human time?

No

And even if it does, is the benefit really worth it?

No

I really don't want to be dismissive here, packaging is thankless work, but I just want to put things in perspective.

Totally!

But we should still work together! We will hopefully figure out a way.

[ngam]

@Keno et al, heads up:

I know unsupported builds, etc., but heads up. You use v7.73.0 in your builds, but julia 1.6.x and 1.7.x will segfault with upcoming curl v7.81.0.

Already coming... #43682 🤷

[ngam]

But we should still work together! We will hopefully figure out a way.

Just a side note: This reporting by us and other downstream users will likely save you a lot of human and CI hours, giving you warnings about potential problems before they occur, etc. --- don't you think this is a valuable discussion? That's why it is not just in the spirit of the open-source community, but also for your own benefit. Believe it or not, we will help and make you better... but we can't do that if you don't want us and keep aggressively closing issues and shutting down discussions!

[ngam]

Also, I guess, we could also work together on making sure your test suite is actually good enough to ensure any repackaging can work. Isn't that part of what test suites are for? @mkitti and I are working on enabling all tests in conda-forge/julia-feedstock#172; part of that will be tailoring the test suite so that it runs within a reasonable timeframe (e.g. 3 hours or so) while legitimately ensuring the build is good enough.

[mkitti]

I actually find it quite counter productive to bring up curl on this issue. A more productive conversation would be had in a separate issue requesting or proposing a version bump to of libcurl to 7.81, where the matter could be tracked and discovered by others.