reorganize & cleanup

[ngam]

Closes #67
Closes #14
Closes #140
Closes #87
Closes #160
Closes #138
Closes #76

TODO:

• isolate julia inside conda
• sync julia project with conda env
• stack julia envs

Checklist

• Used a personal fork of the feedstock to propose changes
• Bumped the build number (if the version is unchanged)
• Reset the build number to 0 (if the version changed)
• Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
• Ensured the license file is being packaged.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[ngam]

@mkitti about the tests, should we try to expand them?

# all except stdlib, add "opaque_closure", "filesystem" with 1.7
# [not osx] due to failure on math.jl:296, pre-haswell processor

[ngam]

@mkitti feel free to review and merge. Or let me know if there are other cleanup/organization things we should deal with first. We should also move this to the other branch (v1.6.5) once we settle on a procedure.

Should we expand the tests above?

[ngam]

@conda-forge-admin, please rerender

[github-actions]

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/julia-feedstock/actions/runs/1622318079.

[ngam]

These pass just fine locally: # all except stdlib, add "opaque_closure", "filesystem" with 1.7

[ngam]

Okay, all look good --- will test the stdlib as well soon

[mkitti]

1.7.2 is probably weeks to a few months away. 1.7.0 and 1.7.1 were separated by three weeks.

1.8.0 is likely 6 months to a year away.

[mkitti]

Why are we keeping the old juliarc.jl and pre-unlink.sh around?

Is there any functionality in pre-unlink.sh that should be replicate?

[mkitti]

I'm good to merge this if you are, @ngam .

[ngam]

Should we address #161 or wait?

[ngam]

@conda-forge-admin, please rerender

[github-actions]

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/julia-feedstock/actions/runs/1635585005.

[zklaus]

It's great to see the rapid improvements coming in here! Big steps forward on many fronts.

But if you want a bit more community engagement you might want to consider slowing down just a little bit. This PR, which closes 7 issues, some of which have been around for a long time and have been opened and worked upon by other people, took all of 5 days from creation to merging. Over the Christmas days.
Still, I don't want to dampen the enthusiasm; it really is great to see this move forward.

For now, I have two concerns/comments:

The issue of separate depots for separate environments is really all about binary compatibility. Basically, anything that has been compiled for one environment cannot reliably be used with another environment because linked libraries may have changed in an ABI incompatible way (cf the recent libunwind problems, but I would expect this with a number of libraries, e.g. for I/O of special file formats like netCDF) and the compilers themselves may be different. If compiled bits are shared, this will lead to surprising segfaults in the future. This kind of problem is not likely to show up in tests that use similar environments created at roughly the same time, but is virtually guaranteed if very different environments with library overlap are used, particularly when those environments are created a long time apart.
To me, it is not clear to what degree compiled bits are an integral part of the Julia depots, but this is my main problem with ~/.julia currently (i.e. conda-forge Julia <=1.6).

My second concern is that it is not completely clear to me to what degree this PR really addresses the issues it closes. For example, #14 is about packaging Julia packages for conda-forge, i.e. how should I create the conda-forge feedstock julia-rainfarm from https://github.com/jhardenberg/RainFARM.jl. While sorting out the depot paths certainly helps and might be necessary, from reading this PR I still don't know how to do this or where to look up the documentation. In terms of documentation, at a minimum I would expect a bullet point in the enumeration for new packages.
This is probably beyond the scope of this PR, but in that case we might want to keep #14 open for now.

[ngam]

But if you want a bit more community engagement you might want to consider slowing down just a little bit. This PR, which closes 7 issues, some of which have been around for a long time and have been opened and worked upon by other people, took all of 5 days from creation to merging. Over the Christmas days.

Yeah, sorry... But, I'd rather interested people (like yourself) come back and reopen the issues and explain their current needs/expectations. It's very difficult for us to guess what people were thinking about 4+ years ago especially as the ecosystems (julia as well as conda) have moved so much since then. Obviously, we couldn't address all concerns (and you bring up really important ones) --- to help us push forward, could you please reopen issues/PRs that you think weren't adequately addressed and add your perspective?

I will start by opening two issues based on your two main concerns and we will work --- together and less rapidly --- to address them.

To me, it is not clear to what degree compiled bits are an integral part of the Julia depots, but this is my main problem with ~/.julia currently (i.e. conda-forge Julia <=1.6).

For this, @mkitti was of the opinion that we shouldn't change anything in the 1.6.x and just bring all newer changes to 1.7.1+. I concurred with his view (we don't want to mess up the setups of people who got comfortable with those versions...)

[ngam]

My second concern is that it is not completely clear to me to what degree this PR really addresses the issues it closes. For example, #14 is about packaging Julia packages for conda-forge, i.e. how should I create the conda-forge feedstock julia-rainfarm from https://github.com/jhardenberg/RainFARM.jl

I certainly didn't read it this way. I am not sure if this is even applicable now that julia has their own ecosystem and package-building setup. Our real goal here was to better isolate julia inside conda envs, so that a user can have many different julias (via conda) and also a separate standalone julia smoothly. How the user ends up using such isolation is obviously none of our business, right? And I am not sure if we should pursue changes to actually enmesh julia and conda more tightly; again, my thinking/perspective was to better isolate them from each other (not vice versa). But that's just one single opinion and I am happy to discuss this further and work on bringing changes/improvements you'd like to see :)

[ngam]

7 issues

only 3. The others are PRs.

[mkitti]

Native machine code is not cached by Julia during normal operation, so I do not expect there to be be ABI issues for compiled code generated from Julia unless one is making direct accalls. What is cached is lowered code and type inference on a per Julia environment basis in $JULIA_DEPOT/compiled. That said this does change between Julia versions and is under active development.

Thr chosen defaults for the 1.7 releases are quite conservative with unique Julia depots and environments for each conda environment. My concern is actually that is too conservative and that we are duplicating too many things.

The use of PackageCompiler.jl, which actually involves native code generation and storage is another matter, but its location of the generated system image shared library is determined by the user.

[zklaus]

Thanks for your replies, @ngam and @mkitti. I'll pick up the discussion in #14, #164 and possibly new issues as needed.