How can I install CxxWrap.jl without libcxxwrap? #309

SylvainCorlay · 2021-12-25T09:57:20Z

(and point to a specific build of libcxxwrap).

An Overrides.toml entry does prevent the installation of the artifacts.

fingolfin · 2021-12-25T17:03:57Z

What do you want to do, exactly? libcxxwrap_julia_jll is an essential dependency of CxxWrap...

SylvainCorlay · 2021-12-25T17:13:48Z

I want to use my own build of it, not the vendored version.

fingolfin · 2021-12-25T19:10:31Z

You'll still have to install the JLL, though. But you can then use the artifacts override system.

there is also an alternate override systems specifically for JLLs, but it is not (yet) documented. @staticfloat will know more

barche · 2021-12-28T22:01:52Z

I just tried with a clean depot, adding a ~/.julia/artifacts/Overrides.toml with these contents:

[3eaa8342-bff7-56a5-9981-c04077f7cee7]
libcxxwrap_julia = "/Users/user/src/build/libcxxwrap-julia"

While this uses the library at the override location, it does still install the artifact from the JLL when adding CxxWrap. Still, using Overrides.toml is right now the only supported way to use alternative binaries.

staticfloat · 2021-12-29T00:38:07Z

You can use Preferences.jl to override libraries one at a time. You'll still download the artifacts, but at runtime, it'll load a different path. It's undocumented, but you can set preferences (owned by the JLL itself) for any of the products defined within it. The relevant code is here, but basically you'd do something like this:

$ julia --project=.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.0 (2021-11-30)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using libtree_jll
       libtree_jll.libtree_path
"/home/sabae/.julia/artifacts/64b9267a1bf36ab73934be621062c521e8a944fd/bin/libtree"

julia> using Preferences

julia> Preferences.set_preferences!(libtree_jll, "libtree_path" => "/tmp/libtree")

julia> 
julia --project=. took 00:00:32 [0]
kholin:tmp.zMV1yPwJf6 sabae [00:35:01]$ julia --project=.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.0 (2021-11-30)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using libtree_jll
       libtree_jll.libtree_path
[ Info: Precompiling libtree_jll [210123e1-2ab9-5e3e-92f0-8698273d9eb6]
"/tmp/libtree"

The set_preferences!() function takes in either a Module, (for an already-loaded package) or a UUID (if you happen to know the UUID of the JLL whose preference you want to modify). It will write out a LocalPreferences.toml file next to your Project.toml that looks like this:

[libtree_jll]
libtree_path = "/tmp/libtree"

It's really quite simple; but the real magic happens the next time Julia tries to load libtree_jll; it will check the list of preferences, discover that something is different, and it will recompile libtree_jll to instead have a different value for libtree_path (which is a compile-time constant!).

This only works with Julia 1.6+, and it requires the JLLs to use JLLWrappers, which basically means that the JLL needs to have been built within the last year or so.

SylvainCorlay · 2021-12-29T18:23:22Z

You'll still download the artifacts, but at runtime, it'll load a different path. It's undocumented, but you can set preferences (owned by the JLL itself) for any of the products defined within it.

Thanks @staticfloat. Now that Julia is properly packaged for conda-forge, I was looking at making conda packages for Julia packages. I think that we should be able to automate the process for pure Julia packages (like it is the case for R and Python packages), but this vendoring of binary dependencies complicates the picture.
If we get things right for CxxWrap, we should be able to find the right pattern. (I think this is especially interesting for things that involve other stacks, such as Qt, Eigen, Xtensor. CxxWrap is also the right place to start for this).

SylvainCorlay · 2021-12-31T17:16:46Z

Maybe a way to generate a conda-based distribution would be to mock the jll packages without vendoring the artifacts.

fingolfin · 2021-12-31T21:01:40Z

But why even bother to do this? You'll play a never ending game of catch up. What is the goal/win here, exactly

SylvainCorlay · 2021-12-31T21:05:50Z

But why even bother to do this? You'll play a never ending game of catch up. What is the goal/win here, exactly

Oh yes conda-forge did catch up on R by automating the process.

The main interest is to have a consistent distribution across the whole stack. From Qt to web browsers including R, Python, and more. It is immensely successful (200M package downloads per month) and that would benefit a lot to julia if it was better integrated.

fingolfin · 2021-12-31T21:11:04Z

i still don't get it; what is inconsistent about JLLs resp their distribution as they are? (No offense intended, of course you don't have to justify yourself to me :-), I am simply curious)

SylvainCorlay · 2022-01-01T08:46:36Z

i still don't get it; what is inconsistent about JLLs resp their distribution as they are? (No offense intended, of course you don't have to justify yourself to me :-), I am simply curious)

There is really nothing inconsistent. It is a completely different approach. Vendoring binary dependencies in Julia packages has a limit. Even though it is certainly possible, I don't think it is reasonable to vendor Qt, Chromium, GDAL, or CPython in a Julia package. Really, such packages should be peers to Julia in a broader packaging sense.

There is also a social aspect to this in that it duplicates efforts to make a BinaryBuilder recipe for such a package while there is a cross-community effort to build these stacks in a binary-compatible way across tens of thousands of packages. Integrating in such a cross-language effort would help Julia become more a glue language and less an island. You could start seeing popular packages embed a Julia interpreter just like Blender and FreeCAD embed Python.

barche · 2022-01-02T22:11:47Z

I think this is a bit beyond the scope of CxxWrap, but it seems this post enumerates the options quite nicely:

conda-forge/julia-feedstock#14 (comment)

I agree it would be nice to have a way to avoid downloading the artifacts though, some chains of dependencies (such as Qt) are quite big.

mkitti · 2022-01-03T07:25:32Z

@SylvainCorlay One idea is that conda-forge could just provide a forked version of libcxxwrap_julia_jll.

Essentially,

Fork https://github.com/JuliaBinaryWrappers/libcxxwrap_julia_jll.jl .
Customize it to point to conda-forge binaries. Then use Pkg.add with a path or URL pointing to the conda-forge fork.

All CxxWrap.jl cares about is there is a package called libcxxwrap_julia_jll with the appropriate UUID. It is not specified where that package must come from.

From the user perspective, the place the package was installed from will be noted in the user's project's Manifest.toml.

SylvainCorlay · 2022-01-03T08:11:50Z

@SylvainCorlay One idea is that conda-forge could just provide a forked version of libcxxwrap_julia_jll.

Absolutely, that's what I meant by "Maybe a way to generate a conda-based distribution would be to mock the jll packages without vendoring the artifacts."

Ideally, we should be able to automate that for all jll packages produced by BinaryBuilder.jl.

mkitti · 2022-01-03T09:41:27Z

Absolutely, that's what I meant by "Maybe a way to generate a conda-based distribution would be to mock the jll packages without vendoring the artifacts."

Here's a proof of concept:
https://github.com/mkitti/libcxxwrap_julia_jll_mock

First I downloaded https://github.com/JuliaBinaryWrappers/libcxxwrap_julia_jll.jl/releases/download/libcxxwrap_julia-v0.8.8+1/libcxxwrap_julia.v0.8.8.x86_64-linux-gnu-cxx11-julia_version+1.7.0.tar.gz and untarred it to /home/mkitti/src/libcxxwrap_julia_jll_mock/artifact/. I got this from libcxxwrap_julia_jll's Artifacts.toml.

# Enter Pkg REPL by pressing ]

(@julia171_test) pkg> activate --temp
  Activating new project at `/tmp/jl_WT0Yu5`

(jl_WT0Yu5) pkg> add https://github.com/mkitti/libcxxwrap_julia_jll_mock
     Cloning git-repo `https://github.com/mkitti/libcxxwrap_julia_jll_mock`
    Updating git-repo `https://github.com/mkitti/libcxxwrap_julia_jll_mock`
    Updating registry at `~/anaconda3/envs/julia171_test/share/julia/registries/General.toml`
   Resolving package versions...
    Updating `/tmp/jl_WT0Yu5/Project.toml`
  [3eaa8342] + libcxxwrap_julia_jll v0.8.8+1 `https://github.com/mkitti/libcxxwrap_julia_jll_mock#master`
    Updating `/tmp/jl_WT0Yu5/Manifest.toml`
  [3eaa8342] + libcxxwrap_julia_jll v0.8.8+1 `https://github.com/mkitti/libcxxwrap_julia_jll_mock#master`

(jl_WT0Yu5) pkg> add CxxWrap
   Resolving package versions...
    Updating `/tmp/jl_WT0Yu5/Project.toml`
  [1f15a43c] + CxxWrap v0.11.2
    Updating `/tmp/jl_WT0Yu5/Manifest.toml`
  [1f15a43c] + CxxWrap v0.11.2
  [1914dd2f] + MacroTools v0.5.9
  [2a0f44e3] + Base64
  [8f399da3] + Libdl
  [d6f4376e] + Markdown
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization

(jl_WT0Yu5) pkg> precompile

(jl_WT0Yu5) pkg> status --manifest
      Status `/tmp/jl_WT0Yu5/Manifest.toml`
  [1f15a43c] CxxWrap v0.11.2
  [1914dd2f] MacroTools v0.5.9
  [3eaa8342] libcxxwrap_julia_jll v0.8.8+1 `https://github.com/mkitti/libcxxwrap_julia_jll_mock#master`
  [2a0f44e3] Base64
  [8f399da3] Libdl
  [d6f4376e] Markdown
  [9a3f8284] Random
  [ea8e919c] SHA
  [9e88b42a] Serialization

julia> using CxxWrap

julia> CxxWrap.CxxWrapCore.libcxxwrapversion()
v"0.8.3"

julia> CxxWrap.CxxWrapCore.libcxxwrap_julia_jll.libcxxwrap_julia_path
"/home/mkitti/src/libcxxwrap_julia_jll_mock/artifact/lib/libcxxwrap_julia.so"

mkitti · 2022-01-03T09:43:09Z

Since you already have created https://github.com/conda-forge/libcxxwrap-julia-feedstock

(@julia171_test) pkg> activate --temp
  Activating new environment at `/tmp/jl_LWs50i/Project.toml`

(jl_LWs50i) pkg> add https://github.com/mkitti/libcxxwrap_julia_jll_mock.git#cf
    Updating git-repo `https://github.com/mkitti/libcxxwrap_julia_jll_mock.git`
    Updating registry at `~/anaconda3/envs/julia171_test/share/julia/registries/General`
   Resolving package versions...
    Updating `/tmp/jl_LWs50i/Project.toml`
  [3eaa8342] + libcxxwrap_julia_jll v0.8.8+1 `https://github.com/mkitti/libcxxwrap_julia_jll_mock.git#cf`
    Updating `/tmp/jl_LWs50i/Manifest.toml`
  [3eaa8342] + libcxxwrap_julia_jll v0.8.8+1 `https://github.com/mkitti/libcxxwrap_julia_jll_mock.git#cf`
  Progress [========================================>]  1/1
1 dependency successfully precompiled in 2 seconds

(jl_LWs50i) pkg> add CxxWrap
   Resolving package versions...
    Updating `/tmp/jl_LWs50i/Project.toml`
  [1f15a43c] + CxxWrap v0.11.2
    Updating `/tmp/jl_LWs50i/Manifest.toml`
  [1f15a43c] + CxxWrap v0.11.2
  [1914dd2f] + MacroTools v0.5.9
  [2a0f44e3] + Base64
  [8f399da3] + Libdl
  [d6f4376e] + Markdown
  [9a3f8284] + Random
  [9e88b42a] + Serialization
  Progress [========================================>]  1/1
1 dependency successfully precompiled in 11 seconds (2 already precompiled)

julia> using CxxWrap

julia> CxxWrap.CxxWrapCore.libcxxwrap_julia_jll.libcxxwrap_julia_path
"/home/mkitti/anaconda3/envs/julia171_test/lib/libcxxwrap_julia.so"

SylvainCorlay · 2022-01-03T09:45:49Z

Brilliant!

mkitti · 2022-01-03T23:34:53Z

@barche I think this may be closeable now as Sylvain now has multiple options above. Perhaps one remaining aspect would be to document how CxxWrap.jl uses libcxxwrap_julia_jll.jl.

My current understanding based on the minimalist mock package I produced above is as follows.

Use of libcxxwrap_julia_jll is contained within CxxWrap.CxxWrapCore and CxxWrap.StdLib

CxxWrap.jl/src/CxxWrap.jl

Line 15 in 8fa28ca

using libcxxwrap_julia_jll # for libcxxwrap_julia and libcxxwrap_julia_stl

CxxWrap.jl/src/StdLib.jl

Line 16 in cb14a4e

import libcxxwrap_julia_jll
Two exported symbols from libcxxwrap_julia_jll are used: libcxxwrap_julia and libcxxwrap_julia_stl
CxxWrap.jl depends on libcxxwrap_julia_jll.libcxxwrap_julia_path

CxxWrap.jl/src/CxxWrap.jl

Line 17 in 8fa28ca

if !isdefined(libcxxwrap_julia_jll, :libcxxwrap_julia_path)
All three symbols (libcxxwrap_julia, libcxxwrap_julia_stl, libcxxwrap_julia_path) above are expected to be a subtype of AbstractString.

The symbols above are generated by
https://github.com/JuliaPackaging/JLLWrappers.jl/blob/master/src/products/library_generators.jl
For example, see their use in
https://github.com/JuliaBinaryWrappers/libcxxwrap_julia_jll.jl/blob/main/src/wrappers/x86_64-linux-gnu-cxx11-julia_version%2B1.7.0.jl

mkitti · 2022-01-04T01:09:39Z

xref: JuliaLang/Pkg.jl#2664 "Don't download artifact when overwritten via Overrides.toml"

SylvainCorlay · 2022-01-04T02:21:09Z

Thanks for posting all the details for mocking libcxxwrap_julia_jll. The mock package is really simple. I wonder if this could lead to a systematic approach to this for the ~950 jll packages of the https://github.com/JuliaBinaryWrappers/ GitHub organization.

Maybe JuliaLang/Pkg.jl#2664 would be a better solution for that - however one issue with Overrides.toml is that it is a single file that needs to be edited for each jll, which is not great for a package manager like conda or apt-get that knows really well how to add files. Maybe I should open an issue suggesting to allow the use of a configuration directory approach rather than a configuration file (such as Overrides.d or one override file per package, that could be joined eventually).

fingolfin · 2022-01-04T06:10:16Z

I understand your reasoning for vendoring things like Qt. But still not for, well, libcxxwrap, which is Julia specific... I mean I can understand why a package manager wants to install the artifact data itself, instead of letting Julia download it, and not in ~/.julia. (which should be relatively easy to achieve by adding a separate JULIA_DEPOT). But it sounds as if you want to also substitute different binaries. Or maybe I misunderstand?

If it really is to be a custom build, you'll have to be very careful to apply the same patches and tweaks to packages, else things will break legt and right. E.g. GMP is patched to add an API for installing an alloc overflow callback. There are many more packages with custom tweaks.

mkitti · 2022-01-04T07:21:35Z

I wonder if this could lead to a systematic approach to this for the ~950 jll packages

I posted an outline on approaches to this here: conda-forge/julia-feedstock#164 (comment)
Essentially, one approach could be to produce a CondaForgeJLLWrappers.jl version of JLLWrappers.jl that provides the same API with a distinct implementation. In particular, one would just have find_artifact_dir point to $CONDA_PREFIX. One would then just swap the dependencies.

Additionally, Julia allows for packages to exist within subdirectories of a git repository, so one could presumably embed the alternate JLL within the feedstock repository, https://github.com/conda-forge/xtensor-julia-feedstock repository, for example.

An alternative would be to just improve the overrides system to avoid download unneeded artifacts and make it easier to manage either through an overrides.d directory based system or an API.

edit: Also there is talk of using https://github.com/JuliaPackaging/Preferences.jl to provide per-package and per-project based options that may affect how artifacts are downloaded or not.

mkitti · 2022-01-04T07:55:32Z

If it really is to be a custom build, you'll have to be very careful to apply the same patches and tweaks to packages, else things will break legt and right. E.g. GMP is patched to add an API for installing an alloc overflow callback. There are many more packages with custom tweaks.

The missing background is that https://github.com/conda-forge/julia-feedstock is moving along. I am one of the lead maintainers, partially because I am one of the few with active Julia experience. There is currently a working build of Juila 1.7.1. Sylvain has C++ packages distributed via conda-forge, such as xtensor, so it makes sense to be able to distribute the Julia bindings via conda-forge as well. The Julia bindings depend on CxxWrap.jl. Hence, the question is whether the underlying C++ dependency could be distributed via conda-forge also.

I understand there are some willingness within conda-forge to apply specific patches. Perhaps others may be able to benefit from them as well? For GMP, the place to apply them would be https://github.com/conda-forge/gmp-feedstock . I would also appreciate if you could give me an example of behavior is that is broken without those patches as there are likely issues in https://github.com/conda-forge/julia-feedstock that I am not aware of. I'm not sure why the patches are not applied upstream, but perhaps wider distribution of those patches may convince upstream to adopt them.

Overall the endeavor is fraught with issues as you point out. There is some redundancy. However, I think there is room to work cooperatively in this area. Working within and between NumFocus projects makes complete sense. We are sending patches back and forth. In the process of adapting Julia for conda-forge, we have added and modified tests that have arisen. Ultimately, the end result is a flexible and robust Julia with a wide user base.

For the particular issue of artifacts, conda-forge is not alone in this desire to use "system" artifacts rather than vendored ones. Any Linux distribution that has a Julia package has similar issues.

mkitti · 2022-01-04T08:02:01Z

I encourage the conda-forge specific discussion to continue in conda-forge/julia-feedstock#164.
The discussion about Julia's artifacts and overrides is perhaps best continued in JuliaLang/Pkg.jl#2664

This is not my package, but I think I will exercise my membership in JuliaInterop to close the issue. Feel free to reopen it if there is something I have overlooked. Thank you for the discussion.

SylvainCorlay mentioned this issue Dec 28, 2021

Use libcxxwrap-julia directly xtensor-stack/xtensor-julia#29

Merged

ngam mentioned this issue Jan 2, 2022

binary compatibility between conda and julia conda-forge/julia-feedstock#164

Closed

simonbyrne mentioned this issue Jan 3, 2022

Add a lightweight MPIABI sub-package that controls the MPI ABI JuliaParallel/MPI.jl#529

Closed

mkitti mentioned this issue Jan 3, 2022

mistake in readme + conda julia discussion xtensor-stack/xtensor-julia#41

Closed

mkitti closed this as completed Jan 4, 2022

mkitti mentioned this issue Jan 11, 2022

julia: enable all tests + stricter pinning conda-forge/julia-feedstock#172

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How can I install CxxWrap.jl without libcxxwrap? #309

How can I install CxxWrap.jl without libcxxwrap? #309

SylvainCorlay commented Dec 25, 2021

fingolfin commented Dec 25, 2021

SylvainCorlay commented Dec 25, 2021

fingolfin commented Dec 25, 2021

barche commented Dec 28, 2021

staticfloat commented Dec 29, 2021

SylvainCorlay commented Dec 29, 2021

SylvainCorlay commented Dec 31, 2021

fingolfin commented Dec 31, 2021

SylvainCorlay commented Dec 31, 2021

fingolfin commented Dec 31, 2021

SylvainCorlay commented Jan 1, 2022

barche commented Jan 2, 2022

mkitti commented Jan 3, 2022 •

edited

Loading

SylvainCorlay commented Jan 3, 2022

mkitti commented Jan 3, 2022

mkitti commented Jan 3, 2022

SylvainCorlay commented Jan 3, 2022

mkitti commented Jan 3, 2022

mkitti commented Jan 4, 2022

SylvainCorlay commented Jan 4, 2022

fingolfin commented Jan 4, 2022

mkitti commented Jan 4, 2022 •

edited

Loading

mkitti commented Jan 4, 2022 •

edited

Loading

mkitti commented Jan 4, 2022

How can I install CxxWrap.jl without libcxxwrap? #309

How can I install CxxWrap.jl without libcxxwrap? #309

Comments

SylvainCorlay commented Dec 25, 2021

fingolfin commented Dec 25, 2021

SylvainCorlay commented Dec 25, 2021

fingolfin commented Dec 25, 2021

barche commented Dec 28, 2021

staticfloat commented Dec 29, 2021

SylvainCorlay commented Dec 29, 2021

SylvainCorlay commented Dec 31, 2021

fingolfin commented Dec 31, 2021

SylvainCorlay commented Dec 31, 2021

fingolfin commented Dec 31, 2021

SylvainCorlay commented Jan 1, 2022

barche commented Jan 2, 2022

mkitti commented Jan 3, 2022 • edited Loading

SylvainCorlay commented Jan 3, 2022

mkitti commented Jan 3, 2022

mkitti commented Jan 3, 2022

SylvainCorlay commented Jan 3, 2022

mkitti commented Jan 3, 2022

mkitti commented Jan 4, 2022

SylvainCorlay commented Jan 4, 2022

fingolfin commented Jan 4, 2022

mkitti commented Jan 4, 2022 • edited Loading

mkitti commented Jan 4, 2022 • edited Loading

mkitti commented Jan 4, 2022

mkitti commented Jan 3, 2022 •

edited

Loading

mkitti commented Jan 4, 2022 •

edited

Loading

mkitti commented Jan 4, 2022 •

edited

Loading