Make SparseArrays a weak dependency #134

IanButterworth · 2022-12-07T21:28:25Z

Uses the recently merged JuliaLang/julia#47695 to set up SparseArrays as a weak dependency of Statistics, via a package extension.

All code was copied across to the package extension untouched, with the addition of some imports.

This is backwards compatible. On older julia versions that don't support package extensions, SparseArrays will be loaded as per normal.

What this would look like on julia master, if SparseArrays is removed from the sysimage

julia> @time using Statistics
  0.011658 seconds (13.76 k allocations: 1.030 MiB)

julia> @time using SparseArrays
  4.618848 seconds (7.05 M allocations: 980.989 MiB, 1.46% compilation time)

cc. @KristofferC

codecov-commenter · 2022-12-07T22:40:37Z

Codecov Report

Base: 96.94% // Head: 97.26% // Increases project coverage by +0.31% 🎉

Coverage data is based on head (8ab238a) compared to base (e9ac70b).
Patch has no changes to coverable lines.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #134      +/-   ##
==========================================
+ Coverage   96.94%   97.26%   +0.31%     
==========================================
  Files           1        1              
  Lines         426      365      -61     
==========================================
- Hits          413      355      -58     
+ Misses         13       10       -3

Impacted Files	Coverage Δ
src/Statistics.jl	`97.26% <ø> (+0.31%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

src/Statistics.jl

IanButterworth · 2022-12-09T01:18:18Z

@ViralBShah good to go ahead with this, and remove SparseArrays from the sysimage once bumped on julia master? (I don't have merge rights here)

KristofferC · 2022-12-09T10:00:18Z

good to go ahead with this

Didn't we find out that extensions didn't work right now for stdlibs? To me, it only makes sense to merge this once Statistics.jl is not an stdlib?

vtjnash · 2023-08-31T20:46:45Z

It seemed to work for me, so I think we should try this

KristofferC · 2023-09-01T07:28:16Z

Okay, let's go with this then. Thanks for testing!

nalimilan · 2023-09-15T18:01:32Z

Unfortunately this fails if I bump Statistics to the latest commit on Julia master. And indeed even after using SparseArrays, the cov method defined in the extension isn't used for some reason.

   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-DEV.473 (2023-09-15)
 _/ |\__'_|_|_|\__'_|  |  Commit 42d98a2ed14* (0 days old master)
|__/                   |

julia> Base.runtests(["Statistics"])
Running parallel tests with:
  nworkers() = 1
  nthreads() = 1
  Sys.CPU_THREADS = 4
  Sys.total_memory() = 15.282 GiB
  Sys.free_memory() = 7.305 GiB

Test   (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
Statistics  (1) |        started at 2023-09-15T19:57:33.267
Statistics  (1) |         failed at 2023-09-15T19:58:48.347
Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:963
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 1, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 1, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 1 … 1 1; … ; 0 1 … 1 1; 0 1 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:964
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 1, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 1, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 1 … 1 1; … ; 0 1 … 1 1; 0 1 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:963
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 1, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 1, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 1 … 1 1; … ; 0 1 … 1 1; 0 1 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:964
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 1, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 1, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 1 … 1 1; … ; 0 1 … 1 1; 0 1 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:963
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 1, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 1, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 1 … 1 1; … ; 0 1 … 1 1; 0 1 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:964
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 1, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 1, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 1 … 1 1; … ; 0 1 … 1 1; 0 1 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:963
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 1, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 1, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 1 … 1 1; … ; 0 1 … 1 1; 0 1 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:964
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 1, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 1, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 1 … 1 1; … ; 0 1 … 1 1; 0 1 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:978
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 0, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 0, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 0 … 0 0; … ; 0 0 … 1 1; 0 0 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:979
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 0, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 0, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 0 … 0 0; … ; 0 0 … 1 1; 0 0 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:978
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 0, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 0, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 0 … 0 0; … ; 0 0 … 1 1; 0 0 … 1 1]

Test Failed at /home/milan/Dev/julia/usr/share/julia/stdlib/v1.11/Statistics/test/runtests.jl:979
  Expression: isfinite.(cov_sparse) == isfinite.(cov_dense)
   Evaluated: sparse([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1  …  10, 10, 10, 10, 10, 10, 10, 10, 10, 10], Bool[0, 0, 0, 0, 0, 1, 1, 0, 0, 0  …  0, 0, 1, 1, 1, 1, 1, 1, 1, 1], 10, 10) == Bool[0 0 … 0 0; 0 0 … 0 0; … ; 0 0 … 1 1; 0 0 … 1 1]



Test Summary: | Pass  Fail  Total     Time
  Overall     |  821    12    833  1m16.5s
    Statistics |  821    12    833  1m15.7s
    FAILURE

The global RNG seed was 0x7eb6b98a471ff52f1a5a23d7acd773bb.

Some groundwork for JuliaStats/Statistics.jl#134 which is bumped (and thus tested) in #52431 that will be merged after this

KristofferC · 2023-12-19T12:26:47Z

@nalimilan, I think this is fixed now with JuliaLang/julia#52428

IanButterworth force-pushed the ib/glue branch from e568806 to f3f7b18 Compare December 7, 2022 22:05

IanButterworth force-pushed the ib/glue branch 3 times, most recently from 35c3fff to 4c3e6ac Compare December 8, 2022 00:24

IanButterworth commented Dec 8, 2022

View reviewed changes

src/Statistics.jl Outdated Show resolved Hide resolved

make SparseArrays a weak dependency

61dcb12

IanButterworth force-pushed the ib/glue branch from 8ab238a to 61dcb12 Compare December 9, 2022 00:54

dkarrasch mentioned this pull request Jul 6, 2023

Handle SparseArrays.jl dep as Pkg extension #146

Closed

vtjnash mentioned this pull request Aug 31, 2023

move SparseArrays to an extension #149

Closed

KristofferC merged commit 81a90af into JuliaStats:master Sep 1, 2023

IanButterworth deleted the ib/glue branch September 1, 2023 10:36

This was referenced Dec 6, 2023

bad/incorrect/missing handling of stdlibs in Manifest JuliaLang/Pkg.jl#3719

Closed

loading: support extensions for stdlibs (implicit env) JuliaLang/julia#52428

Merged

vtjnash added a commit to JuliaLang/julia that referenced this pull request Dec 8, 2023

loading: support extensions for stdlibs (an implicit env) (#52428)

9723de5

Some groundwork for JuliaStats/Statistics.jl#134 which is bumped (and thus tested) in #52431 that will be merged after this

giordano mentioned this pull request Aug 5, 2024

Try to improve package load time carstenbauer/ThreadPinning.jl#55

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make SparseArrays a weak dependency #134

Make SparseArrays a weak dependency #134

IanButterworth commented Dec 7, 2022 •

edited

Loading

codecov-commenter commented Dec 7, 2022 •

edited

Loading

IanButterworth commented Dec 9, 2022

KristofferC commented Dec 9, 2022

vtjnash commented Aug 31, 2023

KristofferC commented Sep 1, 2023

nalimilan commented Sep 15, 2023

KristofferC commented Dec 19, 2023

Make SparseArrays a weak dependency #134

Make SparseArrays a weak dependency #134

Conversation

IanButterworth commented Dec 7, 2022 • edited Loading

codecov-commenter commented Dec 7, 2022 • edited Loading

Codecov Report

IanButterworth commented Dec 9, 2022

KristofferC commented Dec 9, 2022

vtjnash commented Aug 31, 2023

KristofferC commented Sep 1, 2023

nalimilan commented Sep 15, 2023

KristofferC commented Dec 19, 2023

IanButterworth commented Dec 7, 2022 •

edited

Loading

codecov-commenter commented Dec 7, 2022 •

edited

Loading