download ca cert error on Linux on Julia 1.10.1

[jaakkor2]

On Julia 1.10.1, download regressed compared to Julia 1.10.0. This is on empty depot. Side effect is that registries cannot be updated (]up).

julia> download("https://julialang.org/")
ERROR: RequestError: Error reading ca cert file /etc/ssl/certs/ca-certificates.crt - mbedTLS: (-0x3E00) PK - Read/write of file failed while requesting https://julialang.org/
Stacktrace:
  [1] (::Downloads.var"#9#18"{…})(easy::Downloads.Curl.Easy)
    @ Downloads ~/julia/julia-1.10.1/share/julia/stdlib/v1.10/Downloads/src/Downloads.jl:413
  [2] with_handle(f::Downloads.var"#9#18"{…}, handle::Downloads.Curl.Easy)
    @ Downloads.Curl ~/julia/julia-1.10.1/share/julia/stdlib/v1.10/Downloads/src/Curl/Curl.jl:95
  [3] #8
    @ ~/julia/julia-1.10.1/share/julia/stdlib/v1.10/Downloads/src/Downloads.jl:354 [inlined]
  [4] arg_write(f::Downloads.var"#8#17"{…}, arg::IOStream)
    @ ArgTools ~/julia/julia-1.10.1/share/julia/stdlib/v1.10/ArgTools/src/ArgTools.jl:134
  [5] #7
    @ ~/julia/julia-1.10.1/share/julia/stdlib/v1.10/Downloads/src/Downloads.jl:353 [inlined]
  [6] arg_read
    @ ~/julia/julia-1.10.1/share/julia/stdlib/v1.10/ArgTools/src/ArgTools.jl:76 [inlined]
  [7] request(url::String; input::Nothing, output::IOStream, method::Nothing, headers::Vector{…}, timeout::Float64, progress::Nothing, verbose::Bool, debug::Nothing, throw::Bool, downloader::Nothing)
    @ Downloads ~/julia/julia-1.10.1/share/julia/stdlib/v1.10/Downloads/src/Downloads.jl:352
  [8] request
    @ ~/julia/julia-1.10.1/share/julia/stdlib/v1.10/Downloads/src/Downloads.jl:320 [inlined]
  [9] #3
    @ ~/julia/julia-1.10.1/share/julia/stdlib/v1.10/Downloads/src/Downloads.jl:258 [inlined]
 [10] arg_write(f::Downloads.var"#3#4"{Nothing, Vector{…}, Float64, Nothing, Bool, Nothing, Nothing, String}, arg::Nothing)
    @ ArgTools ~/julia/julia-1.10.1/share/julia/stdlib/v1.10/ArgTools/src/ArgTools.jl:123
 [11] #download#2
    @ ~/julia/julia-1.10.1/share/julia/stdlib/v1.10/Downloads/src/Downloads.jl:257 [inlined]
 [12] download(url::String, output::Nothing)
    @ Downloads ~/julia/julia-1.10.1/share/julia/stdlib/v1.10/Downloads/src/Downloads.jl:246
 [13] #invokelatest#2
    @ ./essentials.jl:892 [inlined]
 [14] invokelatest
    @ ./essentials.jl:889 [inlined]
 [15] do_download(url::String, path::Nothing)
    @ Base ./download.jl:24
 [16] download(url::String)
    @ Base ./download.jl:20
 [17] top-level scope
    @ REPL[2]:1
Some type information was truncated. Use `show(err)` to see complete types.

on

$ cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)

julia> versioninfo()
Julia Version 1.10.1
Commit 7790d6f0641 (2024-02-13 20:41 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 64 × AMD EPYC 7371 16-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver1)
Threads: 8 default, 0 interactive, 4 GC (on 64 virtual cores)
Environment:
  JULIA_DEPOT_PATH = /home/jaakkor2/julia/depo1

There is no /etc/ssl/certs/ca-certificates.crt on this system.

[lbilli]

I get just a warning, but similar issue:

(@v1.10) pkg> up
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = RequestError: Error reading ca cert file /etc/ssl/certs/ca-certificates.crt - mbedTLS: (-0x3E00) PK - Read/write of file failed while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry ~/apps/julia-1.10.1/share/julia/stdlib/v1.10/Pkg/src/Registry/Registry.jl:69
    Updating registry at `~/.julia/registries/General.toml`
  No Changes to `~/.julia/environments/v1.10/Project.toml`
  No Changes to `~/.julia/environments/v1.10/Manifest.toml`

Still, it's a regression compared to v1.10

julia> versioninfo()
Julia Version 1.10.1
Commit 7790d6f0641 (2024-02-13 20:41 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × Intel(R) Core(TM) i3-7100 CPU @ 3.90GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, skylake)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)

[jaakkor2]

@atbug wrote on Slack workaround

export JULIA_SSL_CA_ROOTS_PATH=/etc/ssl/certs/ca-bundle.crt

[KristofferC]

We did bump the Downloads and NetworkOptions stdlibs for 1.10.1 so could have something to do with that.

[benlorenz]

This is probably the same as JuliaLang/NetworkOptions.jl#33 which got triggered by JuliaLang/Downloads.jl#226 in the backports for 1.10.

[KristofferC]

I thought that was fixed by JuliaLang/NetworkOptions.jl#31?

[benlorenz]

I thought that was fixed by JuliaLang/NetworkOptions.jl#31?

No, that fix was only for BUNDLED_KNOWN_HOSTS_FILE but this is SYSTEM_CA_ROOTS:

$ julia-1.10.0/bin/julia -e 'using NetworkOptions; println(NetworkOptions.SYSTEM_CA_ROOTS);'
Base.RefValue{String}(#undef)
$ julia-1.10.1/bin/julia -e 'using NetworkOptions; println(NetworkOptions.SYSTEM_CA_ROOTS);'
Base.RefValue{String}("/etc/ssl/certs/ca-certificates.crt")

(this is on opensuse where this file does not exist)

[KristofferC]

Okay.. that's unfortunate

[KristofferC]

JuliaLang/NetworkOptions.jl#34 should hopefully fix that.

[j-fu]

For me, on opensuse, this popped up when trying to access a second registry besides General.

The band-aid described in
#41019 (comment)
fixed this for me for the moment.

[tpoisot]

Same as @j-fu -- this problem pops up on openSUSE, is specific to 1.10.1 as far as I can tell, and is also fixed with the same trick in the comment linked. It still gives a warning, but it can install packages.

The warning I get is:

   Resolving package versions...
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = RequestError: Error reading ca cert file /etc/ssl/certs/ca-bundle.crt - mbedTLS: (-0x3E00) PK - Read/write of file failed while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry ~/.julia/juliaup/julia-1.10.1+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Registry/Registry.jl:69
     Cloning [8bb1440f-4735-579b-a4ab-409b98df4dab] DelimitedFiles from https://github.com/JuliaData/DelimitedFiles.jl.git
   Installed DelimitedFiles ─ v1.9.1

[JBlaschke]

Ha! At least regressions such as this make you learn new and fun things about your system. E.g. HPE/Cray like to put theirs here: /etc/mft/ca-bundle.crt

Why? No clue!

Edit: after talking to the sysadmins (wow! they work weekends), if you're working on NERSC Perlmutter, this is the path: /etc/ssl/ca-bundle.pem -- I'm adding it to the Julia modules now. It might take until the next maintenance to stage the changes into production, in the meantime you can use the dev modules here:

ml use /global/common/software/nersc/n9/julia/modules

[bclyons12]

This is affecting some of our installations right now as well. I see some commits being made. Do we need to wait for a new release version to have this fixed? Right now we're just avoiding upgrading all together.

[melodyjulia]

I run into the same issue here.

I get just a warning, but similar issue:

(@v1.10) pkg> up
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = RequestError: Error reading ca cert file /etc/ssl/certs/ca-certificates.crt - mbedTLS: (-0x3E00) PK - Read/write of file failed while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry ~/apps/julia-1.10.1/share/julia/stdlib/v1.10/Pkg/src/Registry/Registry.jl:69
    Updating registry at `~/.julia/registries/General.toml`
  No Changes to `~/.julia/environments/v1.10/Project.toml`
  No Changes to `~/.julia/environments/v1.10/Manifest.toml`

Still, it's a regression compared to v1.10

[KristofferC]

Do we need to wait for a new release version to have this fixed? Right now we're just avoiding upgrading all together.

Yes, we will put out a 1.10.2 shortly with the fix included.

[jaantollander]

We also run into this issue while running Julia 1.10.0 in the LUMI supercomputer.
It seems that in LUMI, the certificate is also named /etc/ssl/ca-bundle.pem which is a symlink to /var/lib/ca-certificates/ca-bundle.pem.
We used the environment variable to fix the issue for now.

export JULIA_SSL_CA_ROOTS_PATH="/etc/ssl/ca-bundle.pem"

[JBlaschke]

I wonder if it's a Cray EX thing...

[KristofferC]

The release-1.10 branch should have the fix of this included. To test this you cannot really build from source (that would hide the problem) but you can download the corresponding artifact from https://buildkite.com/julialang/julia-release-1-dot-10/builds/227#_ . For example:

for the x86-64 linux build.

[kazuakiyama]

Just wanted to share that we got the same issue in a local institutional computing cluster with Julia 1.10.1 because of the inconsistent path to the ssl's .crt file, and the above ad-hoc fix with the environmental variable JULIA_SSL_CA_ROOTS_PATH worked.

[jaakkor2]

Closing as Julia 1.10.2 has been released.