Crash: Could not load platform certs

[Firionus]

Trying to perform any downloads with my juliaup installation doesn't work and tells me to make a bug report.

Steps to Reproduce

Trying to update juliaup gives an error:

$ juliaup self update
Checking for self-updates
Well, this is embarrassing.

Juliaup had a problem and crashed. To help us diagnose the problem you can send us a crash report.

We have generated a report file at "/tmp/report-68b24b49-fef5-40bc-9d07-d57142c17d62.toml". Submit an issue or email with the subject of "Juliaup Crash Report" and include the report as an attachment.

- Homepage: https://github.com/JuliaLang/juliaup

We take privacy seriously, and do not perform any automated error collection. In order to improve the software, we rely on people to submit reports.

Thank you kindly!

The same happens with every other operation downloading something. The associated report shows it's a cert problem:

name = 'Juliaup'
operating_system = 'unix:Ubuntu'
crate_version = '1.6.10'
explanation = '''
Panic occurred in file '/cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/ureq-2.5.0/src/rtls.rs' at line 66
'''
cause = 'Could not load platform certs: Custom { kind: InvalidData, error: "Could not load PEM file \"/usr/lib/ssl/certs/ca-certificates.crt\"" }'
method = 'Panic'
backtrace = '''

   0: 0x562b4a867c2b - once_cell::imp::OnceCell<T>::initialize::{{closure}}::h5bfcc1d88491955f
   1: 0x562b4a76bc64 - once_cell::imp::initialize_or_wait::h1ec0dac3b5a4b4b2
   2: 0x562b4a64ee2b - once_cell::imp::OnceCell<T>::initialize::hb0b596b192b33afe
   3: 0x562b4a866080 - ureq::agent::AgentBuilder::new::hd3bc112ff06394f3
   4: 0x562b4a761454 - juliaup::command_selfupdate::run_command_selfupdate::hd36b7b3ec05e50b9
   5: 0x562b4a65e6ab - juliaup::main::h23171212412d87b3
   6: 0x562b4a64f743 - std::sys_common::backtrace::__rust_begin_short_backtrace::h601d65d54d71e8ea
   7: 0x562b4a678852 - main
   8: 0x7fd016347d90 - __libc_start_call_main
                at ./csu/../sysdeps/nptl/libc_start_call_main.h:58
   9: 0x7fd016347e40 - __libc_start_main_impl
                at ./csu/../csu/libc-start.c:392
  10: 0x562b4a64f639 - _start
  11:        0x0 - <unresolved>'''

Versions

julia> versioninfo()
Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)

$ lsb_release -d
Description:	Ubuntu 22.04 LTS

Ideas

I recently upgraded to Ubuntu 22.04 LTS which might or might not be relevant.

I removed the installation with juliaup self uninstal. Reinstalling with curl -fsSL https://install.julialang.org | sh shows the same error again after accepting the default config. The report file is this:

name = 'Juliainstaller'
operating_system = 'unix:Ubuntu'
crate_version = '1.6.10'
explanation = '''
Panic occurred in file '/cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/ureq-2.5.0/src/rtls.rs' at line 66
'''
cause = 'Could not load platform certs: Custom { kind: InvalidData, error: "Could not load PEM file \"/usr/lib/ssl/certs/ca-certificates.crt\"" }'
method = 'Panic'
backtrace = '''

   0: 0x55b4a2a5441b - once_cell::imp::OnceCell<T>::initialize::{{closure}}::h5bfcc1d88491955f
   1: 0x55b4a2959274 - once_cell::imp::initialize_or_wait::h1ec0dac3b5a4b4b2
   2: 0x55b4a285030b - once_cell::imp::OnceCell<T>::initialize::hb0b596b192b33afe
   3: 0x55b4a2931dee - juliaup::operations::download_extract_sans_parent::h50719b73d0245e24
   4: 0x55b4a28666b6 - juliainstaller::main::h2e3f8c843dc9ac5a
   5: 0x55b4a28512e3 - std::sys_common::backtrace::__rust_begin_short_backtrace::h2fce2242684986fd
   6: 0x55b4a286af02 - main
   7: 0x7f53662aed90 - __libc_start_call_main
                at ./csu/../sysdeps/nptl/libc_start_call_main.h:58
   8: 0x7f53662aee40 - __libc_start_main_impl
                at ./csu/../csu/libc-start.c:392
   9: 0x55b4a2850af9 - _start
  10:        0x0 - <unresolved>'''

[Firionus]

I just managed to reinstall juliaup by using the 1.6.9 installer:

curl -o juliaup_installer https://julialang-s3.julialang.org/juliaup/bin/juliainstaller-1.6.9-x86_64-unknown-linux-gnu
chmod +x juliaup_installer
./juliaup_installer

Afterwards, version 1.6.10 is succesfully installed:

$ juliaup -V
Juliaup 1.6.10

But trying a self update is again frought with trouble:

$ juliaup self update
Checking for self-updates
Well, this is embarrassing.

Juliaup had a problem and crashed. To help us diagnose the problem you can send us a crash report.

We have generated a report file at "/tmp/report-169ea3ca-b323-45f7-94fc-45c56f032c6a.toml". Submit an issue or email with the subject of "Juliaup Crash Report" and include the report as an attachment.

- Homepage: https://github.com/JuliaLang/juliaup

We take privacy seriously, and do not perform any automated error collection. In order to improve the software, we rely on people to submit reports.

Thank you kindly!

Report file:

name = 'Juliaup'
operating_system = 'unix:Ubuntu'
crate_version = '1.6.10'
explanation = '''
Panic occurred in file '/cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/ureq-2.5.0/src/rtls.rs' at line 66
'''
cause = 'Could not load platform certs: Custom { kind: InvalidData, error: "Could not load PEM file \"/usr/lib/ssl/certs/ca-certificates.crt\"" }'
method = 'Panic'
backtrace = '''

   0: 0x558925e67c2b - once_cell::imp::OnceCell<T>::initialize::{{closure}}::h5bfcc1d88491955f
   1: 0x558925d6bc64 - once_cell::imp::initialize_or_wait::h1ec0dac3b5a4b4b2
   2: 0x558925c4ee2b - once_cell::imp::OnceCell<T>::initialize::hb0b596b192b33afe
   3: 0x558925e66080 - ureq::agent::AgentBuilder::new::hd3bc112ff06394f3
   4: 0x558925d61454 - juliaup::command_selfupdate::run_command_selfupdate::hd36b7b3ec05e50b9
   5: 0x558925c5e6ab - juliaup::main::h23171212412d87b3
   6: 0x558925c4f743 - std::sys_common::backtrace::__rust_begin_short_backtrace::h601d65d54d71e8ea
   7: 0x558925c78852 - main
   8: 0x7ffb5bac8d90 - __libc_start_call_main
                at ./csu/../sysdeps/nptl/libc_start_call_main.h:58
   9: 0x7ffb5bac8e40 - __libc_start_main_impl
                at ./csu/../csu/libc-start.c:392
  10: 0x558925c4f639 - _start
  11:        0x0 - <unresolved>'''

So, the problem is likely related to the changes between the two versions: v1.6.9...v1.6.10

I did try running update-ca-certificates, but to no effect:

$ sudo update-ca-certificates --fresh
Clearing symlinks in /etc/ssl/certs...
done.
Updating certificates in /etc/ssl/certs...
rehash: warning: skipping telesec.rootcert.pem,it does not contain exactly one certificate or CRL
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
128 added, 0 removed; done.
[rest of output omitted]

Afterwards the error with juliaup is the same. Though ca-certificates.crt was skipped in the update process and I'm not sure what that implies.

[davidanthoff]

Yeah, I changed how we handle TLS for 1.6.10, instead of using a hard coded list of root certificates that we ship ourselves (the webpki list), we are now trying to use the system root cert list. Apparently, that doesn't always work reliably... My current plan is to change this on Linux so that it first attempts to load the system certificates, and if that fails (like it does in your case) to fall back on the webpki list of certificates that we used previously.

Note that on Windows on Mac we stopped validating certs ourselves entirely and just had that off to the OS.

CC @StefanKarpinski does this all make sense to you?

[davidanthoff]

Ok, new plan: I will go back to the hardcoded webpki list that we used for the last year or so on Linux. The crate that implements the functionality of using the native cert list (https://github.com/rustls/rustls-native-certs) has this language:

The quality of the ca-certificates package on debian-based Linux distributions is poor. At the time of writing, this ships many certificates not included in the Mozilla set, either because they failed an audit and were withdrawn or were removed for mississuance.

That to me sounds sufficiently scary that I think we probably don't want to use that as a default no matter what.

We can still add a configuration option at some later point to use the native cert list on Linux, but as an opt-in.

[Firionus]

Thanks for the fast response 🚀, it is appreciated. If I can test something, point me at it.

As for the underlying reason: While I can't comment on whether it is a good idea or not to use Debian's ca-certificates, it can't be the issue in this case, since curl eats my ca-certificates.crt just fine, but rustls-native-certs does not. Example:

$ curl --verbose https://julialang-s3.julialang.org/juliaup/bin/juliainstaller-1.6.10-x86_64-unknown-linux-gnu
*   Trying 2a04:4e42:3::561:443...
* Connected to julialang-s3.julialang.org (2a04:4e42:3::561) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.julialang.org
*  start date: Jul  9 16:42:13 2022 GMT
*  expire date: Aug 10 16:42:12 2023 GMT
*  subjectAltName: host "julialang-s3.julialang.org" matched cert's "*.julialang.org"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Atlas R3 DV TLS CA 2022 Q2
*  SSL certificate verify ok.
[rest of output omitted]

So curl works fine with local certs, but with the exact same SSL file juliaup v1.6.10 crashes:

$ SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt juliaup self update
Checking for self-updates
Well, this is embarrassing.
[rest of output omitted]

report:

name = 'Juliaup'
operating_system = 'unix:Ubuntu'
crate_version = '1.6.10'
explanation = '''
Panic occurred in file '/cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/ureq-2.5.0/src/rtls.rs' at line 66
'''
cause = 'Could not load platform certs: Custom { kind: InvalidData, error: "Could not load PEM file \"/etc/ssl/certs/ca-certificates.crt\"" }'
method = 'Panic'
backtrace = '''

   0: 0x5640a8067c2b - once_cell::imp::OnceCell<T>::initialize::{{closure}}::h5bfcc1d88491955f
   1: 0x5640a7f6bc64 - once_cell::imp::initialize_or_wait::h1ec0dac3b5a4b4b2
   2: 0x5640a7e4ee2b - once_cell::imp::OnceCell<T>::initialize::hb0b596b192b33afe
   3: 0x5640a8066080 - ureq::agent::AgentBuilder::new::hd3bc112ff06394f3
   4: 0x5640a7f61454 - juliaup::command_selfupdate::run_command_selfupdate::hd36b7b3ec05e50b9
   5: 0x5640a7e5e6ab - juliaup::main::h23171212412d87b3
   6: 0x5640a7e4f743 - std::sys_common::backtrace::__rust_begin_short_backtrace::h601d65d54d71e8ea
   7: 0x5640a7e78852 - main
   8: 0x7f313e178d90 - __libc_start_call_main
                at ./csu/../sysdeps/nptl/libc_start_call_main.h:58
   9: 0x7f313e178e40 - __libc_start_main_impl
                at ./csu/../csu/libc-start.c:392
  10: 0x5640a7e4f639 - _start
  11:        0x0 - <unresolved>'''

So it seems like there is an issue with https://github.com/rustls/rustls-native-certs here. I'll see if I can reproduce the issue with a little Rust example and maybe I can open an issue there. Let's see.

[davidanthoff]

So for now I've just reverted back to always using the Mozilla cert list on Linux. Primarily we need to have a version that works for Juliacon ;)

If we can get the native cert stuff working down the road on Linux as well that would be great!

[Firionus]

Just installed from the dev channel with curl -fsSL https://install.julialang.org/dev | sh and it works 👍

$ juliaup self update
Checking for self-updates
Juliaup unchanged on channel 'dev' - 1.6.12

I'll change back to the release channel once 1.6.12 has that status.

[davidanthoff]

Great!

[StefanKarpinski]

I think this should probably be reverted once the pemfile dependency is upgraded to include rustls/pemfile#7. Yes, the Mozilla certs a known good, so why not use them? Because many users are behind MITM proxying firewalls and the connection will fail using only public certs. On such systems, a sysadmin will typically have added the CA root cert for the firewall to the system so that secure connections can be made. By using Mozilla certs, that will stop working.