Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

get peer certificates and signatures #8005

Merged
merged 5 commits into from
Jun 17, 2021

Conversation

will
Copy link
Contributor

@will will commented Jul 28, 2019

There are two patches here, that go along with doing SCRAM channel binding, with this draft pr in will/crystal-pg#177 using them as monkey patches.

The OpenSSL::SSL::Socket#peer_certificate I think for sure makes sense in the standard library, but the OpenSSL::X509::Certificate#signature, I'm not as sure. The upgrade of MD5 and SHA1 while, required in https://tools.ietf.org/html/rfc5929#section-4.1 is maybe not general purpose. The upgrade could be made optional. Then again maybe nothing else needs these signatures, so I don't know. Also I couldn't figure out how to get a test to go all the way through the method, since I don't see a way to make a OpenSSL::X509::Certificate with a signature.

@RX14
Copy link
Contributor

RX14 commented Jul 29, 2019

Could we add Certificate#signature_algorithm and Certificate#digest(algorithm) instead and move the SCRAM specific logic outside the stdlib?

@will
Copy link
Contributor Author

will commented Jul 29, 2019 via email

@will
Copy link
Contributor Author

will commented Jul 30, 2019

Updated with it split into two methods

Copy link
Contributor

@Sija Sija left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of mostly small stuff.

src/openssl/x509/certificate.cr Outdated Show resolved Hide resolved
src/openssl/x509/certificate.cr Outdated Show resolved Hide resolved
src/openssl/ssl/socket.cr Outdated Show resolved Hide resolved
src/openssl/ssl/socket.cr Outdated Show resolved Hide resolved
src/openssl/x509/certificate.cr Outdated Show resolved Hide resolved
src/openssl/x509/certificate.cr Outdated Show resolved Hide resolved
src/openssl/x509/certificate.cr Outdated Show resolved Hide resolved
Copy link
Member

@straight-shoota straight-shoota left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to have a positive spec on `signature_algorithm'.
Maybe this could help: https://github.com/sfackler/rust-openssl/blob/487963d17a71dd7f4b239d7284c623f1c6a0b64b/openssl/src/x509/tests.rs#L303

spec/std/openssl/ssl/socket_spec.cr Show resolved Hide resolved
Copy link
Member

@straight-shoota straight-shoota left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is fine to merge.

Maybe add a TODO in the #signature_algorithm spec.

@RX14
Copy link
Contributor

RX14 commented Aug 1, 2019

@will if the usage of this PR has been tested in crystal-pg, i'm fine to merge for 0.30.0. Others might want to wait for .1, but since a shard depends on this i'm inclined to merge sooner rather than later.

@bcardiff
Copy link
Member

bcardiff commented Aug 1, 2019

Darwin specs are failing due to missing symbols:

  • _OBJ_find_sigid_algs
  • _X509_get_signature_nid

Since this PR adding and not changing is easy to patch the library until work is done to make the CI in happy.

@will
Copy link
Contributor Author

will commented Aug 2, 2019

@RX14 I have it on a branch for crystal-pg, but it's not merged yet. I wanted to wait for the changes to make it in before committing to it. The branch is over at https://github.com/will/crystal-pg/commits/scram_cbind

@bcardiff the darwin failures are strange to me. I develop on mac, and both openssl 1.0.2 and 1.1.0 work fine for me:

~/c/crystal-pg (scram_cbind)> set -gx PKG_CONFIG_PATH /usr/local/opt/openssl/lib/pkgconfig
~/c/crystal-pg (scram_cbind)> pkg-config --atleast-version=1.0.2 libcrypto || printf %s false
~/c/crystal-pg (scram_cbind)> pkg-config --atleast-version=1.1.0 libcrypto || printf %s false
false⏎                                                                                                                                                                                   ~/c/crystal-pg (scram_cbind)> crystal spec
..............................................................................................................................................

Finished in 546.24 milliseconds
142 examples, 0 failures, 0 errors, 0 pending
~/c/crystal-pg (scram_cbind)> set -gx PKG_CONFIG_PATH "/usr/local/opt/[email protected]/lib/pkgconfig"
~/c/crystal-pg (scram_cbind)> pkg-config --atleast-version=1.0.2 libcrypto || printf %s false
~/c/crystal-pg (scram_cbind)> pkg-config --atleast-version=1.1.0 libcrypto || printf %s false
~/c/crystal-pg (scram_cbind)> crystal spec
..............................................................................................................................................

Finished in 556.07 milliseconds
142 examples, 0 failures, 0 errors, 0 pending
~/c/crystal-pg (scram_cbind)> uname -a
Darwin Wills-MBP.guest.corp.microsoft.com 18.7.0 Darwin Kernel Version 18.7.0: Thu Jun 20 18:42:21 PDT 2019; root:xnu-4903.270.47~4/RELEASE_X86_64 x86_64

@will
Copy link
Contributor Author

will commented Aug 2, 2019

Is it possible that the mac ci build isn't actually setting up openssl 1.0.2 by exporting pkg_config_path? When I don’t do that on mac everything fails to link at all (even before these changes), so I would have expected more things to fail sooner than this patch, but I don’t see "pkg_config" in the circle yaml.

@RX14
Copy link
Contributor

RX14 commented Aug 2, 2019

X509_get_signature_nid was introduced in openssl 1.1: openssl/openssl@dfcf48f

The linking is likely succeeding on your laptop because both openssl libraries are making their way into the library search path.

@will
Copy link
Contributor Author

will commented Aug 2, 2019

Hm, the implementation requires getting info out of the X509 struct. There isn't an easy way to do that in crystal except to redefine the entire struct in crystal, right? Like you cant just get one field out.

It looks like in 1.0.2, the field is only the second one in https://github.com/openssl/openssl/blob/OpenSSL_1_0_2-stable/crypto/x509/x509.h#L270, and it's in the same spot today https://github.com/openssl/openssl/blob/faa9dcd4d468441422254ab2d887bb267e0245b6/crypto/include/internal/x509_int.h#L161 but not all the other fields are the same.

1.0.2 is from January 22, 2015, what are the chances we can drop support for it?

@will
Copy link
Contributor Author

will commented Sep 25, 2019

Sorry I haven't followed up on this in a while. I'm unsure of what changes to make to move this along, or what I should be doing here.

@RX14
Copy link
Contributor

RX14 commented Sep 29, 2019

There's no chance we can drop old openssl support, you'll probably have to get it from the struct directly, depending on openssl version.

@will
Copy link
Contributor Author

will commented Jan 2, 2021

Rebased this. Curious what the current CI situation will be since the last time.

@straight-shoota
Copy link
Member

CI looks mostly good, but we'll still need support for OpenSSL 1.0.x. Unfortunately we don't have automatic tests set up against older OpenSSL versions.

AArch64 times out which doesn't happen in master, so it's likely caused by this change.

@will
Copy link
Contributor Author

will commented Jan 2, 2021

CI looks mostly good, but we'll still need support for OpenSSL 1.0.x.

Is this still true? The OpenSSL project itself dropped support for 1.0.2 on 2019-12-31. Also 1.1.0 (though that version works with this patch) was dropped earlier on 2019-09-11. The only supported current OpenSSL version is 1.1.1 which will last until 2023-09-11.

If it is still true, is there a timeline for how far beyond the library being unsupported by its maintainers that Crystal will support it? I'm okay doing the work here, but it'd be unfortunate if Crystal has to support dead openssl versions forever.

Also would it be sufficient to only have these bindings and functions available for >= 1.1.0 versions? Or does it need to work for 1.0.2 also? I stopped work on this because I couldn’t figure out how to get a single field out of a struct with crystal's c biding support. The only thing I could think of was to manually read some fixed number of bytes offset, which felt gross, but maybe that's the only way.

AArch64 times out which doesn't happen in master, so it's likely caused by this change.

Do you have any tips on debugging this?

@straight-shoota
Copy link
Member

Even if the openssl project has dropped support, 1.0.x is still the supported version in older distributions like Ubuntu xenial which has LTS til April.
However, using such outdated versions really is far from recomended. So maybe it's fine to drop full support for older versions, i.e. merge this as is. Everything else still works on openssl < 1.1 but the new additions only work with more recent releases.
I would be fine with that, but let's hear other opinions on this.

Regarding Aarch64, I guess the first step would be to find out where exactly the spec execution breaks. I don't think you can SSH into a Github actions runner. Best to use native hardware for this. I have an Raspberry Pi laying around somewhere, maybe I can take a stab at this.

@will
Copy link
Contributor Author

will commented Jan 3, 2021

I have an Raspberry Pi laying around somewhere, maybe I can take a stab at this.

I'm taking a stab at it on an ec2 arm instance. I'm not really sure if I'm using docker right or anything, but it seems like I got some of the tests to run and build at least. I can add the ssh key you have in your github if you want to use this instance instead of a raspberry pi. edit: it looks like my change fixed this, so I've destroyed the instance

I'm not sure, but the issue may be spec/std/openssl/ssl/socket_spec.cr

ubuntu@ip-172-31-24-182:~/crystal$ sudo docker run -it  -v ${PWD}:/tmp jhass/crystal:0.35.0-alpine-build /bin/sh
/ $ cd /tmp
/tmp $ bin/crystal spec spec/std/openssl/
Using compiled compiler at .build/crystal
..........................*.......................................................

Pending:
  OpenSSL::SSL::Context ciphers uses intermediate default ciphers

Finished in 2.66 seconds
82 examples, 0 failures, 0 errors, 1 pending
/tmp $ git checkout peer_cert
Switched to branch 'peer_cert'
/tmp $ bin/crystal spec spec/std/openssl/
Using compiled compiler at .build/crystal
..........................*.....................Unhandled exception in spawn: Error connecting to ':::38381': Address not available (Socket::ConnectError)
  from src/socket/addrinfo.cr:73:15 in 'initialize'
  from src/socket/tcp_socket.cr:27:3 in 'initialize'
  from src/socket/tcp_socket.cr:27:3 in 'new'
  from spec/std/openssl/ssl/socket_spec.cr:55:19 in '->'
  from src/primitives.cr:255:3 in 'run'
  from src/fiber.cr:92:34 in '->'
  from ???
^C

Pending:
  OpenSSL::SSL::Context ciphers uses intermediate default ciphers

Aborted!
Finished in 1:22 minutes
48 examples, 0 failures, 0 errors, 1 pending

Now have to figure out why my change would have caused that, if that is the actual issue and not a byproduct or something.

@will
Copy link
Contributor Author

will commented Jan 3, 2021

Maybe found it, trying this, but squashed it into the commit that introduced the test

diff --git c/spec/std/openssl/ssl/socket_spec.cr i/spec/std/openssl/ssl/socket_spec.cr
index fdf16d686..f4265cd0a 100644
--- c/spec/std/openssl/ssl/socket_spec.cr
+++ i/spec/std/openssl/ssl/socket_spec.cr
@@ -47,7 +47,7 @@ private alias Server = OpenSSL::SSL::Socket::Server
 private alias Client = OpenSSL::SSL::Socket::Client

 private def socket_test(server_tests, client_tests)
-  tcp_server = TCPServer.new(0)
+  tcp_server = TCPServer.new("127.0.0.1", 0)
   server_context, client_context = ssl_context_pair

   OpenSSL::SSL::Server.open(tcp_server, server_context) do |server|

It looks like the localhost address was added, in these tests, and my small test refactor didn't pick it up in the rebase. But not sure why that'd be a problem only on ARM

@will
Copy link
Contributor Author

will commented Jan 3, 2021

I tried building on a clean xenial instance, to avoid the problem I was having earlier with multiple openssl libs on the same machine. However I did not get the same 'cant find symbol' errors that were happening on the old Darwin builds from before.

Instead it was these errors, which happened both with and without my patch

ubuntu@ip-172-31-29-169:~/crystal$ lsb_release -c
Codename:	xenial
ubuntu@ip-172-31-29-169:~/crystal$ openssl version
OpenSSL 1.0.2g  1 Mar 2016
ubuntu@ip-172-31-29-169:~/crystal$ bin/crystal --version
Using compiled compiler at .build/crystal
Crystal 1.0.0-dev [cafe978] (2021-01-03)

LLVM: 8.0.0
Default target: x86_64-pc-linux-gnu

ubuntu@ip-172-31-29-169:~/crystal$ bin/crystal spec spec/std/openssl
Using compiled compiler at .build/crystal
................................*..E..F...........................................

Pending:
  OpenSSL::SSL::Context ciphers uses intermediate default ciphers

Failures:

  1) OpenSSL::SSL::Context changes security level
     Failure/Error: context.security_level.should eq(level + 1)

       Expected: 1
            got: 0

     # spec/std/openssl/ssl/context_spec.cr:140

  2) OpenSSL::SSL::Context ciphers sets modern ciphers

       SSL_CTX_set_cipher_list: error:1410D0B9:SSL routines:SSL_CTX_set_cipher_list:no cipher match (OpenSSL::Error)
         from src/openssl/ssl/context.cr:263:5 in 'ciphers='
         from src/openssl/ssl/context.cr:284:5 in 'set_modern_ciphers'
         from spec/std/openssl/ssl/context_spec.cr:124:7 in '->'
         from src/primitives.cr:255:3 in 'internal_run'
         from src/spec/example.cr:33:16 in 'run'
         from src/spec/context.cr:18:23 in 'internal_run'
         from src/spec/context.cr:330:7 in 'run'
         from src/spec/context.cr:18:23 in 'internal_run'
         from src/spec/context.cr:330:7 in 'run'
         from src/spec/context.cr:18:23 in 'internal_run'
         from src/spec/context.cr:147:7 in 'run'
         from src/spec/dsl.cr:274:7 in '->'
         from src/primitives.cr:255:3 in 'run'
         from src/crystal/main.cr:45:14 in 'main'
         from src/crystal/main.cr:114:3 in 'main'
         from __libc_start_main
         from _start
         from ???


Finished in 2.28 seconds
82 examples, 1 failures, 1 errors, 1 pending

Failed examples:

crystal spec spec/std/openssl/ssl/context_spec.cr:136 # OpenSSL::SSL::Context changes security level
crystal spec spec/std/openssl/ssl/context_spec.cr:123 # OpenSSL::SSL::Context ciphers sets modern ciphers

maybe support for 1.0.2 is already broken in crystal? I am surprised that the linking error didn't happen though

@straight-shoota
Copy link
Member

straight-shoota commented Jan 3, 2021

Awesome! I have no explanation as to why this error showed up with the changes in this PR, and only on aarch64. The spec code for establishing a connection stayed the same. (missed your explanation and looked only at the fixed patch)

The error message Error connecting to ':::38381': Address not available actually already showed up in the original CI failure message. It's above the spec progress dots. Maybe stderr is just written before stdout for some reason...

Side note: Please do not squash and force push fixes for better traceability of the intermediary changes.

@straight-shoota straight-shoota removed the pr:needs-work A PR requires modifications by the author. label Jan 3, 2021
@straight-shoota
Copy link
Member

straight-shoota commented Jan 3, 2021

It looks like #9831 already introduced optional features that don't work with openssl < 1.1, so I'd say this PR can be accepted as is.

We should probably consider documenting the support levels somewhere, though. Would probably be good to add the required openssl version at least to the method docs.

@bcardiff
Copy link
Member

In #9831 the policy was to allow functionality that will work only on certain version as long as

  • the modules still compiles in older version
  • a warn is logged

I tried to use 1.0.2 via nix and things build fine there. (X509_get_signature_nid was indeed present in 1.0.2 by openssl/openssl@bd9fc1d)

In debian:9 de openssl is 1.0.1 and there I got

# bin/crystal spec spec/std/openssl/**/*_spec.cr 
O-penS-S-L-5858X-5095858C-ertificate.o: In function `signature_algorithm':
/src/src/openssl/x509/certificate.cr:65: undefined reference to `X509_get_signature_nid'
collect2: error: ld returned 1 exit status

I guess something similar will happen with libressl, I am not sure. It's fine if we support the propsed features in this PR only with openssl (again, as long as things compile if it's not used)

In order to move forward I think the following things are needed.

  1. wrap fun declaration with {% if compare_versions(OPENSSL_VERSION, "1.0.2") >= 0 %}
  2. use {% if compare_versions(OPENSSL_VERSION, "1.0.2") >= 0 %} to emit Log.warn { "X509_get_signature_nid not supported" } when that method is not supported and avoid trying to link an undefined symbol. The crystal method could either raise or return a stub value depending on what make sense.

@bcardiff
Copy link
Member

Using libressl (replacing openssl with libressl and $ nix-shell --pure)

[nix-shell:~/Projects/crystal/master]$ bin/crystal eval 'require "openssl"; pp! LibSSL::LIBRESSL_VERSION'
LibSSL::LIBRESSL_VERSION # => "3.0.2"

[nix-shell:~/Projects/crystal/master]$ bin/crystal eval 'require "openssl"; pp! LibSSL::OPENSSL_VERSION'
LibSSL::OPENSSL_VERSION # => "0.0.0"

[nix-shell:~/Projects/crystal/master]$ bin/crystal spec spec/std/openssl/**/*_spec.cr
........*..E..F...........F................................

Pending:
  OpenSSL::SSL::Context ciphers uses intermediate default ciphers

Failures:

  1) OpenSSL::SSL::Context changes security level
     Failure/Error: context.security_level.should eq(level + 1)

       Expected: 1
            got: 0

     # spec/std/openssl/ssl/context_spec.cr:140

  2) OpenSSL::SSL::Context .from_hash errors
     Failure/Error: expect_raises(OpenSSL::Error, "SSL_CTX_use_PrivateKey_file: error:02001002:system library:fopen:No such file or directory") do

       Expected OpenSSL::Error with "SSL_CTX_use_PrivateKey_file: error:02001002:system library:fopen:No such file or directory", got #<OpenSSL::Error: SSL_CTX_use_PrivateKey_file: error:02FFF002:system library:func(4095):No such file or directory> with backtrace:
         # src/openssl/ssl/context.cr:254:5 in 'private_key='
         # src/openssl/ssl/context.cr:474:7 in 'from_hash'
         # src/openssl/ssl/context.cr:78:7 in 'from_hash'
         # spec/std/openssl/ssl/context_spec.cr:243:9 in '->'
         # src/primitives.cr:255:3 in 'internal_run'
         # src/spec/example.cr:33:16 in 'run'
         # src/spec/context.cr:18:23 in 'internal_run'
         # src/spec/context.cr:330:7 in 'run'
         # src/spec/context.cr:18:23 in 'internal_run'
         # src/spec/context.cr:330:7 in 'run'
         # src/spec/context.cr:18:23 in 'internal_run'
         # src/spec/context.cr:147:7 in 'run'
         # src/spec/dsl.cr:274:7 in '->'
         # src/primitives.cr:255:3 in 'run'
         # src/crystal/main.cr:45:14 in 'main'
         # src/crystal/main.cr:119:3 in 'main'

     # spec/std/openssl/ssl/context_spec.cr:242

  3) OpenSSL::SSL::Context ciphers sets modern ciphers

       SSL_CTX_set_cipher_list: error:14FFF0B9:SSL routines:(UNKNOWN)SSL_internal:no cipher match (OpenSSL::Error)
         from src/openssl/ssl/context.cr:263:5 in 'ciphers='
         from src/openssl/ssl/context.cr:284:5 in 'set_modern_ciphers'
         from spec/std/openssl/ssl/context_spec.cr:124:7 in '->'
         from src/primitives.cr:255:3 in 'internal_run'
         from src/spec/example.cr:33:16 in 'run'
         from src/spec/context.cr:18:23 in 'internal_run'
         from src/spec/context.cr:330:7 in 'run'
         from src/spec/context.cr:18:23 in 'internal_run'
         from src/spec/context.cr:330:7 in 'run'
         from src/spec/context.cr:18:23 in 'internal_run'
         from src/spec/context.cr:147:7 in 'run'
         from src/spec/dsl.cr:274:7 in '->'
         from src/primitives.cr:255:3 in 'run'
         from src/crystal/main.cr:45:14 in 'main'
         from src/crystal/main.cr:119:3 in 'main'
       

Finished in 1.48 seconds
59 examples, 2 failures, 1 errors, 1 pending

Failed examples:

crystal spec spec/std/openssl/ssl/context_spec.cr:136 # OpenSSL::SSL::Context changes security level
crystal spec spec/std/openssl/ssl/context_spec.cr:234 # OpenSSL::SSL::Context .from_hash errors
crystal spec spec/std/openssl/ssl/context_spec.cr:123 # OpenSSL::SSL::Context ciphers sets modern ciphers

@bcardiff
Copy link
Member

To clarify 2. the partially supported methods should either a) warn + return a stub value or b) raise (no need to warn in this case).

@bcardiff bcardiff removed their assignment Jan 27, 2021
@will
Copy link
Contributor Author

will commented Mar 20, 2021

Hi @bcardiff, I just did your suggestions. Sorry this took so long. I was trying to get nix set up having never used it before, then ran into a lot of issues with that, then other irl stuff came up and it fell off my radar. Please let me know if this looks acceptable.

@will
Copy link
Contributor Author

will commented Jun 15, 2021

@bcardiff can you please take another look at this? Happy to do any other changes you think if there is anything

Copy link
Member

@bcardiff bcardiff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't follow why the Client#peer_certificate method is specialized to raise on nil while the base implementation is nilable.

Besides that, on my end the PR can be merged as is.

(Sorry I've been a bit disconnected)

@bcardiff bcardiff added this to the 1.1.0 milestone Jun 15, 2021
@will
Copy link
Contributor Author

will commented Jun 15, 2021

I don't follow why the Client#peer_certificate method is specialized to raise on nil while the base implementation is nilable.

If you're a client, there will always be a peer certificate (the one the server is using), but if you are a server there might not be a peer cert (the client may or may not have one). So I added not_nil! for the client class.

(Sorry I've been a bit disconnected)

No problem, thanks for taking a look. Let me know if you want me to rebase and/or squash down to a single commit or anything.

@straight-shoota
Copy link
Member

We'll squash on merge.

Merging master would be good before merging. I can do that myself.

@straight-shoota straight-shoota merged commit a9bbc0b into crystal-lang:master Jun 17, 2021
@straight-shoota
Copy link
Member

Thank you @will

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants