aiohttp ignoring SSL_CERT_DIR and or SSL_CERT_FILE environment vars. Results in [SSL: CERTIFICATE_VERIFY_FAILED] #3180

creslinux · 2018-08-08T09:47:30Z

Long story short

The CA file is working with cURL, Python Requests, but not aiohttp, when using SSL_CERT_DIR and or SSL_CERT_FILE environment variables.

Our environment uses its own CA root used to decode/encode HTTPS API requests/responses to provide a short lived cache to prevent excessing external requests.

The environment has the following set:

$ (set -o posix; set) | egrep 'SSL|_CA'
CURL_CA_BUNDLE=/home/creslin/poo/freqcache/cert/ca.pem
REQUESTS_CA_BUNDLE=/home/creslin/poo/freqcache/cert/ca.pem
SSL_CERT_DIR=/home/creslin/poo/freqcache/cert/
SSL_CERT_FILE=/home/creslin/poo/freqcache/cert/ca.pem

The ca.pem can be successfully used by cURL - with both a positive and negative test shown:

curl --cacert /home/creslin/poo/freqcache/cert/ca.pem https://api.binance.com/api/v1/time
{"serverTime":1533719563552}

curl --cacert /home/creslin/NODIRHERE/ca.pem https://api.binance.com/api/v1/time
curl: (77) error setting certificate verify locations:
  CAfile: /home/creslin/NODIRHERE/ca.pem
  CApath: /etc/ssl/certs

A simple python requests script req.py also works as expected, positive and negative tests

cat req.py 
import requests
req=requests.get('https://api.binance.com/api/v1/time', verify=True)
print(req.content)

CURL_CA_BUNDLE=/home/creslin/poo/freqcache/cert/ca.pem
REQUESTS_CA_BUNDLE=/home/creslin/poo/freqcache/cert/ca.pem
SSL_CERT_DIR=/home/creslin/poo/freqcache/cert/
SSL_CERT_FILE=/home/creslin/poo/freqcache/cert/ca.pem

python3 req.py 
b'{"serverTime":1533720141278}'

CURL_CA_BUNDLE=/
REQUESTS_CA_BUNDLE=/
SSL_CERT_DIR=/
SSL_CERT_FILE=/

python3 req.py 
Traceback (most recent call last):
  File "/home/creslin/freqt .......

..... ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)

Using aysnc/aiohttp [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833) is returned always.
The environment settings pointing to the ca.pem shown to work for both cURL and requests are seemingly ignored

CURL_CA_BUNDLE=/home/creslin/poo/freqcache/cert/ca.pem
REQUESTS_CA_BUNDLE=/home/creslin/poo/freqcache/cert/ca.pem
SSL_CERT_DIR=/home/creslin/poo/freqcache/cert/
SSL_CERT_FILE=/home/creslin/poo/freqcache/cert/ca.pem

I have the test script a.py as

cat a.py 
import aiohttp
import ssl
import asyncio
import requests

print("\n requests.certs.where", requests.certs.where())
print("\n ssl version", ssl.OPENSSL_VERSION)
print("\n ssl Paths", ssl.get_default_verify_paths() ,"\n")
f = open('/home/creslin/poo/freqcache/cert/ca.crt', 'r') # check perms are ok
f.close()

async def main():
    session = aiohttp.ClientSession()
    async with session.get('https://api.binance.com/api/v1/time') as response:
        print(await response.text())
    await session.close()

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

Which will always produce the failure - output in full:

 requests.certs.where /home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/certifi/cacert.pem

 ssl version OpenSSL 1.1.0g  2 Nov 2017

 ssl Paths DefaultVerifyPaths(cafile=None, capath='/home/creslin/poo/freqcache/cert/', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/lib/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/lib/ssl/certs') 

Traceback (most recent call last):
  File "/home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/aiohttp/connector.py", line 822, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 804, in create_connection
    sock, protocol_factory, ssl, server_hostname)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 830, in _create_connection_transport
    yield from waiter
  File "/usr/lib/python3.6/asyncio/sslproto.py", line 505, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/usr/lib/python3.6/asyncio/sslproto.py", line 201, in feed_ssldata
    self._sslobj.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "a.py", line 20, in <module>
    loop.run_until_complete(main())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "a.py", line 14, in main
    async with session.get('https://api.binance.com/api/v1/time') as response:
  File "/home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/aiohttp/client.py", line 843, in __aenter__
    self._resp = await self._coro
  File "/home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/aiohttp/client.py", line 366, in _request
    timeout=timeout
  File "/home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/aiohttp/connector.py", line 445, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/aiohttp/connector.py", line 757, in _create_connection
    req, traces, timeout)
  File "/home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/aiohttp/connector.py", line 879, in _create_direct_connection
    raise last_exc
  File "/home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/aiohttp/connector.py", line 862, in _create_direct_connection
    req=req, client_error=client_error)
  File "/home/creslin/freqtrade/freqtrade_mp/freqtrade_technical_jul29/freqtrade/.env/lib/python3.6/site-packages/aiohttp/connector.py", line 827, in _wrap_create_connection
    raise ClientConnectorSSLError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host api.binance.com:443 ssl:None [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)]
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fa6bf9de898>

Expected behaviour

aiohttp does not reject server certificate.

Actual behaviour

SSL verification error

Steps to reproduce

use own CA root certificate to trust HTTPS server.

Your environment

Ubuntu 18.04
Python 3.6.5
ssl version OpenSSL 1.1.0g 2 Nov 2017
Name: aiohttp Version: 3.3.2
Name: requests Version: 2.19.1
Name: certifi Version: 2018.4.16

The text was updated successfully, but these errors were encountered:

webknjaz · 2018-08-08T10:15:31Z

There's this PR #2735 hanging since February. Do you mind picking it up? It'd allow you to have a custom ssl.SSLContext, where you could configure trust as you wish.

creslinux · 2018-08-08T10:27:16Z

Hi @webknjaz

Is 2735 related to aiohttp not using SSL_CERT_DIR and or SSL_CERT_FILE environment variables to set a trusted root ca locaton?

The PR does not read this way, i may *easily be mistaken

asvetlov · 2018-08-08T11:12:14Z

Looks like @webknjaz misread the mentioned PR.

aiohttp has no specific code for SSL certificates loading.
It is out of the scope of the library.

Please read the standard documentation to figure out how to create SSLContext with custom certs.
Using environment variables for this is an error-prone and leaking way, please learn how to create proper SSL context by Python API calls.

creslinux · 2018-08-08T11:54:10Z

I dont think its as simple as that - there is a catch 22 here.

requests and aiohttp are commonly used by other libraries - changing those are out of scope for many users and use-cases

To explain further
requests does not use the system key store, they can be pointed at CA file via ENV to work around this.
This then also updates where SSL is looking, printing out ssl.get_default_verify_paths() shows this.

aiohttp does use the system key store, via SSL.
But if an application has requests calls this has changed where SSL looks for ca certs.
and seemingly aiohttp does not follow where SSL is looking

We're left where an application can make no obvious use of requests and aiohttp and a companies own root_ca certificate.

asvetlov · 2018-08-08T12:31:46Z

aiohttp follows Python decisions which use system store.
certifi approach has own benefits and pitfalls.
I prefer to follow Python.
If Python core devs will decide to switch to a custom certs store or respect environment variables -- aiohttp will do it too. This discussion raised in the Python dev team several times and proposals were constantly declined for security reasons.

If you want requests-like configuration by default -- you can assemble a thin wrapper on top of aiohttp to do it. Personally I dont like this approach

creslinux · 2018-08-08T13:06:56Z

OK - thanks for the response.

webknjaz · 2018-08-09T08:09:31Z

@asvetlov AFAIR aiohttp does not have any hooks where the end-user could supply their own SSLContext objects and that PR adds one. Am I right?

creslinux · 2018-08-09T12:24:45Z

system certs are a mess. an unmanageable real mess if we're honest once scratch the surface of it.

OSX and Windows you have half a chance, even then its problematic as certs are lazy loaded via s-channel so may or may not be there.

Linux is beyond a lost cause - system certs can be in any 1 of 10 locations reported, finding one those isn't enough to assume its "THE" one.
Debian and thereafter ubuntu (so thats 90% of users in my linux world) certs are ridiculously out of date, unmanaged which moots entirely the point of having it - if you're trusting root CAs even the root ca authority are trying to revoke for years.....
Then anaconda may have trodden all over openssl anyway etc

Pip tries to use system certs, but falls back to mozillas certifi bundle because system certs are anything anywhere

None of that is criticism of aiohttp or python.
Being able to simply and absolutely point at a store, cognitive that aiohttp/asynio are now often a couple libraries down from the library or app users are building upon would be tremendously useful.

There is no portability in it without - code on one os is not going to behave the same on another on a lift and shift.

creslinux · 2018-08-09T12:30:59Z

Google couldn't make using system certs work on linux for Chrome, the average python scripter - myself in this bracket - has no hope.

webknjaz · 2018-08-09T12:41:38Z

@creslinux recently I've been trying to automate (with ansible) adding a coprorate certificate to my laptop and found out that neither Chrome nor Firefox can read system folder, only user's one: turns out that underlying nsslib only promises to read /etc, but in fact does not do that.

creslinux · 2018-08-09T12:53:33Z

it all seems a bit religious.
Whats wrong with options such as pip install aiohttp[certifi] or pip install requests[system] and let the users / deployment profile choose.

asvetlov · 2018-08-09T15:08:15Z

Help yourself.
Say again, for using certifi you don't need aiohttp source code modification, just make a wrapper and use it happily.

webknjaz · 2018-08-09T15:20:40Z

What about SSLContext? Only monkey-patching?

creslinux · 2018-08-09T15:26:35Z

Its a great answer when building on aiohttp directly. When youre using a library thats is itself using aiohttp is not that simple The only thing thats true here is I'm not the first or the last to hit this. Users will carry the pain again and again and again The longer its around the deeper and more abstract aiohttp will often get buried under under other libraries. Asking users to then fork layers within apps to maintain something that works for them - "them" being maybe 50% users as python is split on certs - isnt really an answer that will stand the test of time.

…

On 9 August 2018 at 15:20, Sviatoslav Sydorenko ***@***.***> wrote: What about SSLContext? Only monkey-patching? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3180 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AhCkw5slrWkTrrfGpwi0aSsVkRN_zihQks5uPFNPgaJpZM4VznQt> .

asvetlov · 2018-08-09T18:36:40Z

@webknjaz await client.get(url, ssl=...) supports SSLContext argument, no need for monkey patching

webknjaz · 2018-08-09T18:40:28Z

Okay, that's what I was asking about :)

asvetlov · 2018-08-09T18:49:35Z

@creslinux your arguments are applicable to any TSL connection, not only HTTP.
I have a feeling that the problem should be fixed on OS or Python level.

Would you try to convince python core devs to ship Python with custom root certs?
I recall @tiran had objections for this approach.
Honestly, I don't remember the whole pros/cons list.
The security is very sensitive matter, I don't want to make the decision fast.

creslinux · 2018-08-09T19:06:31Z

I'm not sure what you mean by python. Aiohttp is my only experience a popular large library for http/s that does not support the vars. Requests, pip, twisted, ssl, pycurl all use them. Outside of python, curl, wget, openssl, and the linux OS use them. Fairly well the defacto TLS in linux is openssl, so why when openssl is pointing to the right root CA to be used via ENV vars aiohttp would ignore that I don't know Redhat as example. https://access.redhat.com/articles/2039753

…

On Thu, Aug 9, 2018, 9:49 PM Andrew Svetlov ***@***.***> wrote: @creslinux <https://github.com/creslinux> your arguments are applicable to any TSL connection, not only HTTP. I have a feeling that the problem should be fixed on OS or Python level. Would you try to convince python core devs to ship Python with custom root certs? I recall @tiran <https://github.com/tiran> had objections for this approach. Honestly, I don't remember the whole pros/cons list. The security is very sensitive matter, I don't want to make the decision fast. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3180 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AhCkw2RMYH9DFIiAzaGlKj6c6fap7GZ7ks5uPIRFgaJpZM4VznQt> .

webknjaz · 2018-08-09T19:19:39Z

@creslinux while there are various projects supporting such feature, it's out of the scope of aiohttp to do so.

Moreover, as @asvetlov outlined, there's a security risk related to this feature. Given that aiohttp is and underlying building block for apps it would be unwise to have a library-specific env var for hijacking root CA. I can imagine that user having two apps in the same env wanting to add a trusted CA to one of them would unintentionally (probably lacking some knowledge) influence the other one as well, creating an extra "invisible" security hole. Also, we would like to avoid creation of the false sense of protection among users.

On the other hand, most of the software built on top of aiohttp will probably want their own env vars (unique names etc.), which would we non-conflicting and won't abuse global settings.

Based on the above I strongly encourage you to implement this on the level of your app.

creslinux · 2018-08-09T19:23:27Z

its not risk when the OS its build on uses them....
its aiohttp saying we'll use this part of the OS but not the other

Its true requests, twisted, aiohttp, ssl - are pillars that many apps are built upon.
Only 1 in that list does not support the vars.

Linux has no trusted key-store, its misnomer to point out it should be used.
Tell me my trusted keystore in linux ill point to 20 linux distributions that say different and the major ones that use the VARs to manage that problem.

creslinux · 2018-08-09T19:30:01Z

So in summary

requests - wrong
pip - wrong
ssl - wrong
twisted - wrong
openssl - wrong
redhatd - wrong
ubuntu - wrong
curl - wrong
wget - wrong
aiohttp - correct

creslinux · 2018-08-09T19:39:36Z

The thing is the Python SSL module does support the Env Vars - aiohttp simply ignores SSLs when its using them.

Thats the part i dont understand - why use SSL if going to ignore that its instructing to do.

https://access.redhat.com/articles/2039753#modifying-python-programs-to-control-certificate-verification-9

MyNameIsCosmo · 2018-08-09T19:57:38Z

Correct me if I'm wrong, but aiohttp uses SSL Context and does not directly handle ssl.wrap_context()
So global variables shouldn't apply to aiohttp

creslinux · 2018-08-09T20:08:24Z

python SSL sees the path to be used.
aiohttp *seemingly ignores it and only uses only 1 of the paths returned.

It is deciding which parts of the OS to honour and is unusual in this as other apps behave as the OS would intend.

SSL_CERT_DIR = /tmp
export SSL_CERT_DIR

import ssl
print("\n ssl Paths", ssl.get_default_verify_paths() ,"\n")

returns:

ssl Paths DefaultVerifyPaths(cafile=None, capath='/tmp', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/lib/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/lib/ssl/certs')

MyNameIsCosmo · 2018-08-09T20:09:03Z

https://docs.python.org/3/library/ssl.html#libressl-support

"SSLContext.set_default_verify_paths() ignores the env vars SSL_CERT_FILE and SSL_CERT_PATH although get_default_verify_paths() still reports them."

I think it makes sense to pass the cert from get_default_verify_paths to your ssl context.

If you think aiohttp should handle this, make a pull request with your changes for review.

Edit:
I just realized the link above is to LibreSSL support.
I'm unsure if the ssl module supports the global path variables.

webknjaz · 2018-08-09T20:09:17Z

@MyNameIsCosmo true

@creslinux (1) it depends on the SSL implementation (particularly, python doc emphasizes that LibreSSL ignores env vars) and (2) methods used from there (see more docs).

webknjaz · 2018-08-09T20:16:14Z

@creslinux as for your "wrong" accusations, it's not about being "right" or "incorrect". It's a design decision and keeping the responsibility for breaches on the user's side.
Following your suggestions, I could say that you're wrong trying to supersede maintainers' design decisions: we need to stick to certain design patterns to prevent the project from becoming a mess just because everyone wants their bells and whistles there. If you step back and take a look at the ecosystem here, you'll see that the currently encouraged way to go is to create a third-party extension to aiohttp.

creslinux · 2018-08-09T20:17:04Z

Thats a very tight cut and past - the full section reads

ssl.get_default_verify_paths()
Returns a named tuple with paths to OpenSSL’s default cafile and capath. The paths are the same as used by SSLContext.set_default_verify_paths(). The return value is a named tuple DefaultVerifyPaths:

cafile - resolved path to cafile or None if the file doesn’t exist,
capath - resolved path to capath or None if the directory doesn’t exist,
openssl_cafile_env - OpenSSL’s environment key that points to a cafile,
openssl_cafile - hard coded path to a cafile,
openssl_capath_env - OpenSSL’s environment key that points to a capath,
openssl_capath - hard coded path to a capath directory
Availability: LibreSSL ignores the environment vars openssl_cafile_env and openssl_capath_env

Thats a different argument libressl not supporting the full spec and not passing a path back so cant used is hardly a great argument for when it is passed back by the default build

Python certificate handling explicitly states openssl_capath_env and openssl_cafile_env are used by openssl. Thats 90% linux distributions right there.
https://docs.python.org/3/library/ssl.html#certificate-handling

creslinux · 2018-08-09T20:23:14Z

aiohttp is undermining how an OS, openssl, python ssl, and other popular pillars of python libraries manage company CAs.

There's no getting away from that. To ignore where the OS says to take certs from and claim its for security when only encourages people to disable verification doesn't really add up.

As a simple observation, guys n gals in deployment, users deploying etc - really are not in the main going to re-code and and pick apart an application that ignores their OS, they're just going to disable verification.

creslinux · 2018-08-09T20:42:57Z

As a final comment, what this policy means in Linux is:

Only a root user may verify the server connecting to is who they claim to be.
As only the root user may modify /etc

In windows and OSX user have their own keychains/ceritficate stores.
In Linux they have ENV vars.

tiran · 2018-08-10T00:12:22Z

The env vars SSL_CERT_FILE and SSL_CERT_PATH are handled by OpenSSL, not by Python. They override the location of the default CA cert file and CA cert directory for https://docs.python.org/3/library/ssl.html#ssl.SSLContext.set_default_verify_paths . aiohttp has to use SSLContext.load_default_certs, SSLContext.set_default_verify_paths, or ssl.create_default_context in order to load default CA certs from either the default location or SSL_CERT_FILE location.

LibreSSL ignores the vars.
The files must be readable by the current user. OpenSSL doesn't report any errors if the file is missing or not readable.
The env var must be set before the context is created.
Some runtime environments unset all env vars.

creslinux · 2018-08-10T04:46:28Z

LibreSSL ignores the vars.

If the var its not passed it wont be used. Same as any var

The major distributions are on OpenSSL by default. So unless an admin has explicitly moved away from default SSL this seems pretty moot. Debian by inference is Ubuntu/Mint etc.

Name	Release	System OpenSSL ver	TLS 1.1 and 1.2 Supported	Notes
Debian	10*	1.1.0	Yes	Buster (dev)
Debian	9	1.1.0 + 1.0.1	Yes	Stretch
Debian	8	1.0.1	Yes	Jessie
Debian	7	1.0.1	Yes	Wheezy
RedHat/CentOS	8*	1.1.0	Yes	(dev)
RedHat/CentOS	7	1.0.2	Yes
RedHat/CentOS	6	1.0.1	Yes

The files must be readable by the current user. OpenSSL doesn't report any errors if the file is missing or not readable.

True for all the paths returned by OpenSSL, including the host certificate store -- if a user is lucky enough their sys admin root user can be bothered to add a certificate, or accepts 1 users company CA should be added globally for ALL users on the host.

That would be a questionable security decision to force unto a multi user host.

Mr Foo from Company Foo MUST trust Company Bar's root CA as user Mrs Bar wants to trust her company Bar's CA? -- ???

The env var must be set before the context is created.

The var must be set to be read? well...

Some runtime environments unset all env vars.

The var must be set to be read? we....

And the flip side it

Aiohttp is out of step *most other popular python libraries in this area, inc ssl, requests, twisted etc
Only root users may have the privilege of knowing the server they are connecting to is that server
For 1 user to trust a CA means ALL users must trust a CA
aiohttp knows better and its a good policy to pick-and-choose the ca-paths the OS asks to be used
its fine to take the stance all users should pick apart and recode applications that for a puritan pursuit

Im sorry. This simple encourages to not verify servers. Its counter productive in the extreme. To take the stance you should either install a root CA for 1 user globally, must be root user, or must recode the application.

Im shocked - truly SHOCKED Google did not go down this path with Chrome on Linux

asvetlov · 2018-08-10T08:34:54Z

aiohttp calls create_default_context(): https://github.com/aio-libs/aiohttp/blob/master/aiohttp/connector.py#L761-L772
@tiran does it mean that SSL_CERT_FILE and SSL_CERT_DIR are processed?

creslinux · 2018-08-10T08:49:22Z

SSL_CERT_FILE and SSL_CERT_DIR not SSL_CERT_PATH are the ENV vars

asvetlov · 2018-08-10T09:03:45Z

Sorry, was my typo.

creslinux · 2018-08-10T09:17:24Z

exporting SSL_CERT_DIR =/tmp
results in print("ssl.get_default_verify_paths())
returning the verify paths to include a set capath

DefaultVerifyPaths(
cafile=None, 
capath='/tmp', 
openssl_cafile_env='SSL_CERT_FILE',
openssl_cafile='/usr/lib/ssl/cert.pem', 
openssl_capath_env='SSL_CERT_DIR', 
openssl_capath='/usr/lib/ssl/certs')

similar for SSL_CERT_FILE / cafile

the verify openssl_* paths are being honored but not capath / cafile verify paths.

tiran · 2018-08-10T11:11:50Z

aiohttp calls create_default_context(): /aiohttp/connector.py@master#L761-L772
@tiran does it mean that SSL_CERT_FILE and SSL_CERT_DIR are processed?

Yes, ssl.create_default_context() calls SSLContext.set_default_verify_paths, which calls SSL_CTX_set_default_verify_paths internally. https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_set_default_verify_paths.html will load the certificates from file or directory of hashed passwords. aiohttp does everything correctly.

asvetlov · 2018-08-10T11:16:19Z

Cool! Thank you!

creslinux · 2018-08-10T11:49:51Z

The link is good, aiohttp does not behave as it describes, it failing to verify servers when a hashed cert is in capath.

Can this be raised as a bug then?

The capath below is not null its shown to be capath=/tmp/certs/ printed from the python ssl module, and is a dir thats contains a hashed certificate that is shown in the same test to work when added to openssl_capath store.

from the link provided by @tiran
https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_set_default_verify_paths.html

Tested below.

Test script:

cat a.py 
import aiohttp
import asyncio

async def main():
    session = aiohttp.ClientSession()
    async with session.get('https://api.binance.com/api/v1/time') as response:
        print(await response.text())
    await session.close()

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

Run test script with no cert in SSL_CERT_DIR or in store (fails as expected)

creslin@creslab:~$ python3 a.py
Traceback (most recent call last):
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 822, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 804, in create_connection
    sock, protocol_factory, ssl, server_hostname)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 830, in _create_connection_transport
    yield from waiter
  File "/usr/lib/python3.6/asyncio/sslproto.py", line 505, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/usr/lib/python3.6/asyncio/sslproto.py", line 201, in feed_ssldata
    self._sslobj.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "a.py", line 12, in <module>
    loop.run_until_complete(main())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "a.py", line 6, in main
    async with session.get('https://api.binance.com/api/v1/time') as response:
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/client.py", line 843, in __aenter__
    self._resp = await self._coro
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/client.py", line 366, in _request
    timeout=timeout
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 445, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 757, in _create_connection
    req, traces, timeout)
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 879, in _create_direct_connection
    raise last_exc
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 862, in _create_direct_connection
    req=req, client_error=client_error)
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 827, in _wrap_create_connection
    raise ClientConnectorSSLError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host api.binance.com:443 ssl:None [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)]
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f48d70e8ef0>

set up SSL_CERT_DIR and add a hashed certificate

creslin@creslab:~$ mkdir /tmp/certs
creslin@creslab:~$ cp ca.crt /tmp/certs
creslin@creslab:~$ openssl x509 -hash -in /tmp/certs/ca.crt 
9426e611
-----BEGIN CERTIFICATE-----
MIICEzCCAXygAwIBAgIJAKBgdWawyyZnMA0GCSqGSIb3DQEBCwUAMA0xCzAJBgNV
BAMMAmZ0MB4XDTE4MDcyNzEwMDkyN1oXDTI4MDcyNDEwMDkyN1owDTELMAkGA1UE
AwwCZnQwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAK+lfec0yUzTHvyHkcKk
C/xZyZtb80D1pxa1/UpT3LpzAopEqtHSuBosx7yhWBMjcvUCk1fkk9CCyfrdd1Q1
784rba+AMR4XbX0GHShpYchmkBHaoAk9zRMcgHrAhLux/Mp3rR8oOO4JJI0g0OHn
1s674hbS2cmNnfFeC7jy3rUjAgMBAAGjezB5MB0GA1UdDgQWBBT6JFAIzTCwTQem
Gv/ttlGxrJ3jlzA9BgNVHSMENjA0gBT6JFAIzTCwTQemGv/ttlGxrJ3jl6ERpA8w
DTELMAkGA1UEAwwCZnSCCQCgYHVmsMsmZzAMBgNVHRMEBTADAQH/MAsGA1UdDwQE
AwIBBjANBgkqhkiG9w0BAQsFAAOBgQBNpWNFWz68dRDvsVlAYTZKegCJsholBwW4
BaGKvqA8BtZTm/B6in3NmFjNH3AafRv3qceX1oOE0NNsC4aBKt34TovuE/bNehid
mitqeptL14lJpDSjltIh6QjMBEuhgnlhUB9zuPwOfJbdJihNK5H9Mys3/Zpflz0H
creX6zj9GQ==
-----END CERTIFICATE-----
creslin@creslab:~$ cp /tmp/certs/ca.crt /tmp/certs/9426e611
creslin@creslab:~$ ls -al /tmp/certs/9426e611
-rw-rw-r-- 1 creslin creslin 782 Aug 10 14:28 /tmp/certs/9426e611

set, export, SSL_CERT_DIR
print ssl.get_default_verify_paths from within python to show SSL see the capath /tmp/certs

cat s.py 
import ssl
print("\n ssl Paths", ssl.get_default_verify_paths() ,"\n")

creslin@creslab:~$ SSL_CERT_DIR=/tmp/certs/
creslin@creslab:~$ export SSL_CERT_DIR
creslin@creslab:~$ python3 s.py 

 ssl version OpenSSL 1.1.0g  2 Nov 2017

 ssl Paths DefaultVerifyPaths(cafile=None, capath='/tmp/certs/', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/lib/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/lib/ssl/certs')

Run test script again - still fails <<---- This is where its broken. We've show SSL module sees capath and there is a hashed certificate in here.

creslin@creslab:~$ python3 a.py
Traceback (most recent call last):
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 822, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 804, in create_connection
    sock, protocol_factory, ssl, server_hostname)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 830, in _create_connection_transport
    yield from waiter
  File "/usr/lib/python3.6/asyncio/sslproto.py", line 505, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/usr/lib/python3.6/asyncio/sslproto.py", line 201, in feed_ssldata
    self._sslobj.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "a.py", line 12, in <module>
    loop.run_until_complete(main())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "a.py", line 6, in main
    async with session.get('https://api.binance.com/api/v1/time') as response:
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/client.py", line 843, in __aenter__
    self._resp = await self._coro
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/client.py", line 366, in _request
    timeout=timeout
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 445, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 757, in _create_connection
    req, traces, timeout)
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 879, in _create_direct_connection
    raise last_exc
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 862, in _create_direct_connection
    req=req, client_error=client_error)
  File "/home/creslin/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 827, in _wrap_create_connection
    raise ClientConnectorSSLError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host api.binance.com:443 ssl:None [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)]
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7efc256f9ef0>

Add to OS store

creslin@creslab:~$ sudo mkdir /usr/local/share/ca-certificates/test/
creslin@creslab:~$ sudo cp /tmp/certs/ca.crt /usr/local/share/ca-certificates/test/
creslin@creslab:~$ sudo update-ca-certificates
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

done.
Updating Mono key store
Mono Certificate Store Sync - version 5.12.0.226
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 133, your new list has 134
Certificate added: CN=ft
1 new root certificates were added to your trust store.
Import process completed.

Importing into BTLS system store:
I already trust 133, your new list has 134
Certificate added: CN=ft
1 new root certificates were added to your trust store.
Import process completed.
Done
done.

Now works.

creslin@creslab:~$ unset SSL_CERT_DIR
creslin@creslab:~$ python3 a.py
{"serverTime":1533900636485}

asvetlov · 2018-08-10T12:26:29Z

@creslinux could you try your snippet with explicitly crafted SSLContext?

creslinux · 2018-08-10T12:30:55Z

I could but that would be another test / different thing

The issue/bug im chasing down is aiohttp is not following OS/openssl/python-ssl guidence.

Every link being pointed at by others explicitly states capath when passed by openssl should be used to look for a certificate. There seems to be a paper-review opinion that "everything is correct" but technical testing shows it is not as assumed.

The challenge is users of apps / apps being deployed into environments where "their" company ca certificate is in a location deployed for general use.

creslinux · 2018-08-10T12:32:42Z

(plus my 6mnth puppy is telling me its her toilet/walk time -- and ... well, carpet priorities)

creslinux · 2018-08-10T13:09:07Z

@tiran wrote

Yes, ssl.create_default_context() calls SSLContext.set_default_verify_paths, which calls >SSL_CTX_set_default_verify_paths internally. >https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_set_default_verify_paths.html will load the >certificates from file or directory of hashed passwords. aiohttp does everything correctly.

As tests show will load the certificates from file or directory of hashed passwords is false, isn't this then a bug?

asvetlov · 2018-08-10T13:15:18Z

Reminder: asyncio uses MemoryBIO, requests and others work with wrap_socket.

@creslinux I have no certificates to test myself, I need your help to pin down the problem.

webknjaz · 2018-08-10T20:18:26Z

@asvetlov JFYI I'm trying to track SSL testing helpers here: cherrypy/cheroot#95. Maybe we could try to add trustme to the test suite?

ldbfpiaoran · 2018-10-29T07:22:46Z

@asvetlov So how do you ignore the ssl certificate error? Is there a parameter like requests verify=False?

asvetlov · 2018-10-29T07:38:35Z

RTFM please: http://docs.aiohttp.org/en/stable/client_advanced.html#ssl-control-for-tcp-sockets
r = await session.get('https://example.com', ssl=False)

ldbfpiaoran · 2018-10-29T07:51:51Z

@asvetlov
Thank you, but I used your method to get an exception.
TypeError: _request() got an unexpected keyword argument 'ssl'
The code is like this
Async def scan(url):
     Url = url +'/trace'
     # try:
     Async with aiohttp.ClientSession() as session:
         Res = await session.get(url,ssl=False).text()

asvetlov · 2018-10-29T07:55:15Z

Perhaps you use an old aiohttp version

ldbfpiaoran · 2018-10-29T08:00:38Z

@asvetlov thank you very much

lock · 2019-10-29T08:01:29Z

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.

creslinux mentioned this issue Aug 8, 2018

Using asyncio/aiohttp causes certificate failure creslinux/freqcache#6

Closed

creslinux changed the title ~~Own root ca.pem - ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)~~ aiohttp ignoring SSL_CERT_DIR and or SSL_CERT_FILE environment vars. Results in [SSL: CERTIFICATE_VERIFY_FAILED] Aug 8, 2018

asvetlov closed this as completed Aug 8, 2018

asvetlov added the invalid This doesn't seem right label Aug 8, 2018

webknjaz added the question StackOverflow label Aug 9, 2018

TheSnoozer mentioned this issue Dec 18, 2018

aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host discordapp.com:443 ssl:None [[SSL] unknown error (_ssl.c:847)] zacholade/Rift-Companion#10

Closed

webknjaz mentioned this issue Jan 5, 2019

Integrate trustme for testing TLS support against a "real CA" with "real" certificates #3487

Closed

5 tasks

lock bot added the outdated label Oct 29, 2019

lock bot locked as resolved and limited conversation to collaborators Oct 29, 2019

aiohttp ignoring SSL_CERT_DIR and or SSL_CERT_FILE environment vars. Results in [SSL: CERTIFICATE_VERIFY_FAILED] #3180

aiohttp ignoring SSL_CERT_DIR and or SSL_CERT_FILE environment vars. Results in [SSL: CERTIFICATE_VERIFY_FAILED] #3180

Comments

creslinux commented Aug 8, 2018 • edited Loading

Long story short

Expected behaviour

Actual behaviour

Steps to reproduce

Your environment

webknjaz commented Aug 8, 2018

creslinux commented Aug 8, 2018

asvetlov commented Aug 8, 2018

creslinux commented Aug 8, 2018

asvetlov commented Aug 8, 2018

creslinux commented Aug 8, 2018

webknjaz commented Aug 9, 2018

creslinux commented Aug 9, 2018 • edited Loading

creslinux commented Aug 9, 2018

webknjaz commented Aug 9, 2018

creslinux commented Aug 9, 2018

asvetlov commented Aug 9, 2018

webknjaz commented Aug 9, 2018

creslinux commented Aug 9, 2018 via email

asvetlov commented Aug 9, 2018

webknjaz commented Aug 9, 2018

asvetlov commented Aug 9, 2018

creslinux commented Aug 9, 2018 via email

webknjaz commented Aug 9, 2018

creslinux commented Aug 9, 2018 • edited Loading

creslinux commented Aug 9, 2018 • edited Loading

creslinux commented Aug 9, 2018

MyNameIsCosmo commented Aug 9, 2018

creslinux commented Aug 9, 2018 • edited Loading

MyNameIsCosmo commented Aug 9, 2018 • edited Loading

webknjaz commented Aug 9, 2018

webknjaz commented Aug 9, 2018

creslinux commented Aug 9, 2018 • edited Loading

creslinux commented Aug 9, 2018 • edited Loading

creslinux commented Aug 9, 2018 • edited Loading

tiran commented Aug 10, 2018

creslinux commented Aug 10, 2018 • edited Loading

asvetlov commented Aug 10, 2018 • edited Loading

creslinux commented Aug 10, 2018

asvetlov commented Aug 10, 2018

creslinux commented Aug 10, 2018

tiran commented Aug 10, 2018

asvetlov commented Aug 10, 2018

creslinux commented Aug 10, 2018 • edited Loading

asvetlov commented Aug 10, 2018

creslinux commented Aug 10, 2018 • edited Loading

creslinux commented Aug 10, 2018

creslinux commented Aug 10, 2018

asvetlov commented Aug 10, 2018

webknjaz commented Aug 10, 2018

ldbfpiaoran commented Oct 29, 2018

asvetlov commented Oct 29, 2018

ldbfpiaoran commented Oct 29, 2018

asvetlov commented Oct 29, 2018

ldbfpiaoran commented Oct 29, 2018

lock bot commented Oct 29, 2019

creslinux commented Aug 8, 2018 •

edited

Loading

creslinux commented Aug 9, 2018 •

edited

Loading

creslinux commented Aug 9, 2018 •

edited

Loading

creslinux commented Aug 9, 2018 •

edited

Loading

creslinux commented Aug 9, 2018 •

edited

Loading

MyNameIsCosmo commented Aug 9, 2018 •

edited

Loading

creslinux commented Aug 9, 2018 •

edited

Loading

creslinux commented Aug 9, 2018 •

edited

Loading

creslinux commented Aug 9, 2018 •

edited

Loading

creslinux commented Aug 10, 2018 •

edited

Loading

asvetlov commented Aug 10, 2018 •

edited

Loading

creslinux commented Aug 10, 2018 •

edited

Loading

creslinux commented Aug 10, 2018 •

edited

Loading