FastHttpLocust gives ssl error with let's encrypt certs

[matti]

SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)')

domains which have regular cert work okay. also the normal http client works okay.

os: alpine linux (with apk add ca-certificates)
python: 3.7
locust master branch

[cyberw]

I got the same error today (unsure about what kind of cert the site is using).

Using geventhttpclient directly works, so does not seem to be an issue with geventhttpclient itself.

e.g. this works:

from geventhttpclient import HTTPClient
from geventhttpclient.url import URL
url = URL("https://mydomain")
http = HTTPClient(url.host)
response = http.post(url.request_uri)

[heyman]

@cyberw I don't think your example will use SSL/TLS since you use url.host which won't include https://.

This example should fail:

from geventhttpclient import HTTPClient
from geventhttpclient.url import URL
url = URL("https://mydomain")
http = HTTPClient.from_url(url)
response = http.post(url.request_uri)

[cyberw]

@heyman yes, you are right.

[cyberw]

Gah. I'm completely stuck on this. Any ideas @skivis ?

I'm fine with disabling the checks, but I cant figure out how to do that either.

[cyberw]

I finally figured out how to disable the check (without editing my pip-installed geventhttpclient in place :)

I added this to my locustfile:

import gevent
import geventhttpclient.connectionpool

geventhttpclient.connectionpool.SSLConnectionPool.default_options = {
    "cert_reqs": gevent.ssl.CERT_NONE,
}

[heyman]

I think changing this line:

locust/locust/contrib/fasthttp.py

Line 91 in cfad799

self.client = LocustUserAgent(max_retries=1, cookiejar=self.cookiejar, **kwargs)

to:

self.client = LocustUserAgent(max_retries=1, cookiejar=self.cookiejar, insecure=True, **kwargs)

should disable SSL verification, which I think should be okay for us to do in a client which is supposed to be as fast as possible (in a load testing tool).

[heyman]

Should be fixed by 66d68da

[cyberw]

I could have sworn I tried that (or at least something similar), but it didnt work for me. looks good now though!

[cyberw]

... on second thought, it doesnt work for self signed certificates. I still get [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)

But my workaround does work.

[heyman]

on second thought, it doesnt work for self signed certificates

Ah, good catch! I've incorporated your fix in 274677f

I made a small locustfile for testing all sorts of different SSL/TLS scenarios using the neat https://badssl.com/ website. Posting it here for future reference:

from locust import HttpLocust, TaskSet, task, constant
from locust.contrib.fasthttp import FastHttpLocust
from pyquery import PyQuery


class UserTasks(TaskSet):
    def get(self, url):
        return self.client.get(url, name=url)
    
    @task
    def test_bad_ssl(self):
        response = self.get("https://badssl.com")
        pq = PyQuery(response.text)
        for link in pq("#links a:not(.external)"):
            url = link.attrib["href"]
            if not url.startswith("https://"):
                continue
            self.get(url)
    
class SSLTestUser(FastHttpLocust):
    host = "https://badssl.com"
    wait_time = constant(1)
    task_set = UserTasks

It depends in PyQuery. Run it with:

$ locust -f examples/ssl_test.py -c 1 -r 1 -t 1 --no-web --stop-timeout 300

Currently it only fails for the client certificate pages, which makes sense, and the subdomain.preloaded-hsts which also doesn't seem strange since geventhttpclient doesn't support HSTS.

[cyberw]

Great! badssl.com is a neat concept!

[matti]

thanks for fixing this, even if the fix is kinda weird :)