linkcheck does not use SNI

[shimizukawa]

When checking an HTTPS link hosted on a server that requires SNI, linkcheck fails to establish a connection (or perhaps "connects" to the wrong virtual host). This causes spurious link check failures with errors like:

<urlopen error [Errno 1] _ssl.c:510: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure>

and makes make linkcheck exit with error.

---

• Bitbucket: https://bitbucket.org/birkenfeld/sphinx/issue/1655
• Originally reported by: Jean-Paul Calderone
• Originally created at: 2014-12-19T14:38:35.706

[shimizukawa]

From Takayuki Shimizukawa on 2014-12-19 13:50:21+00:00

I think the issue relates to #1373.

[shimizukawa]

From Takayuki Shimizukawa on 2014-12-19 14:03:46+00:00

Duplicate of #1373.

[shimizukawa]

From Jean-Paul Calderone on 2014-12-19 14:05:16+00:00

This isn't a duplicate. This isn't about validating certificates. It's about using the SNI TLS protocol feature so it's possible to begin a TLS handshake at all.

[shimizukawa]

From Takayuki Shimizukawa on 2014-12-19 14:08:16+00:00

Which combination of versions are you using?

[shimizukawa]

From Jean-Paul Calderone on 2014-12-19 14:13:32+00:00

I think this affects pretty much all versions. linkcheck uses urllib (well, six.moves.urllib) which doesn't do SNI on any version of Python ever.

I can verify that this happens against Python 2.7.6 / Sphinx 1.2.2 specifically, though.

[shimizukawa]

From Takayuki Shimizukawa on 2014-12-19 14:43:23+00:00

I tested SNI with several python version.

• Python-2.7.6: NG
• Python-2.7.9: OK ( ssl module is backported from Python3)
• Python-3.2.5: OK
• Python-3.3.5: OK
• Python-3.4.1: OK

The cause of this problem that you reported lies in traditional Python ssl module.
I think it is not possible to fix from Sphinx side.

[shimizukawa]

From Jean-Paul Calderone on 2014-12-19 14:49:00+00:00

Switching to a different HTTP client (not urllib) would be one way to fix it.

[shimizukawa]

From Jean-Paul Calderone on 2014-12-19 14:49:59+00:00

I tested SNI with several python version.

What did you test, exactly? Not six.moves.urllib? My understanding is that doesn't do SNI on Python 2.7.9 - but maybe I missed something.

[shimizukawa]

From Tom Prince on 2014-12-21 02:26:45+00:00

Testing with fedora rawhide (python 2.7.9), it appears that SNI works.

[shimizukawa]

From Tom Prince on 2014-12-21 05:53:19+00:00

On the other hand, I believe that pip uses ndg-httpsclient to support SNI on python 2.7.

[shimizukawa]

ref: https://github.com/kennethreitz/requests/issues/749 it seems hard.

[mgifford]

Any progress on this. There are way more https sites now it seems than a year ago.

[shimizukawa]

Actually, no progress. I think sphinx should use requests to fix this problem, but it's a big change.

[tk0miya]

@shimizukawa In master branch, we already use requests to check links at #2389.
Does this help this issue?

[shimizukawa]

Oh, I hadn't realized that.. Thanks!