Missing ca-certificates

[denibertovic]

When trying out r <- get "http://httpbin.org/get" from the tutorial on a Debian system with no ca-certificates package installed (meaning no /etc/ssl/certs directory) the command exits with an Exception "*** Exception: /etc/ssl/certs/: getDirectoryContents: does not exist (No such file or directory)".

Seems we should handle this more gracefully (ie allow the user not to check certificates perhaps? in Python requests that verify=False I believe).

The other question is why was this triggered when the URL that's being accessed is http only?

Awesome work btw! 👍

[bos]

This sounds like the tls package being more eager than makes sense. Neither wreq nor http-client-tls contains any code that I'm aware of that cares about certificates. Maybe @vincenthz has some clue what might be going on?

[neko-kai]

@bos @snoyberg

Neither wreq nor http-client-tls contains any code that I'm aware of that cares about certificates

During Manager creation http-client-tls parses system certs twice or thrice depending on whether socks proxy is used. https://github.com/snoyberg/http-client/blob/master/http-client-tls/Network/HTTP/Client/TLS.hs#L19 (initConnectionContext in getTls[Proxy]Connection calls getSystemCertificateStore)

Old http-conduit-1 had code to delay parsing until a secure request is made.

[vincenthz]

@bos it's all in the connection package, and yes it's quite eager at the moment (I haven't used connection to not do a https connection apart from testing in a while, so haven't noticed it). If someone want to make it load on demand instead, I gladly accept a PR.

@denibertovic I fixed the underlaying issue with the missing certificate directory (x509-system 1.4.5), but beware if you try to do any https connection they will all fail with certificate validation errors (unknown CA), unless you turn off certificate validation, which is usually a bad idea, or that you put certificate fingerprint exceptions (safe & recommended provided you do validate the identity yourself).

[ocharles]

We're seeing the same in NixOS - http://hydra.nixos.org/build/10663548/nixlog/1/raw.

[vincenthz]

@ocharles you're not using x509-system 1.4.5

[denibertovic]

@vincenthz @bos I can confirm that with x509-system 1.4.5 the exception no longer happens when making HTTP requests.

When trying to do a HTTPS request the exception I get is:

*** Exception: FailedConnectionException2 "httpbin.org" 443 True getProtocolByName: does not exist (no such protocol name: tcp)

@vincenthz This is the Exception you talked about in you comment (about unknown CA) right? If so could it be made a little bit more descriptive? Or did I hit another exception?

[snoyberg]

That doesn't sound like a TLS issue at all; that exception comes from the network package, and the FailedConnectionException2 is coming from http-client as a way to give more information to the error message (specifically, host/port we were connecting to, and whether it was a secure connection).

What happens when you try to make a non-HTTPS connection?

[denibertovic]

@snoyberg Yeah that's what I thought. A non-HTTPS connection works just fine.

[snoyberg]

Huh, that's very strange. I can think of no reason why this would be TLS-dependent. I guess it's possible that the getprotobyname call is erroring out for some other reason.

@vincenthz Is it possible that tls is doing something to open up the protocols file, and not closing it immediately thereafter? It looks like network is taking some kind of global lock every time it tries to make a call that would access that file.

[denibertovic]

Any progress on this? I'd be happy to test again if there has been a new release or something?

[vincenthz]

@denibertovic are you using a socks proxy ?

[denibertovic]

@vincenthz No proxy in use, no.

[codygman]

I'm also affected by the https bug. Is there any work-around? I'm using x509-system==1.4.5 already. My program works fine on my local computer (Ubuntu 14.04) but it doesn't work on my Ubuntu 14.04 LTS server. I'm guessing that I don't have something installed on the server which I have on my development machine?

EDIT: I was apparently using

GHC-Options:         -with-rtsopts=-T -O2 -optl-pthread -optl-static

Without -optl-pthread and -optl-static things work fine for me.

[denibertovic]

@codygman Did you install the package "ca-certificates" ?

[noteed]

I don't know if this is related to this issue, but I was having

getProtocolByName: does not exist (no such protocol name: tcp)

in a project of mine, and I solved it by apt-get install netbase, which provides /etc/protocols.

[tomfitzhenry]

I'm running Ubuntu 14.04's Docker image, and also ran into this problem. Installing Ubuntu's netbase package resolved this for me too.

[ondrap]

I'm closing this as it seems it's not quite wreq related.