Disable HTTP2 for subdomain

[lenovouser]

I need to disable HTTP2 for my subdomain ws.domain.tld as WebSockets don't seem too work with HTTP2 yet. Is there any way to do that? Couldn't find one in the docs.

[mholt]

Hmm, I thought about that at GopherCon because the whole HTTP/2-WebSockets thing was mentioned... right now the only way I can think of is to run that subdomain on a different instance of Caddy and use the -http2=false flag.

If your client makes an HTTP 1.1 request the server will only respond with 1.1.

[ghost]

@mholt perhaps having an option to modify this behavior in the domain blocks, e.g:

mysite.com {
    ..
}

sub.mysite.com {
    http2 false
}

or something like that.

[lenovouser]

Yes. I really like @norwack's suggestion. Two instances would be possible but I could not let both of them run on standard ports, which would be really annoying.

[mholt]

HTTP/2 is configured at the server-level, not the virtualhost level, meaning that it has to be on or off for the whole server.

Can the client just make an HTTP 1.1 request when initiating a WS connection?

[mholt]

@Apfeluser You should be able to use bind 0.0.0.0 to hack your way around that port restriction.

[lenovouser]

@mholt I just tried it and get this error when I start the 2nd process:

2015/07/16 22:03:29 listen tcp 0.0.0.0:443: bind: address already in use

[lenovouser]

Caddy doesn't even start with a websocket directive by the way. It just crashes

[mholt]

@Apfeluser Re: websocket causing it to crash, what's the error message? All tests pass and it runs fine here using websocket.

Also can I get your system information? OS/arch? Privileged or regular user? Caddy version? etc.

[lenovouser]

I am sorry. It doesn't crash. But it doesn't work and gives a 502 Bad Gateway whenever I use the websocket directive. Also on normal HTML documents or CSS files.

I am using:

• Debian 8 x64
• Root user
• Newest build from download page with git included.

Do you need hardware info too? I doubt it, but you never know 😄

[mholt]

No, that's fine. Anything in your error log? And you get a 502 serving static files? Sounds like you have a proxy or fastcgi directive in there too. Let me see your Caddyfile, that will help.

[lenovouser]

It is a NodeJS app. It serves static files which get inaccessible once I use the websocket directive.

Caddyfile (Static files work, websockets don't):

https://domain.tld {
        bind 0.0.0.0
        tls /x/ssl.crt /x/ssl.key
        gzip
        log /x/log/webserver/access.log
        errors /x/log/webserver/error.log
        proxy / 0.0.0.0:5021 0.0.0.0:5022 0.0.0.0:5023 0.0.0.0:5024 0.0.0.0:5025 {

        }
}

Caddyfile (Static files and websockets don't work):

https://domain.tld {
        bind 0.0.0.0
        tls /x/ssl.crt /x/ssl.key
        gzip
        log /x/log/webserver/access.log
        errors /x/log/webserver/error.log
        proxy / 0.0.0.0:5021 0.0.0.0:5022 0.0.0.0:5023 0.0.0.0:5024 0.0.0.0:5025 {
                websocket
        }
}

Starting 2 caddy processes with bind 0.0.0.0 for using a HTTP1.1 subdomain (ws.domain.tld) fails with:

listen tcp 0.0.0.0:443: bind: address already in use

[lenovouser]

The error.log is full with:

16/Jul/2015:23:19:22 +0200 [ERROR 502 /socket.io/] unreachable backend
16/Jul/2015:23:19:23 +0200 [ERROR 502 /] unreachable backend
16/Jul/2015:23:20:21 +0200 [ERROR 502 /] unreachable backend
16/Jul/2015:23:20:22 +0200 [ERROR 502 /socket.io/] unreachable backend
16/Jul/2015:23:20:25 +0200 [ERROR 502 /socket.io/] unreachable backend
16/Jul/2015:23:20:51 +0200 [ERROR 502 /] unreachable backend
16/Jul/2015:23:20:52 +0200 [ERROR 502 /socket.io/] unreachable backend
16/Jul/2015:23:20:53 +0200 [ERROR 502 /socket.io/] unreachable backend
16/Jul/2015:23:20:58 +0200 [ERROR 502 /socket.io/] unreachable backend
16/Jul/2015:23:20:59 +0200 [ERROR 502 /socket.io/] unreachable backend
16/Jul/2015:23:21:51 +0200 [ERROR 502 /] unreachable backend
16/Jul/2015:23:22:00 +0200 [ERROR 502 /socket.io/] unreachable backend
16/Jul/2015:23:22:51 +0200 [ERROR 502 /] unreachable backend
16/Jul/2015:23:22:56 +0200 [ERROR 502 /socket.io/] unreachable backend
16/Jul/2015:23:22:56 +0200 [ERROR 502 /socket.io/] unreachable backend
16/Jul/2015:23:23:51 +0200 [ERROR 502 /] unreachable backend
16/Jul/2015:23:24:51 +0200 [ERROR 502 /] unreachable backend
16/Jul/2015:23:24:52 +0200 [ERROR 502 /socket.io/] unreachable backend
16/Jul/2015:23:24:54 +0200 [ERROR 502 /socket.io/] unreachable backend
16/Jul/2015:23:24:58 +0200 [ERROR 502 /socket.io/] unreachable backend
16/Jul/2015:23:25:00 +0200 [ERROR 502 /socket.io/] unreachable backend
16/Jul/2015:23:25:51 +0200 [ERROR 502 /] unreachable backend
16/Jul/2015:23:25:52 +0200 [ERROR 502 /socket.io/] unreachable backend
16/Jul/2015:23:25:54 +0200 [ERROR 502 /socket.io/] unreachable backend

[mholt]

Ohhh, you're proxying websockets. It will forward all requests to any of those backends with Connection and Upgrade headers. Have you confirmed that your backends are up?

As for the bind issue, only bind one Caddy instance to 0.0.0.0 - let the other resolve the IP. See if that works.

[lenovouser]

The backend is definitely online, as I already tried the same thing with NGINX and it works.

I got an error earlier today when I didn't use bind due to my hostname resolving to a different IP than the actual one. Which is not true, I've checked that.

[lenovouser]

@mholt sadly I gotta go to bed. The app I am trying to proxy is NodeBB. It heavily relies on WebSockets and barely functions without them. I found out you can configure it to use a subdomain for WebSocket requests and tried to do that, but that didn't work as you see in this issue 😄 Normally the WebSockets run over the same domain as the forum itself but this also doesn't work because of the 502 errors when I use the directive. I'll open a issue on their bug tracker tomorrow asking if they can configure the WebSockets to always use HTTP/1.1 but then there would still be the issue with the 502's even if they get that working (Which I doubt at the moment). If you have time you can maybe play around with it on a DO droplet or something else so you get a better feeling of what the problem is. I also could spin up a droplet for you real quick if you need it. I'll be back in about 8-10 hours or to be precise at 10:00 AM UTC+2. Here is my NodeBB configuration for easier setup, just in case you're trying out:

{
    "url": "https://community.domain.tld",
    "port": ["5021", "5022", "5023", "5024", "5025"],
    "secret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "database": "redis",
    "secondary_database": "mongo",
    "redis": {
        "host": "127.0.0.1",
        "port": "6379",
        "password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "database": "5"
    },
    "secondary_db_modules": "hash, list, sets, sorted",
    "mongo": {
        "host": "127.0.0.1",
        "port": "27017",
        "username": "community",
        "password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "database": "community"
    },
    "socket.io": {
        "transports": ["websocket", "polling"],
        "address": "ws.domain.tld"
    }
}

I currently use 2 databases, but you can just remove the secondary_database and just use 1 for faster testing.

My HTTP/2 caddyfile:

https://community.domain.tld {
        bind 0.0.0.0
        tls /x/ssl/domain.crt /x/ssl/domain.key
        gzip
        log /x/log/webserver/access/domain.log
        errors /x/log/webserver/error/domain.log
        proxy / 0.0.0.0:5021 0.0.0.0:5022 0.0.0.0:5023 0.0.0.0:5024 0.0.0.0:5025 {

        }
}

Starting this one with ./caddy -conf="/x/config/webserver/domain.conf"

And my HTTP/1.1 (WebSocket) caddyfile:

https://ws.domain.tld {
        bind 0.0.0.0
        tls /x/ssl/domain.crt /x/ssl/domain.key
        log /x/log/webserver/access/domain.ws.log
        errors /x/log/webserver/error/domain.ws.log
        proxy / 0.0.0.0:5021 0.0.0.0:5022 0.0.0.0:5023 0.0.0.0:5024 0.0.0.0:5025 {
                websockets
        }
}

Starting this one with ./caddy -http2=false -conf="/x/config/webserver/domain.ws.conf"

The error I get when starting 2 caddy processes (also with one not using bind)

2015/07/17 00:43:12 listen tcp 188.165.213.33:443: bind: address already in use

Thanks a lot for your help already! Appreciate it a lot!

Cheers,

ApfelUser

[mholt]

Cool, thanks for the info dump! And for your patience. I'll take a look... no guarantees it'll be tonight or anything, but this is something that I think should be fixed.

[lenovouser]

@mholt Yeah, it doesn't seems like it is possible to choose a HTTP version with javascript as e.g. this Google Search shows.

[lenovouser]

@mholt Any updates?

Also, is there any way to make the people who design the HTTP/2 specification aware of this issue? I've looked at the http2 specification, and WebSockets aren't even mentioned there. There seems to be a draft here. I'll comment on httpwg/http2-spec#386 but I don't know if that helps.

[mholt]

Not yet, sorry. Given that the http2 spec is still kind of crawling more than I had anticipated, I'm considering moving the -http2=false command line option into the Caddyfile as something like http2 off or http 1 to specify the version. It's going to complicate the initialization code since http2 and http1 servers need to be separated, and I don't even know at this point if it's feasible/possible with the current architecture, but it's something that can be looked into.

This is where I'm swamped with quite a few other things with this project right now and am looking for anyone to collaborate with - let me know if you're interested in giving this a go.

[lenovouser]

Yeah, I really like @norwack's idea:

mysite.com {
    ..
}

sub.mysite.com {
    http2 false
}

This is the only thing currently holding me off from using caddy, as we heavy-use WebSockets. Sadly I don't know Golang, so I can't help with that. I have no problem waiting, I just hope it will be implemented before NGINX does it because then the reasons to switch to Caddy get fewer and fewer 😄 I can't convince my colleagues then anymore 😞

[mholt]

Unfortunately, I don't think this is possible at the moment. I can't bind two listeners to port 443, and I need two listeners because one needs to serve HTTP/1.1 and the other serves HTTP/2.

We will have to wait until at least one of these happens (in order of increasing preference):

• Our http2 lib matures (would still require some hacking on my part, I think, to support http/2 on a per-domain basis)
• Clients provide the ability to force HTTP/1.1 connections
• The HTTP/2 spec supports WebSockets (connection upgrade)

For now, clients should make websocket requests over HTTP/1.1. Sorry if you don't have control over this. Thing is, if the client can make an HTTPS connection to Caddy and the client does not indicate that it supports HTTP/2, Caddy will serve HTTP/1.1 over TLS which is exactly what you want for your websocket app.

I'll keep looking into this, of course, but in the meantime while the http/2 spec is under construction, it might be worth pushing for the ability in clients to force HTTP/1.1 connections.

[mholt]

For the record, I'm going to keep my eye on the development of the now-semi-official-Go http2 package and see if this becomes a possibility. If it doesn't, I might chime in with an issue. But for now I want them to get through the backlog of PRs and issues.

[augustoroman]

Now that http/2 is officially in the stdlib, any further thoughts on this?

[mholt]

It's more complicated than that. The protocol has to be decided before the Host is read from the headers, because how the headers are transmitted depends on the protocol.

[danderson]

Thinking about this a bit because this is problematic for me, here are some thoughts.

HTTP2 gets selected during the TLS handshake. So, the only way to do per-domain protocol selection is by manipulating the TLS handshake, based on the TLS SNI (server name indication).

Go 1.7 does not support this easily, you'd have to have two TLS listeners and hand-parse the TLS ClientHello to figure out which of the two listeners should handle the connection. Eww.

The good news is Go 1.8's crypto/tls implements exactly what is needed, a GetConfigForClient hook that lets you inspect the TLS ClientHello and return a customized TLS configuration to use. It's in https://tip.golang.org/pkg/crypto/tls/#Config .

So, with Go 1.8, Caddy could support per-site HTTP2 negotiation, by using that hook to select what protocol(s) to offer to each new connection.

Go 1.8 is due out any time now. Supporting this is also fairly relevant to my interests, so I'd be interested in offering a pull request to make this work (something like an http2 off site directive, to mirror tls off ?). Would you accept such a pull request if I can put one together, obviously after Go 1.8 is officially out?

I'm currently proxying a server through Caddy that wants to use websockets, and so I'm stuck between either breaking part of its functionality or turning H2 off completely. I'd rather have the third option :)

[mholt]

@danderson That sounds plausible -- and super useful -- wanna give it a shot? You can build on #1389 by @wendigo if you want to submit a PR!

[wendigo]

I've added HTTP/2 disabling to #1389 :)

[mholt]

This is now possible for a single site:

tls {
    alpn http/1.1
}

Thanks @wendigo !

[dolanor]

I was talking about this issue with other people and I saw that it is now fixed, thanks @wendigo !
But a fellow gopher (Egon Elbre) shared some go playground link that works out of the box.
https://play.golang.org/p/bal1ciaeGn
Here, the HTTP GET on / happens on HTTP2 but the javascript wesbsocket code does a / on wss:// scheme and it is HTTP1.1 in the network console of firefox.
So, did 1.8 made that completely automatic? Why did we need the patch from @wendigo? Am I comparing apple to oranges?

[mholt]

@dolanor That's quite possible; but it's still useful to have this change because there may be reasons that people want to disable http/2 entirely.

[daiaji]

@mholt This update is very useful, and there are some exceptions to WebDav that must work under HTTP 1.1.
It seems that KeePass's PC version has similar problems.
PhilippC/keepass2android#747

[martindale]

This is now possible for a single site:

tls {
    alpn http/1.1
}

Thanks @wendigo !

As of v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw= this appears to cause the following TLS error attempting to access a site hosted using this configuration:

$ curl https://mydomain.com
curl: (35) error:14094460:SSL routines:ssl3_read_bytes:reason(1120

[mholt]

@martindale if you believe this is a bug in Caddy, please open a new issue rather than commenting on an unrelated 7 year old closed issue. Thank you!