NodeJS unable to make https requests when h2 enabled in Apache 2.4.18

[tunetheweb]

Since upgrading to Apache 2.4.18 node is unable to get https requests, though I can see the request in the log files and it is returning a 200 status.

I have the following protocols set:

Protocols h2 http/1.1

Turning off h2 in the protocols, or rolling back to 2.4.17 resolves the issue. Node currently makes a http/1.1 request so looks like some incompatibility. Not sure if node is at fault here or Apache but as it used to work in 2.4.17 with same protocol settings.

I wonder if this change might caused the issue?:

*) core: if the first HTTP/1.1 request on a connection goes to a server that
prefers different protocols, these protocols are announced in a Upgrade:
header on the response, mentioning the preferred protocols.

I also tried reversing the Protocols config:

Protocols http/1.1 h2

This worked for Node but stopped http/2 working in Chrome and Opera (even if ProtocolsHonorOrder is set to off - though that seems to work for Firefox and Edge).

I've not seen any other issues with any other clients (browsers, scripts or online tools) except Node ones.

To repeat install Node, add an example script with code like below to call my site (www.tunetheweb.com):

var https = require('https');

var options = {
  hostname: 'www.tunetheweb.com',
  path: '/',
};

var req = https.request(options, function(res) {
  console.log("statusCode: ", res.statusCode);
  console.log("headers: ", res.headers);

  res.on('data', function(d) {
    process.stdout.write(d);
  });
});
req.end();

req.on('error', function(e) {
  console.log(e);
});

in a file called example.js and run it with the following command:

node example.js

When h2 is enabled in apache 2.4.18 nothing will be returned, but the request is logged in the apache log with a 200 status.
When h2 is enabled in apache 2.4.17 (or h2 is disabled in apache 2.4.18) the page is returned.

Alternatively sign up to a uptimerobot.com account (which uses node to test your website responds) which is where I first noticed this :-)

Let me know if there is anything more I can do to help debug or resolve.

Thanks,
Barry

[icing]

Thanks for the good description. Will have to install node and run your test script against my server. Seems a protocol handling incompatiblity..

As to the behavior of chrome/opera with changed protocols order, that is a known bug in the chrome ALPN handling that is fixed in development and will at some point be released.

[icing]

This is very strange. If I do not load mod_http2 into the server, but have a vhost with
Protocols xxx http/1.1, nodejs does not work either, but Protocols http/1.1 xxx works (really "xxx"). For all other clients, it does not make a difference...

[icing]

Ah, got it. The answer is sent to nodejs https, but it fails to parse it. This answer fails:

HTTP/1.1 200 OK
Date: Thu, 17 Dec 2015 15:17:46 GMT
Server: Apache/2.5.0-dev (Unix) OpenSSL/1.0.2e
Upgrade: xxx
Connection: Upgrade, Keep-Alive
Last-Modified: Thu, 17 Dec 2015 15:17:39 GMT
ETag: "7d5-5271985fa6ac0"
Accept-Ranges: bytes
Content-Length: 2005
Keep-Alive: timeout=5
Content-Type: text/html

while this answer succeeds:

HTTP/1.1 200 OK
Date: Thu, 17 Dec 2015 15:19:59 GMT
Server: Apache/2.5.0-dev (Unix) OpenSSL/1.0.2e
Last-Modified: Thu, 17 Dec 2015 15:19:56 GMT
ETag: "7d5-527198e24df00"
Accept-Ranges: bytes
Content-Length: 2005
Keep-Alive: timeout=5
Connection: Keep-Alive
Content-Type: text/html

Either it is the Upgrade response header or the Upgrade token in the Connection, but that is probably easy for the nodejs https maintainer to figure out.

[tunetheweb]

So we're saying needs fixed in Node?

Are we happy to leave in Apache until then given Node's popularity?

Should we make sending of this Upgrade message configurable until then?

I don't really understand the benefit of it to be honest since clients that want h2 were requesting it anyway in 2.4.17 but sure you know more about this than me!

[icing]

Needs fix in Node. Sending the Upgrade header in this fashion is definitely allowed by HTTP/1.1 and Connection headers can have more than one token since ages. Whatever the problem in receiving such headers (and aborting without error), it needs fixing.

Just tested, "http" in node has the same problem.

If I make a work around, it will not be available until February. And I am even not sure I'd want to. The Upgrade header on https: connection is mostly unnecessary. The one on http: connection is very useful to clients. So, even if I remove it on https: then the problem remains for the cleartext ones.

[tunetheweb]

Fair enough. Raised with the Node guys (nodejs/node#4334). Thanks for looking into this.

[tunetheweb]

Hi @icing, the Node team have come back stating the Upgrade header is a request to upgrade, rather than an advertisement from server that clients can upgrade if they wish:
nodejs/node#4334

And if the server sends it, the client MUST upgrade. Which is difficult given that Apache is sending this header to Node even without knowing if it supports HTTP/2 (which it doesn't), which is I guess why they are disconnecting.

Not a HTTP expert but having a (very) quick read of the HTTP/1.1 and HTTP/2 RFCs I would tend to agree with that point of view.

Thoughts? Happy to go back to them if you disagree and can point me to documentation regarding this.

[icing]

Look here: http://tools.ietf.org/html/rfc7230#section-6.7

A server MAY send an Upgrade header field in any other response to
   advertise that it implements support for upgrading to the listed
   protocols, in order of descending preference, when appropriate for a
   future request.

Upgrade never happens in the client. Never, ever. They can ask anyone on the IETF http working group.

[bagder]

I'm with @icing here. RFC 7230 clearly says an Upgrade from a server "advertises support" for the protocols. That doesn't require the client to do anything really.

[tunetheweb]

Agreed and Node team have come round too and are having a think how to correct.

It would have helped if I had read the correct HTTP/1.1 RFC - originally looked at the outdated 2616 which didn't have above paragraph :-)

[tunetheweb]

Hi @icing, seen a few more people with this problem.

Node have implemented the fix but it's not in general release yet and they have concerns over back porting the fix meaning it could be some time before it's generally available to most node users and servers running them.

I know we've closed this issue and we all agree it's a Node issue and Apache is doing nothing wrong here, but any chance of a workaround with a "H2EmitUpgradeHeader" option, defaulting to true but with the ability for those affected to explicitly set this to false?

Not sure how technically feasible this is but think it would help with all our desire to push http/2 as much as possible and remove any impediments to that, especially given Node's popularity.

Thanks for your consideration and continued work on mod_http2,
Barry

[icing]

Hi @bazzadp,

the functionality to announce available protocols in an Upgrade response header is part of httpd core, not a feature of mod_http2. Therefore, it is better to open an issue on Apache's bugzilla for this, if you want this changed.

Personally, I do not want a new directive for this defined and specifically HTTP protocol behaviour. The history of making switches for broken clients is not a successful one. But then, the team might decide differently.

[mkauf]

Therefore, it is better to open an issue on Apache's bugzilla for this, if you want this changed.

I have created an entry in Apache's Bugzilla: https://bz.apache.org/bugzilla/show_bug.cgi?id=59311

[tunetheweb]

I had already added this btw: https://bz.apache.org/bugzilla/show_bug.cgi?id=58971
Been there a couple of months but no one biting yet.

BTW node released the fix for this but only in version 4 and above as they don't want to risk backporting it to 0.10 or 0.12: nodejs/node#4337. So imagine it will still cause problems for a lot of people as those versions are still popular.

Uptimerobot.com have upgraded though (thanks guys!) and so my only issue with this (for now) is resolved.

[louishot]

Upgrade: h2" header not good. If I use nginx proxy_hide_header to repression this header may broken websocket service