Fix plugin installer proxy for HTTPS

[timroes]

This PR fixes #15581 and now correctly create a correct HttpsProxyAgent when trying to connect to HTTPS URLs to download plugins. If you try to load a HTTP URL using HttpProxyAgent you will get a socket hang up as described in the above issue.

Testing this is a bit tricky, since a HTTPS proxy request, only starts a CONNECT request against the proxy and relies on it to establish a HTTPS connection between the client and the upstream server. Thus we never get the calls we used in testing for the actual requests. Instead I intercepted the CONNECT request. But we have nothing reasonable to forward, without setting up a proper SSL server, which I consider a lot of overhead for this single test. We cannot use nock to simulate this SSL server, since it will intercept all requests (even before any proxy handling could be done) and destroy the test altogether.

Thus I created the HTTPS proxy in a way, that will cause the client request to fail, but still log, that the client tried to connect to the server, so we can check for this in the test. Since the request actually failed we need to except inside the reject function of the promise instead the resolve one.

/cc @marius-dr

[jbudz]

The request didn't fail for me, it appears to use the proxy over http with the proxy downloading over https. What I did:

1. mini hack to allow self signed certs

   diff --git a/src/cli_plugin/install/downloaders/http.js b/src/cli_plugin/install/downloaders/http.js
   index acd02e81f4..7507c5016c 100644
   --- a/src/cli_plugin/install/downloaders/http.js
   +++ b/src/cli_plugin/install/downloaders/http.js
   @@ -5,6 +5,7 @@ import { createWriteStream } from 'fs';
   import HttpProxyAgent from 'http-proxy-agent';
   import HttpsProxyAgent from 'https-proxy-agent';
   import { getProxyForUrl } from 'proxy-from-env';
   +const url = require('url')
   
   function getProxyAgent(sourceUrl, logger) {
     const proxy = getProxyForUrl(sourceUrl);
   @@ -15,10 +16,14 @@ function getProxyAgent(sourceUrl, logger) {
   
     logger.log(`Picked up proxy ${proxy} from environment variable.`);
   
   +  const proxyObj = Object.assign({
   +    rejectUnauthorized: false
   +  }, url.parse(proxy))
   +
     if (/^https/.test(sourceUrl)) {
   -    return new HttpsProxyAgent(proxy);
   +    return new HttpsProxyAgent(proxyObj);
     } else {
   -    return new HttpProxyAgent(proxy);
   +    return new HttpProxyAgent(proxyObj);
     }
   }
   

2. brew install squid

3. generate self signed certs

4. https_port 3129 cert=/usr/local/etc/tmp/squid.crt key=/usr/local/etc/tmp/squid.key >> /usr/local/etc/squid.conf

I have an http proxy on 3128, https on 3129:

>> https_proxy=127.0.0.1:3129 node scripts/kibana-plugin install https://artifacts.elastic.co/downloads
/kibana-plugins/x-pack/x-pack-6.1.0.zip
Attempting to transfer from https://artifacts.elastic.co/downloads/kibana-plugins/x-pack/x-pack-6.1.0.zip
Picked up proxy https://127.0.0.1:3129 from environment variable.
Transferring 269804219 bytes....................
Transfer complete

>> https_proxy=http://127.0.0.1:3128 node scripts/kibana-plugin install https://artifacts.elastic.co/downloads/kibana-plugins/x-pack/x-pack-6.1.0.zip
Found previous install attempt. Deleting...
Attempting to transfer from https://artifacts.elastic.co/downloads/kibana-plugins/x-pack/x-pack-6.1.0.zip
Picked up proxy http://127.0.0.1:3128 from environment variable.
Transferring 269804219 bytes....................
Transfer complete

>> http_proxy=127.0.0.1:3128 node scripts/kibana-plugin install https://artifacts.elastic.co/downloads/kibana-plugins/x-pack/x-pack-6.1.0.zip
Attempting to transfer from https://artifacts.elastic.co/downloads/kibana-plugins/x-pack/x-pack-6.1.0.zip
Transferring 269804219 bytes....................
Transfer complete

[jbudz]

LGTM. ^ can be configured at the proxy level.

[ebuildy]

This is why:

env https_proxy=http://X.X.X.X no_proxy=127.0.0.1,localhost  kibana-plugin install https://github.com/sirensolutions/sentinl/releases/download/tag-5.5/sentinl-v5.5.1.zip

Dont work?

[tylersmalley]

The test I had used in the original PR no longer works with this. Should we be testing if the proxy is https and not the sourceUrl?

nginx proxy:

worker_processes  1;
daemon off;

events {
  worker_connections  1024;
}

http {
  access_log  /dev/stdout;
  error_log /dev/stdout debug;

  server {
    listen 8200;

    location /downloads/kibana-plugins/x-pack/x-pack-7.0.0-alpha1.zip {
      alias /Users/tyler/Downloads/x-pack-7.0.0-alpha1.zip;
    }
  }
}

env HTTPS_PROXY=http://localhost:8200 node scripts/kibana-plugin.js install x-pack

[timroes]

@tylersmalley The source url needs to be HTTPS. Whether the proxy is or not, doesn't actually matter, important is the way that the proxy, does the HTTPS proxying. So using nginx would not work at all in this case, since it basically answers your request in your above scenario, and doesn't do the common CONNECT forwarding, in which case it just forwards the CONNECT to the real server, but doesn't do any actual HTTP interception on the traffic. I don't think you can simulate that with nginx. Maybe @marius-dr can share how he tested it!

[timroes]

As discussed with Tyler offline, I will merge this in now.

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request