Proxy support for plugin installer #12753

timroes · 2017-07-11T08:33:19Z

This pull request supersedes #10946. It adds basic proxy support via the (kind of but not really standard) env variables http_proxy and basic support for exclusion of hosts via no_proxy (though no ports or wildcards are supported.

Release Note: Kibana now respects the http_proxy, https_proxy and no_proxy environment variables to download plugins via a proxy.

This will now also accept http_proxy/no_proxy (lower case) env variables. It also logs information about the used (or skipped) proxy to the console.

kimjoar

Some nitpicks, otherwise it would be great with updated docs + tests that cover getProxyAgent

kimjoar · 2017-07-11T09:37:34Z

src/cli_plugin/install/downloaders/http.js

+    const shouldProxy = (excludedHosts.indexOf(hostname) === -1);
+
+    if (shouldProxy) {
+      logger.log(`Use proxy to download plugin.`);


Use -> Using?

kimjoar · 2017-07-11T09:38:13Z

src/cli_plugin/install/downloaders/http.js

+function getProxyAgent(sourceUrl, logger) {
+  const httpProxy = process.env.http_proxy || process.env.HTTP_PROXY;
+  // we have a proxy detected, lets use it
+  if (httpProxy) {


Maybe change to early return instead?

if (!httpProxy) { return undefined; }

Do we have some styleguide about early return? I am personally not a fan of early return, if it's not the very first statement and just checks function parameters.

https://github.com/elastic/kibana/blob/master/style_guides/js_style_guide.md#returnthrow-early-from-functions

kimjoar · 2017-07-11T09:38:47Z

src/cli_plugin/install/downloaders/http.js

+    const excludedHosts = noProxy.split(',');
+
+    // proxy if the hostname is not in noProxy
+    const shouldProxy = (excludedHosts.indexOf(hostname) === -1);


excludedHosts.includes(hostname)?

Also: might need to handle whitespace in excludedHosts:

'foo, bar'.split(',') > ["foo", " bar"]

Done the first. For the second: no_proxy can have a rather complex syntax and lots of tools don't support it all. @tylersmalley and me discussed if we should make an easy implementation at the beginning or try to support it as good as possible and decided to just check for plain hostnames now. We don't support ports at the moment (e.g. no_proxy="artifcacts.elastic.co:443" wouldn't neither match. I would also count this trimming of spaces as "advanced" handling at the moment, since we need to process the all entries again. So we should discuss whether or not we would also enhance the no_proxy support in general then? Discuss plz :)

Works for me, just make it super-clear in the docs :)

Btw, saw https://github.com/Rob--W/proxy-from-env. If it makes our lives easier we could just fork it and bring it into the codebase and make the necessary changes. It also has a good test suite we can use as a starting point: https://github.com/Rob--W/proxy-from-env/blob/master/test.js

kimjoar · 2017-07-11T09:41:18Z

src/cli_plugin/install/downloaders/http.js


-function sendRequest({ sourceUrl, timeout }) {
+function getProxyAgent(sourceUrl, logger) {
+  const httpProxy = process.env.http_proxy || process.env.HTTP_PROXY;


nitpick: Maybe extract a getEnv:

const getEnv = env => process.env[env.toLowerCase()] || process.env[env.toUpperCase()]

kimjoar · 2017-07-11T09:47:52Z

src/cli_plugin/install/downloaders/http.js

+  if (httpProxy) {
+    logger.log(`Picked up proxy ${httpProxy} from http_proxy environment variable.`);
+    // get the hostname of the sourceUrl
+    const hostname = URL.parse(sourceUrl).hostname;


nitpick, maybe:

const { hostname } = URL.parse(sourceUrl);

Extract getEnv method and use destructuring.

kimjoar · 2017-07-11T15:57:20Z

Pulling this out from the comment, as it was "hidden" because of another fix:

Btw, saw https://github.com/Rob--W/proxy-from-env. If it makes our lives easier we could just fork it and bring it into the codebase and make the necessary changes. It also has a good test suite we can use as a starting point: https://github.com/Rob--W/proxy-from-env/blob/master/test.js

timroes · 2017-07-13T14:48:49Z

@kjbekkelund we have now again two options. We could either use - as discussed with @tylersmalley - the very limited implementation of no_proxy now, so we could still merge this before the feature freeze and open a new ticket, to provide a better no_proxy support. Or we implement the proper no_proxy support directly in this PR, but therefore it won't make it till feature freeze anymore, since I am today all day in x-school.

kimjoar · 2017-07-13T14:50:46Z

Unless it's very important for 6.0 I prefer doing it properly. 6.1 isn't that long after 6.0. If it's an important feature to get in, though, I'm open to getting it in before ff.

timroes · 2017-07-13T14:52:10Z

I would also like to have tests and docs for the limited implementation, so I would say, postpone it till 6.1, unless someone vetoes against it :-)

epixa · 2017-07-13T15:13:07Z

++ Focus on doing this right and it'll get released when it gets released

timroes · 2017-07-18T16:23:55Z

@kjbekkelund Why do you think we should fork it? I looked through the sources and it seems to do exactly what we need. Could we just pull it from npm? In that case we also wouldn't "need" to write test cases for that part, since the library is covered pretty well with tests.

kimjoar · 2017-07-18T20:41:18Z

@timroes Yeah, we don't need to fork if it does exactly what we want it to do. However, of it's not a perfect fit we can use it as a starting point instead.

epixa · 2017-07-18T20:49:00Z

We do want tests on the behaviors we're exposing though, even if they happen to be tested in the upstream project, otherwise a gap in coverage or a breaking change in the dependency means a regression in Kibana

kimjoar · 2017-07-18T21:26:10Z

++ every time we directly expose a third-party dep we need to cover our use of that dep with tests.

timroes · 2017-07-19T18:39:34Z

@kjbekkelund changed to use the library now. Would be glad about another review :-)

… plugin-proxy-support

timroes · 2017-07-25T09:16:06Z

Jenkins, please test this

kimjoar

Maybe add a test that has http_proxy specified, but does an https request, and the other way around. Those would both be expectNoProxyHit, right?

timroes · 2017-07-25T09:40:06Z

@kjbekkelund Good catch. will do so!

jbudz · 2017-07-25T14:36:33Z

Do you know how standardized these environment variables are? My only thought with this is someone using these for some other application and not being able to shut it off without unsetting.

I haven't done any research yet but my gut would be to prefix all environment variable usage with kibana. Anyone have thoughts on this? /cc @elastic/kibana-operations, @elastic/kibana-platform

timroes · 2017-07-25T14:45:40Z

From my experience they are pretty common (even though not standardized). Applications using them are e.g. curl (and everything using curl as a library), wget, most package managers, like pacman or yum.

So using Linux since quite some time (and already managing some servers) I was always frustrated when an application doesn't use this env vars - which are mainly Java and node. So if I manage a server where I have set http_proxy and https_proxy I don't see a reason why I would be confused/frustrated/angry if a program uses this, only the other way around, since I used them with the purpose to configure a proxy. If I tried to configure it only for a specific application I would have wrapped the setting of these vars directly in a startup script for a forked shell (and not set them globally). But maybe people with more server managing experience than me could tell about their experiences here :)

Update: If you look at the famous ArchLinux Wiki these environment variables make the first chapter of "Proxy Settings".

salimhb · 2017-07-25T15:06:38Z

I second @timroes's statement.

I'm setting up an ELK stack using deviantony/docker-elk and landed on this PR when I looked for kibana plugin install proxy support. I eventually found a workaround to install it offline, but it would have been much less work if kibana's plugin installer supported these "standard" http_proxy/https_proxy environment variables.

In comparison with another Elastic product, this would be way better than what elasticsearch-plugin install requires.

tylersmalley · 2017-07-25T15:42:44Z

http {
  access_log  /dev/stdout;
  error_log /dev/stdout debug;

  server {
    listen 8200;

    location /downloads/kibana-plugins/x-pack/x-pack-6.0.0-beta1.zip {
      alias /Users/tyler/Downloads/x-pack-6.0.0-beta1-SNAPSHOT.zip;
    }
  }
}

> env HTTPS_PROXY=http://localhost:8200 bin/kibana-plugin install x-pack
Found previous install attempt. Deleting...
Attempting to transfer from x-pack
Attempting to transfer from https://artifacts.elastic.co/downloads/kibana-plugins/x-pack/x-pack-6.0.0-beta1.zip
Picked up proxy http://localhost:8200 from environment variable.
Transferring 118711203 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
...

tylersmalley

LGTM

jbudz

Works for me. LGTM

Proxy support for plugin installer

Thomas Vendetta and others added 3 commits March 29, 2017 18:37

Fixes elastic#5211

de65b36

Cleanup conditional logic

a499033

Improve proxy handling in http downloader

4434ca0

This will now also accept http_proxy/no_proxy (lower case) env variables. It also logs information about the used (or skipped) proxy to the console.

timroes added Team:Operations Team label for Operations Team release_note:enhancement labels Jul 11, 2017

timroes requested a review from tylersmalley July 11, 2017 08:33

timroes mentioned this pull request Jul 11, 2017

Plugin installer proxy support #10946

Closed

kimjoar suggested changes Jul 11, 2017

View reviewed changes

kimjoar reviewed Jul 11, 2017

View reviewed changes

timroes added 4 commits July 11, 2017 14:53

Use includes instead of indexOf

2aecac8

Fix wording in downloader log

eb4291a

Refactor http downloader

6b7b090

Extract getEnv method and use destructuring.

Use early return in plugin downloader

3a67239

Tim Roes and others added 3 commits July 19, 2017 20:27

Use proxy-from-env for plugin proxy support

fe339a8

Add documentation on plugin proxy support

e08c35a

Merge branch 'master' into plugin-proxy-support

bc844f7

Tim Roes added 4 commits July 20, 2017 08:46

Resolve issue with dirty state after previous tests

4a422d8

Wait for server listen and close in before/after

04d7ffa

Merge branch 'plugin-proxy-support' of github.com:timroes/kibana into…

85b2f0e

… plugin-proxy-support

Merge remote-tracking branch 'upstream/master' into plugin-proxy-support

5ba96e1

kimjoar reviewed Jul 25, 2017

View reviewed changes

Add test for interchanged proxy usage

151cb79

kimjoar approved these changes Jul 25, 2017

View reviewed changes

timroes requested a review from jbudz July 25, 2017 11:14

tylersmalley approved these changes Jul 25, 2017

View reviewed changes

tylersmalley added v6.0.0 and removed v6.0.0 labels Jul 25, 2017

jbudz approved these changes Jul 25, 2017

View reviewed changes

timroes changed the title ~~Plugin installer proxy support~~ Proxy support for plugin installer Jul 26, 2017

timroes merged commit 008cf03 into elastic:master Jul 26, 2017

timroes deleted the plugin-proxy-support branch July 26, 2017 07:35

timroes added a commit to timroes/kibana that referenced this pull request Jul 26, 2017

Plugin installer proxy support (elastic#12753)

20f4829

Proxy support for plugin installer

timroes mentioned this pull request Jul 26, 2017

[6.x] Plugin installer proxy support #13115

Merged

timroes added a commit that referenced this pull request Jul 26, 2017

Plugin installer proxy support (#12753) (#13115)

26dff17

Proxy support for plugin installer

timroes added v7.0.0 v6.1.0 and removed v7.0.0 labels Jul 26, 2017

timroes mentioned this pull request Nov 22, 2017

Allow http/https proxy for plugin installation #5902

Closed

Ochir mentioned this pull request Dec 25, 2017

HTTP_PROXY support for connections towards Elasticsearch #5211

Closed

tylersmalley mentioned this pull request Jan 18, 2018

Fix plugin installer proxy for HTTPS #15588

Merged

timroes mentioned this pull request Mar 7, 2018

Fix broken yarn.lock #17001

Merged

tylerjl mentioned this pull request Jul 9, 2018

Please provide proxy support for plugin helper voxpupuli/puppet-kibana#29

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proxy support for plugin installer #12753

Proxy support for plugin installer #12753

timroes commented Jul 11, 2017 •

edited

Loading

kimjoar left a comment

kimjoar Jul 11, 2017

kimjoar Jul 11, 2017

timroes Jul 11, 2017

kimjoar Jul 11, 2017

kimjoar Jul 11, 2017

kimjoar Jul 11, 2017

timroes Jul 11, 2017

kimjoar Jul 11, 2017

kimjoar Jul 11, 2017

kimjoar Jul 11, 2017 •

edited

Loading

kimjoar Jul 11, 2017

kimjoar commented Jul 11, 2017

timroes commented Jul 13, 2017

kimjoar commented Jul 13, 2017

timroes commented Jul 13, 2017

epixa commented Jul 13, 2017

timroes commented Jul 18, 2017

kimjoar commented Jul 18, 2017

epixa commented Jul 18, 2017

kimjoar commented Jul 18, 2017

timroes commented Jul 19, 2017

timroes commented Jul 25, 2017

kimjoar left a comment

timroes commented Jul 25, 2017

jbudz commented Jul 25, 2017

timroes commented Jul 25, 2017 •

edited

Loading

salimhb commented Jul 25, 2017

tylersmalley commented Jul 25, 2017

tylersmalley left a comment

jbudz left a comment

Proxy support for plugin installer #12753

Proxy support for plugin installer #12753

Conversation

timroes commented Jul 11, 2017 • edited Loading

kimjoar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kimjoar Jul 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kimjoar commented Jul 11, 2017

timroes commented Jul 13, 2017

kimjoar commented Jul 13, 2017

timroes commented Jul 13, 2017

epixa commented Jul 13, 2017

timroes commented Jul 18, 2017

kimjoar commented Jul 18, 2017

epixa commented Jul 18, 2017

kimjoar commented Jul 18, 2017

timroes commented Jul 19, 2017

timroes commented Jul 25, 2017

kimjoar left a comment

Choose a reason for hiding this comment

timroes commented Jul 25, 2017

jbudz commented Jul 25, 2017

timroes commented Jul 25, 2017 • edited Loading

salimhb commented Jul 25, 2017

tylersmalley commented Jul 25, 2017

tylersmalley left a comment

Choose a reason for hiding this comment

jbudz left a comment

Choose a reason for hiding this comment

timroes commented Jul 11, 2017 •

edited

Loading

kimjoar Jul 11, 2017 •

edited

Loading

timroes commented Jul 25, 2017 •

edited

Loading