Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only proxy whitelisted request headers to ES server upstream #6896

Merged
merged 19 commits into from
Apr 19, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
9b268cf
Filter out origin header before proxying request to ES server upstream
ycombinator Apr 13, 2016
1ed0a25
Disable automatic passthrough of request/response headers to/from ups…
ycombinator Apr 13, 2016
b102e26
Filter headers using whitelist, not blacklist
ycombinator Apr 13, 2016
b635b2c
Adding elasticsearch.requestHeaders option to kibana.yml and document…
ycombinator Apr 13, 2016
e56e98f
Do not send x-forwarded-* headers
ycombinator Apr 14, 2016
ad5d772
Reformatting
ycombinator Apr 14, 2016
5cca6f7
Normalizing original request headers as well, just in case
ycombinator Apr 14, 2016
5f34d68
Replacing use of _.rearg with more readable code (IMO)
ycombinator Apr 14, 2016
836c740
Adding 'authorization' to the default headers list
ycombinator Apr 14, 2016
81631ff
Extracting filterHeader into its own module for reuse
ycombinator Apr 15, 2016
f9f4b79
Allow for elasticsearch.requestHeaders to be set to null
ycombinator Apr 15, 2016
7f75efb
Make callWithRequest use whitelisted headers
ycombinator Apr 15, 2016
2149f84
Renaming property to make its intent more explicit
ycombinator Apr 15, 2016
eeedc54
Merge branch 'master' into gh-6484
ycombinator Apr 15, 2016
c5ce30f
Adding tests for empty and [] values for elasticsearch.requestHeaders…
ycombinator Apr 15, 2016
3045117
Updating documentation
ycombinator Apr 15, 2016
89dfbee
Use es6 export default syntax
ycombinator Apr 16, 2016
65ddf96
Only document one option for sending no headers
ycombinator Apr 16, 2016
028db3f
Adding back x-forwarded-* headers
ycombinator Apr 16, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions config/kibana.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,10 @@
# must be a positive integer.
# elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
# elasticsearch.requestHeadersWhitelist: [ authorization ]

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
# elasticsearch.shardTimeout: 0

Expand Down
4 changes: 3 additions & 1 deletion docs/kibana-yml.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,9 @@ to `false`.
wait for Elasticsearch to respond to pings.
`elasticsearch.requestTimeout:`:: *Default: 300000* Time in milliseconds to wait for responses from the back end or
Elasticsearch. This value must be a positive integer.
`elasticsearch.shardTimeout:`:: *Default: 0* Time in milliseconds for Elasticsearch to wait for responses from shards. Set
`elasticsearch.requestHeadersWhitelist:`:: *Default: `[ 'authorization' ]`* List of Kibana client-side headers to send to Elasticsearch.
To send *no* client-side headers, set this value to [] (an empty list).
`elasticsearch.shardTimeout:`:: *Default: 0* Time in milliseconds for Elasticsearch to wait for responses from shards. Set
to 0 to disable.
`elasticsearch.startupTimeout:`:: *Default: 5000* Time in milliseconds to wait for Elasticsearch at Kibana startup before
retrying.
Expand Down
4 changes: 4 additions & 0 deletions docs/settings.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,10 @@ deprecated[4.2, The names of several Kibana server properties changed in the 4.2
+
*default*: `500000`

`elasticsearch.requestHeadersWhitelist:` added[5.0]:: List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side headers, set this value to [] (an empty list).
+
*default*: `[ 'authorization' ]`

`elasticsearch.shardTimeout` added[4.2]:: How long Elasticsearch should wait for responses from shards. Set to 0 to disable.
+
*alias*: `shard_timeout` deprecated[4.2]
Expand Down
3 changes: 3 additions & 0 deletions src/plugins/elasticsearch/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ import healthCheck from './lib/health_check';
import exposeClient from './lib/expose_client';
import createProxy, { createPath } from './lib/create_proxy';

const DEFAULT_REQUEST_HEADERS = [ 'authorization' ];

module.exports = function ({ Plugin }) {
return new Plugin({
require: ['kibana'],
Expand All @@ -20,6 +22,7 @@ module.exports = function ({ Plugin }) {
password: string(),
shardTimeout: number().default(0),
requestTimeout: number().default(30000),
requestHeadersWhitelist: array().items().single().default(DEFAULT_REQUEST_HEADERS),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does .items().single() do here?

Copy link
Contributor Author

@ycombinator ycombinator Apr 15, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It allows the user to define a single item array as a bare string. So they can define the setting as:

elasticsearch.requestHeadersWhiteList: x-my-custom-header-1

instead of having to say:

elasticsearch.requestHeadersWhiteList: [ x-my-custom-header-1 ]

.single() will convert x-my-custom-header-1 to [ x-my-custom-header-1 ] because it knows the value of this setting must ultimately be an array.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, thanks.

pingTimeout: number().default(ref('requestTimeout')),
startupTimeout: number().default(5000),
ssl: object({
Expand Down
76 changes: 76 additions & 0 deletions src/plugins/elasticsearch/lib/__tests__/map_uri.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
import expect from 'expect.js';
import mapUri from '../map_uri';
import sinon from 'sinon';

describe('plugins/elasticsearch', function () {
describe('lib/map_uri', function () {

let request;

beforeEach(function () {
request = {
path: '/elasticsearch/some/path',
headers: {
cookie: 'some_cookie_string',
'accept-encoding': 'gzip, deflate',
origin: 'https://localhost:5601',
'content-type': 'application/json',
'x-my-custom-header': '42',
accept: 'application/json, text/plain, */*',
authorization: '2343d322eda344390fdw42'
}
};
});

it('only sends the whitelisted request headers', function () {

const get = sinon.stub()
.withArgs('elasticsearch.url').returns('http://foobar:9200')
.withArgs('elasticsearch.requestHeadersWhitelist').returns(['x-my-custom-HEADER', 'Authorization']);
const config = function () { return { get: get }; };
const server = {
config: config
};

mapUri(server)(request, function (err, upstreamUri, upstreamHeaders) {
expect(err).to.be(null);
expect(upstreamHeaders).to.have.property('authorization');
expect(upstreamHeaders).to.have.property('x-my-custom-header');
expect(Object.keys(upstreamHeaders).length).to.be(2);
});
});

it('sends no headers if whitelist is set to []', function () {

const get = sinon.stub()
.withArgs('elasticsearch.url').returns('http://foobar:9200')
.withArgs('elasticsearch.requestHeadersWhitelist').returns([]);
const config = function () { return { get: get }; };
const server = {
config: config
};

mapUri(server)(request, function (err, upstreamUri, upstreamHeaders) {
expect(err).to.be(null);
expect(Object.keys(upstreamHeaders).length).to.be(0);
});
});

it('sends no headers if whitelist is set to no value', function () {

const get = sinon.stub()
.withArgs('elasticsearch.url').returns('http://foobar:9200')
.withArgs('elasticsearch.requestHeadersWhitelist').returns([ null ]); // This is how Joi returns it
const config = function () { return { get: get }; };
const server = {
config: config
};

mapUri(server)(request, function (err, upstreamUri, upstreamHeaders) {
expect(err).to.be(null);
expect(Object.keys(upstreamHeaders).length).to.be(0);
});
});

});
});
8 changes: 4 additions & 4 deletions src/plugins/elasticsearch/lib/call_with_request.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@ import Promise from 'bluebird';
import Boom from 'boom';
import getBasicAuthRealm from './get_basic_auth_realm';
import toPath from 'lodash/internal/toPath';
import filterHeaders from './filter_headers';

module.exports = (client) => {
module.exports = (server, client) => {
return (req, endpoint, params = {}) => {
if (req.headers.authorization) {
_.set(params, 'headers.authorization', req.headers.authorization);
}
const filteredHeaders = filterHeaders(req.headers, server.config().get('elasticsearch.requestHeadersWhitelist'));
_.set(params, 'headers', filteredHeaders);
const path = toPath(endpoint);
const api = _.get(client, path);
let apiContext = _.get(client, path.slice(0, -1));
Expand Down
6 changes: 4 additions & 2 deletions src/plugins/elasticsearch/lib/create_proxy.js
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,12 @@ function createProxy(server, method, route, config) {
handler: {
proxy: {
mapUri: mapUri(server),
passThrough: true,
agent: createAgent(server),
xforward: true,
timeout: server.config().get('elasticsearch.requestTimeout')
timeout: server.config().get('elasticsearch.requestTimeout'),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we probably want to keep xforward enabled. In this case, this is a header specifically injected by Kibana itself rather than a header sent by the user's browser. This type of header only makes sense in the context of the direct proxy.

What do you think?

Copy link
Contributor Author

@ycombinator ycombinator Apr 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm torn on this one. It's common for proxies to add x-forwarded-* headers and what we have here is indeed a proxy, so for that would be a reason to include these. On the other hand, x-forwarded-for will include information about the client (specifically, their IP address) and comments such as #6221 (comment) makes me think we shouldn't include it.

I think, for now, we should err on the side of not including it. We can always add it in later if it is requested (either explicitly or via a config option) but it would be harder to take away later if we add it now and folks start relying on it somehow.

Copy link
Contributor

@epixa epixa Apr 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the sentiment, but in this case, xforward is already enabled on the proxy, so we're actually talking about removing that feature without any means for people to turn it back on.

It seems like a different issue than this PR is addressing, so why don't we keep xforward in this PR and address it in a separate issue/PR? It'd probably be something that we would at least want to make configurable since there is no workaround for it if we just outright remove it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressing this in a separate issue/PR sounds good to me.

onResponse: function (err, responseFromUpstream, request, reply) {
reply(err, responseFromUpstream);
}
}
},
};
Expand Down
4 changes: 2 additions & 2 deletions src/plugins/elasticsearch/lib/expose_client.js
Original file line number Diff line number Diff line change
Expand Up @@ -78,8 +78,8 @@ module.exports = function (server) {
server.expose('ElasticsearchClientLogging', ElasticsearchClientLogging);
server.expose('client', client);
server.expose('createClient', createClient);
server.expose('callWithRequestFactory', callWithRequest);
server.expose('callWithRequest', callWithRequest(noAuthClient));
server.expose('callWithRequestFactory', _.partial(callWithRequest, server));
Copy link
Contributor Author

@ycombinator ycombinator Apr 15, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lukasolson Hey, I know you originally introduced callWithRequestFactory here in 5be3400 so I was wondering if you could take a look at my refactor here. Let me know if you think there's a better way to do this.

BTW, I noticed that no code is actually using callWithRequestFactory. Is that expected?

Copy link
Member

@lukasolson lukasolson Apr 15, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This refactor looks good to me. It is used in the security plugin in xpack.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @lukasolson!

server.expose('callWithRequest', callWithRequest(server, noAuthClient));
server.expose('errors', elasticsearch.errors);

return client;
Expand Down
22 changes: 22 additions & 0 deletions src/plugins/elasticsearch/lib/filter_headers.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
import _ from 'lodash';

export default function (originalHeaders, headersToKeep) {

const normalizeHeader = function (header) {
if (!header) {
return '';
}
header = header.toString();
return header.trim().toLowerCase();
};

// Normalize list of headers we want to allow in upstream request
const headersToKeepNormalized = headersToKeep.map(normalizeHeader);

// Normalize original headers in request
const originalHeadersNormalized = _.mapKeys(originalHeaders, function (headerValue, headerName) {
return normalizeHeader(headerName);
});

return _.pick(originalHeaders, headersToKeepNormalized);
}
6 changes: 5 additions & 1 deletion src/plugins/elasticsearch/lib/map_uri.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
import querystring from 'querystring';
import { resolve } from 'url';
import filterHeaders from './filter_headers';

module.exports = function mapUri(server, prefix) {

const config = server.config();
return function (request, done) {
const path = request.path.replace('/elasticsearch', '');
Expand All @@ -11,6 +14,7 @@ module.exports = function mapUri(server, prefix) {
}
const query = querystring.stringify(request.query);
if (query) url += '?' + query;
done(null, url);
const filteredHeaders = filterHeaders(request.headers, server.config().get('elasticsearch.requestHeadersWhitelist'));
done(null, url, filteredHeaders);
};
};