[APM] Authorization exception on Kibana startup

[liza-mae]

Kibana version: 7.4.0 BC4

Elasticsearch version: 7.4.0 BC4

Server OS version: Linux

Browser version: Chrome Latest

Original install method (e.g. download page, yum, from source, etc.):
staging

Description of the problem including expected versus actual behavior:
Kibana error message on startup, not sure what it means and if it is expected or not.

Steps to reproduce:

1. Install ES/Kibana default
2. Check Kibana logs, an error is thrown on apm agent configuration authorization exception

{"type":"error","@timestamp":"2019-09-13T02:10:25Z","tags":["warning","process"],"pid":2033,"level":"error","error":{"message":"Authorization Exception :: {"path":"/.apm-agent-configuration","query":{},"statusCode":403,"response":""}\n at respond (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:315:15)\n at checkRespForFailure (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:274:7)\n at HttpConnector. (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n at IncomingMessage.wrapper (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (_stream_readable.js:1103:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)","name":"UnhandledPromiseRejectionWarning","stack":"UnhandledPromiseRejectionWarning: Authorization Exception :: {"path":"/.apm-agent-configuration","query":{},"statusCode":403,"response":""}\n at respond (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:315:15)\n at checkRespForFailure (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:274:7)\n at HttpConnector. (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n at IncomingMessage.wrapper (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (_stream_readable.js:1103:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)\n at emitWarning (internal/process/promises.js:81:15)\n at emitPromiseRejectionWarnings (internal/process/promises.js:120:9)\n at process._tickCallback (internal/process/next_tick.js:69:34)"},"message":"Authorization Exception :: {"path":"/.apm-agent-configuration","query":{},"statusCode":403,"response":""}\n at respond (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:315:15)\n at checkRespForFailure (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:274:7)\n at HttpConnector. (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n at IncomingMessage.wrapper (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (_stream_readable.js:1103:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)"}
{"type":"error","@timestamp":"2019-09-13T02:10:25Z","tags":["warning","process"],"pid":2033,"level":"error","error":{"message":"Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)","name":"UnhandledPromiseRejectionWarning","stack":"Authorization Exception :: {"path":"/.apm-agent-configuration","query":{},"statusCode":403,"response":""}\n at respond (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:315:15)\n at checkRespForFailure (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:274:7)\n at HttpConnector. (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n at IncomingMessage.wrapper (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (_stream_readable.js:1103:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)"}

[elasticmachine]

Pinging @elastic/apm-ui

[sorenlouv]

Ouch, this sounds like something we really need fixed before the release.

From the stacktrace I can't see exactly where the error occurs:

Authorization Exception :: {"path":"/.apm-agent-configuration","query":{},"statusCode":403,"response":""}
    at respond (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:315:15)
    at checkRespForFailure (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:274:7)
    at HttpConnector. (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)
    at IncomingMessage.wrapper (/tmp/kibana-7.4.0-linux-x86_64/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)
    at IncomingMessage.emit (events.js:194:15)
    at endReadableNT (_stream_readable.js:1103:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)

But I'd assume it is this line:

kibana/x-pack/legacy/plugins/apm/server/lib/settings/agent_configuration/create_agent_config_index.ts

Line 20 in ec5d1a9

const indexExists = await callWithInternalUser('indices.exists', { index });

If that's the case I'm a bit surprised. Because that call uses callWithInternalUser which afaiu should have access to all indices (and be able to create indices).

@ogupte Is this something you are able to repo?

[sorenlouv]

@ogupte Could it be that you need to give the internal user access to the .apm-agent-configuration index?
https://www.elastic.co/guide/en/kibana/current/development-security-rbac.html

[sorenlouv]

@liza-mae I was only able to reproduce this problem if I created a new user that didn't have the superuser role, and adding this user's credentials as the "internal user" kibana.yml file. Eg.

elasticsearch.username: myuser
elasticsearch.password: mypassword

With the default user I didn't get any errors. Can you elaborate a bit on which user Kibana is running with, and which roles this user has?

[sorenlouv]

@ogupte Regardless why this happens we should probably wrap createApmAgentConfigurationIndex in a try/catch so as to not spam the kibana screen with errors in these scenarios.

[sorenlouv]

@ogupte To handle the errors a bit more gracefully I've created #45764

(This doesn't address why the issue is happening in the first place which we should dig into)

[sorenlouv]

Turns out that the default kibana user does not have access to the .apm-agent-configuration index. We can fix that in:
https://github.com/elastic/elasticsearch/blob/479ebd18ff365f592b76d158b51a31a3dfb165ed/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/ReservedRolesStore.java#L111-L129

Additionally we need to add access checks to the endpoints that return agent configuration data:

kibana/x-pack/legacy/plugins/apm/server/routes/create_api/index.ts

Lines 57 to 59 in 19262a8

	options: {
	tags: ['access:apm']
	},