Error: endpoint is installed by default and cannot be removed

[cavokz]

Kibana version:

• master
• 7.14

Elasticsearch version:

• master
• 7.14

Server OS version:

• Ubuntu 16.04

Browser version:

• Cypress/Chrome 91
• Firefox 89

Browser OS version:

• MacOS 11.4

Original install method (e.g. download page, yum, from source, etc.):

• snapshot tarball

Describe the bug:

• Found these errors in the logs:

{"type":"log","@timestamp":"2021-07-12T20:47:01+00:00","tags":["error","plugins","fleet"],"pid":4721,"message":"uninstalling endpoint-0.19.1 after error installing"}
{"type":"log","@timestamp":"2021-07-12T20:47:02+00:00","tags":["error","plugins","fleet"],"pid":4721,"message":"failed to uninstall or rollback package after installation error Error: endpoint is installed by default and cannot be removed"}
{"type":"log","@timestamp":"2021-07-12T20:47:03+00:00","tags":["error","plugins","fleet"],"pid":4721,"message":"ResponseError: invalid_index_template_exception: [invalid_index_template_exception] Reason: index_template [logs-endpoint.alerts] invalid, cause [index template [logs-endpoint.alerts] specifies component templates [.fleet_component_template-1] that do not exist]\n    at onBody (/home/vagrant/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:337:23)\n    at IncomingMessage.onEnd (/home/vagrant/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:264:11)\n    at IncomingMessage.emit (events.js:387:35)\n    at endReadableNT (internal/streams/readable.js:1317:12)\n    at processTicksAndRejections (internal/process/task_queues.js:82:21) {\n  meta: {\n    body: { error: [Object], status: 400 },\n    statusCode: 400,\n    headers: {\n      'x-opaque-id': '8beaebaf-5fe0-43ba-a82e-850a37d4fc9e',\n      'x-elastic-product': 'Elasticsearch',\n      'content-type': 'application/json; charset=UTF-8',\n      'content-length': '489'\n    },\n    meta: {\n      context: null,\n      request: [Object],\n      name: 'elasticsearch-js',\n      connection: [Object],\n      attempts: 0,\n      aborted: false\n    }\n  }\n}"}

Steps to reproduce:

1. Visit page app/security/alerts
2. Check Kibana logs

Expected behavior:

• No errors in the logs are found

Potential solution:

• Check to see if Fleet is setup.
• If Fleet is not setup, then don't try to install the package yet.
• If Fleet is not setup, then call setup from the Security app

Additional context:

• The issue seems related to [Fleet] Add global component template to all fleet index templates #102225
• Setting xpack.fleet.agentIdVerificationEnabled: false in kibana.yml makes these errors go away

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[EricDavisX]

I believe the spec is such that the Endpoint package cannot be uninstalled once it is installed, but it should not be installed by default. I think something is installing it (if you are using Endpoint, then that would explain it, of course).

@cavokz Dom, hi - can you expand on the setup for the problem here?

I am also unsure of what the following kibana setting does: agentIdVerificationEnabled

• so i'm hesitant to suggest it is ok to turn it off, what expectations does it modify in the system?

I believe Dom is out on vacation, but @MadameSheema Glo knows the Cypress tests and can maybe speak to any questions we have.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[jen-huang]

Hi @jonathan-buttner @kevinlog, can you help triaging this as this was trigged on visiting the Security app? I wonder if this has anything to do with the fact that the Security app attempts to upgrade the endpoint package on load?

[ferullo]

I unassigned myself because this looks like a purely server side issue. @kevinlog let me know if you disagree.

[kevinlog]

@cavokz

It's expected that the Endpoint package cannot be uninstalled after it's installed. The package will be installed automatically if the user visits the Security app or adds the Endpoint integration in Fleet. Since you visited the Alerts tab, I would expect that the package is installed.

The error you're seeing regarding the alerts index is also a known issue right now until you generate alerts with the Endpoint. @peluja1012 @spong could speak more to that. Let us know if we can close this as a duplicate to some other related issue.

In addition, beyond those logs, are there any other errors in the UI? Does anything in the app become unusable?

[elasticmachine]

Pinging @elastic/security-onboarding-and-lifecycle-mgt (Team:Onboarding and Lifecycle Mgt)

[paul-tavares]

I came across this issue today as well. In my case, I was initializing my local dev. environment (so fresh ES + kibana just started). My Browser was already displaying the Endpoint page from Security Solution (from earlier in the day). Soon after Kibana started, I refreshed that browser page (never visited Fleet section of the UI) and that's when I noticed the error come out on the Kibana log.

In discussing with @kevinlog and @pzl, perhaps the issue here is that the check that Security Solution page does, anytime the app is loaded in the browser, to ensure the latest Endpoint Package is installed, might have been triggered prior to the Fleet setup being done. Will need to look into that further.

[cavokz]

@EricDavisX
I ignore what the connection to option xpack.fleet.agentIdVerificationEnabled is and surely would not suggest to use it as a remedy. The cause-effect is anyway there, maybe @nchaulet could help to explain.

@kevinlog
I agree these are at least two different issues: package installation/upgrade and index template. Feel free to add links to any existing issue and close this as duplicate.

These errors show up as part of automated tests. The tests would pass if only a post test step would not grep for errors in the logs. Therefore as far as I can see the application works good enough.

[nchaulet]

Does the endpoint app call the fleet setup API? I think it's required to do that call before being able to upgrade a package

[jonathan-buttner]

Does the endpoint app call the fleet setup API? I think it's required to do that call before being able to upgrade a package

No, the endpoint app only calls the package upgrade api. We have this code https://github.com/elastic/kibana/blob/master/x-pack/plugins/security_solution/public/app/home/setup.tsx#L37 but I don't think it's referenced anywhere.

[kevinlog]

@paul-tavares can you check with @jonathan-buttner about how critical this is ahead of FF? We should address the errors, but I want to know if we see any critical setup issues.

[paul-tavares]

Thanks @jonathan-buttner for that.
I'm going to look into fixing this for 7.15. My guess is that we'll need to ensure fleet is setup first before attempting to install/upgrade the endpoint package, so that component you reference may come back to life 😄 .

[muskangulati-qasource]

Bug Conversion

• Test-Case updated:
  • https://elastic.testrail.io/index.php?/cases/view/34346

Thanks!