[Ingest Manager] Remove fields from index pattern during package uninstall #80082

jonathan-buttner · 2020-10-08T21:06:59Z

Summary

This PR addresses this issue: #80031

The installIndexPatterns() function relies on the saved objects being absent for removing the fields from a package during uninstall. Since the saved object is still present when the function is called it will collection the same fields as before instead of removing them fields for the package being uninstalled.

Testing

Start a fresh kibana and elasticsearch
Navigate to the ingest manager page to trigger /setup
Install the crowdstrike 0.1.1 package

curl --location --request POST 'http://elastic:changeme@localhost:5601/api/fleet/epm/packages/crowdstrike-0.1.1' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--header 'kbn-xsrf: xxx' \
--data-raw ''

Navigate back to the index pattern management page and observe that the crowdstrike fields exist
Uninstall the crowdstrike package

curl --location --request DELETE 'http://elastic:changeme@localhost:5601/api/fleet/epm/packages/-0.1.1' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--header 'kbn-xsrf: xxx' \
--data-raw ''

Navigate back to the index pattern management page and observe that the crowdstrike fields are removed.

elasticmachine · 2020-10-08T21:07:02Z

Pinging @elastic/ingest-management (Team:Ingest Management)

elasticmachine · 2020-10-08T21:07:02Z

Pinging @elastic/ingest-management (Feature:EPM)

jonathan-buttner · 2020-10-08T21:22:11Z

@ph should we backport this so it's fixed in 7.10?

jfsiii · 2020-10-08T21:38:13Z

x-pack/test/ingest_manager_api_integration/apis/epm/install_remove_assets.ts

@@ -200,6 +203,7 @@ export default function (providerContext: FtrProviderContext) {
    describe('uninstalls all assets when uninstalling a package', async () => {
      skipIfNoDockerRegistry(providerContext);
      before(async () => {
+        await installPackage(pkgKey);


This looks like it adds then immediately removes the package. Is that what's happening?

yeah this is just ensuring that the package was installed to begin with so the fields get added and then immediately removes it to setup the tests to validate the uninstall scenario. I got in a weird state locally when running the tests where the package was never actually installed the first time.

jfsiii · 2020-10-08T21:43:10Z

x-pack/test/ingest_manager_api_integration/apis/epm/install_remove_assets.ts

@@ -324,6 +328,35 @@ export default function (providerContext: FtrProviderContext) {
        }
        expect(resSearch.response.data.statusCode).equal(404);
      });
+      it('should have removed the fields from the index patterns', async () => {


I'm confused about which branch we expect to execute. The try or the catch?

Both could happen. If a test case in another file calls /setup then the system and endpoint packages will be installed and will be present for the remainder of the tests (because they cannot be removed). If that is the case the expect in the try will work because the logs-* and metrics-* index patterns will still be present even after this test uninstalls its package.

If /setup was never called prior to this test, when the package is uninstalled the index pattern code checks to see if there are no packages installed and completely removes the logs-* and metrics-* index patterns. If that happens this code will throw an error and indicate that the index pattern being searched for was completely removed.

Oh, interesting. Maybe we could add that as a comment?

Yeah sure I'll put it as a comment.

skh

Good catch, 👍

…tern-deletion

kibanamachine · 2020-10-12T16:58:54Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: e974c48

Metrics [docs]

✅ unchanged

History

💚 Build #81064 succeeded 0819cbb
💚 Build #80896 succeeded fd3fd01

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…stall (elastic#80082) * Correctly removing index pattern fields * Adding comment about try/catch

…stall (#80082) (#80195) * Correctly removing index pattern fields * Adding comment about try/catch

…stall (#80082) (#80194) * Correctly removing index pattern fields * Adding comment about try/catch

…a-tier-migration * 'master' of github.com:elastic/kibana: (34 commits) Line Visualization improper scaling can result in gaps elastic#79663 (elastic#80135) [Security Solution][Timeline] Fix SelectableTimeline search (elastic#80128) move field_mapping util to saved_objects plugin (elastic#79918) [ML] Datagrid: Ensure column content with 'boolean' schema is not capitalized (elastic#80041) [CI] Correctly resolve repository root for JUnit reports (elastic#80226) [Ingest Manager] Fix package upgrade breaking after first rollover before new data has arrived (elastic#79887) [Security Solution] Fix positioning of Kibana banners list inside Sec… (elastic#80124) add missing await to fix test (elastic#80202) Revert test data changed in previous commit. (elastic#79479) [Security Solution] [Sourcerer] Jest beef up (elastic#79907) Re-enable transaction duration alert story (elastic#80187) [npm] remove canvas dep (elastic#80185) [DOCS] Redirects ILM docs to Elasticsearch reference (elastic#79798) [APM] Catch health status error from ML (elastic#80131) Fix layout and remove title for add alert popover. (elastic#77633) [Discover] Loading spinner cleanup (elastic#79819) [Security Solution] [Resolver] Remove related events api (elastic#79036) [Ingest Manager] Remove fields from index pattern during package uninstall (elastic#80082) do not refetch license if signature header absents from a response (elastic#79645) Only send agent data for non-opentelemetry agents (elastic#79587) ... # Conflicts: # x-pack/plugins/index_lifecycle_management/server/routes/api/nodes/register_list_route.ts

* upstream/master: (34 commits) Improve vis editor typings (elastic#80004) Line Visualization improper scaling can result in gaps elastic#79663 (elastic#80135) [Security Solution][Timeline] Fix SelectableTimeline search (elastic#80128) move field_mapping util to saved_objects plugin (elastic#79918) [ML] Datagrid: Ensure column content with 'boolean' schema is not capitalized (elastic#80041) [CI] Correctly resolve repository root for JUnit reports (elastic#80226) [Ingest Manager] Fix package upgrade breaking after first rollover before new data has arrived (elastic#79887) [Security Solution] Fix positioning of Kibana banners list inside Sec… (elastic#80124) add missing await to fix test (elastic#80202) Revert test data changed in previous commit. (elastic#79479) [Security Solution] [Sourcerer] Jest beef up (elastic#79907) Re-enable transaction duration alert story (elastic#80187) [npm] remove canvas dep (elastic#80185) [DOCS] Redirects ILM docs to Elasticsearch reference (elastic#79798) [APM] Catch health status error from ML (elastic#80131) Fix layout and remove title for add alert popover. (elastic#77633) [Discover] Loading spinner cleanup (elastic#79819) [Security Solution] [Resolver] Remove related events api (elastic#79036) [Ingest Manager] Remove fields from index pattern during package uninstall (elastic#80082) do not refetch license if signature header absents from a response (elastic#79645) ...

Correctly removing index pattern fields

fd3fd01

jonathan-buttner added release_note:fix v8.0.0 Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project Team:Fleet Team label for Observability Data Collection Fleet team v7.11.0 labels Oct 8, 2020

jonathan-buttner requested review from jfsiii, skh, nchaulet, neptunian and a team October 8, 2020 21:06

neptunian approved these changes Oct 8, 2020

View reviewed changes

jfsiii reviewed Oct 8, 2020

View reviewed changes

skh approved these changes Oct 12, 2020

View reviewed changes

jonathan-buttner added 2 commits October 12, 2020 09:07

Merge branch 'master' of github.com:elastic/kibana into fix-index-pat…

7dfce8f

…tern-deletion

Adding comment about try/catch

0819cbb

jonathan-buttner added the v7.10.0 label Oct 12, 2020

Merge branch 'master' of github.com:elastic/kibana into fix-index-pat…

e974c48

…tern-deletion

jonathan-buttner merged commit bc6ba87 into elastic:master Oct 12, 2020

jonathan-buttner deleted the fix-index-pattern-deletion branch October 12, 2020 17:19

jonathan-buttner mentioned this pull request Oct 12, 2020

[7.x] [Ingest Manager] Remove fields from index pattern during package uninstall (#80082) #80194

Merged

jonathan-buttner added a commit to jonathan-buttner/kibana that referenced this pull request Oct 12, 2020

[Ingest Manager] Remove fields from index pattern during package unin…

60c78f3

…stall (elastic#80082) * Correctly removing index pattern fields * Adding comment about try/catch

jonathan-buttner mentioned this pull request Oct 12, 2020

[7.10] [Ingest Manager] Remove fields from index pattern during package uninstall (#80082) #80195

Merged

jonathan-buttner added a commit to jonathan-buttner/kibana that referenced this pull request Oct 12, 2020

[Ingest Manager] Remove fields from index pattern during package unin…

e432077

…stall (elastic#80082) * Correctly removing index pattern fields * Adding comment about try/catch

jonathan-buttner added a commit that referenced this pull request Oct 12, 2020

[Ingest Manager] Remove fields from index pattern during package unin…

ca1b319

…stall (#80082) (#80195) * Correctly removing index pattern fields * Adding comment about try/catch

jonathan-buttner added a commit that referenced this pull request Oct 12, 2020

[Ingest Manager] Remove fields from index pattern during package unin…

2d95b00

…stall (#80082) (#80194) * Correctly removing index pattern fields * Adding comment about try/catch

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest Manager] Remove fields from index pattern during package uninstall #80082

[Ingest Manager] Remove fields from index pattern during package uninstall #80082

jonathan-buttner commented Oct 8, 2020 •

edited

Loading

elasticmachine commented Oct 8, 2020

elasticmachine commented Oct 8, 2020

jonathan-buttner commented Oct 8, 2020

jfsiii Oct 8, 2020

jonathan-buttner Oct 8, 2020 •

edited

Loading

jfsiii Oct 8, 2020

jonathan-buttner Oct 8, 2020

jfsiii Oct 8, 2020

jonathan-buttner Oct 12, 2020

skh left a comment

kibanamachine commented Oct 12, 2020

[Ingest Manager] Remove fields from index pattern during package uninstall #80082

[Ingest Manager] Remove fields from index pattern during package uninstall #80082

Conversation

jonathan-buttner commented Oct 8, 2020 • edited Loading

Summary

Testing

elasticmachine commented Oct 8, 2020

elasticmachine commented Oct 8, 2020

jonathan-buttner commented Oct 8, 2020

jfsiii Oct 8, 2020

Choose a reason for hiding this comment

jonathan-buttner Oct 8, 2020 • edited Loading

Choose a reason for hiding this comment

jfsiii Oct 8, 2020

Choose a reason for hiding this comment

jonathan-buttner Oct 8, 2020

Choose a reason for hiding this comment

jfsiii Oct 8, 2020

Choose a reason for hiding this comment

jonathan-buttner Oct 12, 2020

Choose a reason for hiding this comment

skh left a comment

Choose a reason for hiding this comment

kibanamachine commented Oct 12, 2020

💚 Build Succeeded

Metrics [docs]

History

jonathan-buttner commented Oct 8, 2020 •

edited

Loading

jonathan-buttner Oct 8, 2020 •

edited

Loading