[Fleet] Migrate ES client #92805

afgomez · 2021-02-25T13:14:31Z

Summary

Implements #74111

Migrate to the new Elasticsearch client.

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

skh · 2021-02-25T17:23:41Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts

      body: any;
    } = {
      method: 'PUT',
      path: `/${templateAPIPath}/${templateName}`,
-      ignore: [404],
      body: content,
    };
    // This uses the catch-all endpoint 'transport.request' because there is no
    // convenience endpoint using the new _index_template API yet.
    // The existing convenience endpoint `indices.putTemplate` only sends to _template,
    // which does not support v2 templates.
    // See src/core/server/elasticsearch/api_types.ts for available endpoints.


The new client should have a specific function to install v2 index templates:

https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/api-reference.html#_indices_putindextemplate

The fallback to transport.request shouldn't be necessary any more.

afgomez · 2021-03-02T14:27:02Z

I think there might be more calls using transport.request in installTemplate() in server/services/epm/elasticsearch/template/install.ts. Can you check that?

I pushed those changes. I also went through all request.transport calls in fleet and try to replace them with native methods where possible.

ph · 2021-03-02T21:07:07Z

@EricDavisX We have changed the way to access Kibana using the new ES client. I think any issue should be cached by the normal regression test. I don't think we need to add an explicitly test-plan label for this WDYT?

afgomez · 2021-03-03T14:04:29Z

@elasticmachine merge upstream

afgomez · 2021-03-03T15:08:12Z

I think any issue should be cached by the normal regression test. I don't think we need to add an explicitly test-plan

@ph This PR basically changes, well, everything! If the regression test covers already all possible edge cases we should be good to go

afgomez · 2021-03-03T16:50:49Z

@elasticmachine merge upstream

EricDavisX · 2021-03-03T16:51:44Z

@afgomez @ph hi - I respect it changes 'pretty much everything' but I'd like to dig in to *some amount of literals here, please. :) I think with some of the broad changes we have brewing we could make use of the 'deploy my pr' and stand up an easily available server, and let the manual test team do a pre-merge check.

doing it pre-merge is extremely valuable vs doing it post merge ad it clearly identifies who the author of the change is that adds a break (with the hope that the team is well aware of known issues already merged in, which we are not always, though we try). doing it pre-merge also makes the regression time go more smooth where we always are short of time, so this keeps the cycle more manageable. thanks for reaching out to us.

I suggest when we are ready to push this and test it, that I can offer to run the jenkins job to deploy it and post access creds to the test group in slack. we can't run a full regression pre merge, but we can do a general smoke test, and target tests that relate... so that means... anywhere where Fleet 'talks accesses ES' I guess? What does that mean for us, if we can identify anything more specific that 'everything' we can focus there - but perhaps just the general smoke test is all we want to do?

@dikshachauhan-qasource is our resource who can help test. I will start a deploy now, if the PR is in good shape enough for us to check it out?

afgomez · 2021-03-03T17:14:27Z

@EricDavisX I also think it makes sense to test before merging the PR. I'm struggling to get the build green but the code should be good to go.

Let me know if I can be of any help 🙇

skh · 2021-03-04T11:17:19Z

If I'm not mistaken, all places where we use the ES client are used during package installation. If we cover the installation of all types of elasticsearch assets during our tests (e2e and api integration tests) this change should be adequately tested.

Edited to add: during package installation and for api keys.

EricDavisX · 2021-03-04T21:25:37Z

If I'm not mistaken, all places where we use the ES client are used during package installation and for api keys.

This is very helpful @skh thanks much. We do not have adequate e2e-testing scenarios for this, so we'll plan to test

jen-huang · 2021-03-04T21:29:19Z

While working on #91715, I realized that the new index pattern service requires the new ES client, so getting this in would unblock me on that. @skh Would you be able to pull this down locally and test as part of your review?

I think the changes here are low-level enough that if there are any issues, most would be caught during normal development by engineers after this gets merged into main branch: if things break from this change, they would break in big and obvious ways! It would still be great for QA to run a separate test after merge as @EricDavisX suggests, but I don't want to hold up the merge of this PR on requiring that.

Thank you @afgomez for addressing this tech debt we've had for a while :)

EricDavisX · 2021-03-05T14:26:05Z

@afgomez @jen-huang - one of the goals with pre-check-in tests is to know who to pick and hunt for when a bug comes in unexpectedly. with this info, we know that it may be related to this change to the new client, so it is not a problem for me to proceed as long as it is reviewed and has been tested out. Also, as you note it is likely a more obvious break if at all, then that makes it easier. Cheers

skh

How I tested:

run locally from git checkout, open Kibana, navigate to Fleet
observe in dev tools that the calls to the setup API endpoints return successfully
navigate to "Agents" and click on "Create user and enable central management"
enroll an agent, verify that the agent enrolls and appears as healthy in the list

Installing the endpoint and system packages during the Fleet setup process verifies that installation of the following ES assets works:

Ingest Pipeline
Index Template
Transform

Enrolling an agent verifies that the generation of API keys works.

afgomez · 2021-03-08T14:16:00Z

Alright! I will merge this once the build gets green. Please ping me if you find any issues 🙇

kibanamachine · 2021-03-08T15:46:29Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💔 Build #111680 failed 9df6b37
💚 Build #110802 succeeded d5f4f02
💔 Build #110752 failed fabd93e
💔 Build #110724 failed f6f094a
💔 Build #110675 failed 0198e02

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Co-authored-by: Kibana Machine <[email protected]>

Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

Migrate deletePipeline to new ESclient

2d00471

afgomez added Feature:Fleet Fleet team's agent central management project Team:Fleet Team label for Observability Data Collection Fleet team v7.13.0 v8.0.0 labels Feb 25, 2021

afgomez changed the title ~~74111-migrate-es-client~~ [Fleet] Migrate ES client Feb 25, 2021

Alejandro Fernández Gómez added 18 commits February 25, 2021 14:28

Migrate installILMPolicy to new client

0b52acc

Remove dead function

b6af091

Migrate deleteIlms to new client

c3add82

Migrate deleteTemplate to new client

75be7c2

Migrate deleteTransports to new client

f8ca0f3

Remove old client from deleteAssets

5e553a0

Remove old client from removeInstallation

2c17bf8

Migrate updateCurrentWriteIndices to new client

2e02613

Remove old client from installTransform

1c1a9d5

Migrate installTemplates to new client

cf61b1f

Migrate installPipelines to new client

63a9634

Migrate installIlmForDataStream to new client

dbf08af

Remove old client from installPackage

82b5955

Remove old client from bulkInstallPackages

56b4ad1

Remove legacy client from ensurePackagesCompletedInstall

d842493

Remove legacy client from addPackageToAgentPolicy

95db78c

Migrate setupIngestManager to new client

a31a6d7

Fix types inside createSetupSideEffects

59536b5

skh reviewed Feb 25, 2021

View reviewed changes

Alejandro Fernández Gómez added 5 commits March 1, 2021 14:21

Fix failing jest tests

82270f7

Properly handle resource_already_exists_exception errors

a797237

Cleanup test file

8401da6

Remove dead function

3ecbde6

Migrate getListHandler to new client

14f1965

Use specific transform API methods

9b34fc7

Fix tests

fae1df1

Alejandro Fernández Gómez added 2 commits March 3, 2021 11:57

Fix installTemplat tests

b5f73c3

Merge branch 'master' into 74111-migrate-es-client

0198e02

kibanamachine and others added 2 commits March 3, 2021 09:04

Merge branch 'master' into 74111-migrate-es-client

f6f094a

Run node scripts/build_api_docs locally to try to fix the build

fabd93e

Merge branch 'master' into 74111-migrate-es-client

d5f4f02

Merge branch 'master' into 74111-migrate-es-client

9df6b37

skh approved these changes Mar 8, 2021

View reviewed changes

Merge branch 'master' into 74111-migrate-es-client

c6c3c0a

afgomez enabled auto-merge (squash) March 8, 2021 14:16

afgomez merged commit 2aa0267 into elastic:master Mar 8, 2021

afgomez deleted the 74111-migrate-es-client branch March 8, 2021 15:46

afgomez mentioned this pull request Mar 9, 2021

[7.x] [Fleet] Migrate ES client (#92805) #94051

Merged

afgomez pushed a commit to afgomez/kibana that referenced this pull request Mar 9, 2021

[Fleet] Migrate ES client (elastic#92805)

5f70718

Co-authored-by: Kibana Machine <[email protected]>

afgomez pushed a commit that referenced this pull request Mar 9, 2021

[Fleet] Migrate ES client (#92805) (#94051)

bc91dc2

Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

jen-huang mentioned this pull request Mar 9, 2021

[Fleet] Audit & migrate to new Elasticsearch client #74111

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Migrate ES client #92805

[Fleet] Migrate ES client #92805

afgomez commented Feb 25, 2021 •

edited

Loading

skh Feb 25, 2021

afgomez commented Mar 2, 2021

ph commented Mar 2, 2021

afgomez commented Mar 3, 2021

afgomez commented Mar 3, 2021

afgomez commented Mar 3, 2021

EricDavisX commented Mar 3, 2021

afgomez commented Mar 3, 2021

skh commented Mar 4, 2021 •

edited

Loading

EricDavisX commented Mar 4, 2021

jen-huang commented Mar 4, 2021 •

edited

Loading

EricDavisX commented Mar 5, 2021

skh left a comment •

edited

Loading

afgomez commented Mar 8, 2021

kibanamachine commented Mar 8, 2021

[Fleet] Migrate ES client #92805

[Fleet] Migrate ES client #92805

Conversation

afgomez commented Feb 25, 2021 • edited Loading

Summary

Checklist

For maintainers

skh Feb 25, 2021

Choose a reason for hiding this comment

afgomez commented Mar 2, 2021

ph commented Mar 2, 2021

afgomez commented Mar 3, 2021

afgomez commented Mar 3, 2021

afgomez commented Mar 3, 2021

EricDavisX commented Mar 3, 2021

afgomez commented Mar 3, 2021

skh commented Mar 4, 2021 • edited Loading

EricDavisX commented Mar 4, 2021

jen-huang commented Mar 4, 2021 • edited Loading

EricDavisX commented Mar 5, 2021

skh left a comment • edited Loading

Choose a reason for hiding this comment

afgomez commented Mar 8, 2021

kibanamachine commented Mar 8, 2021

💚 Build Succeeded

Metrics [docs]

History

afgomez commented Feb 25, 2021 •

edited

Loading

skh commented Mar 4, 2021 •

edited

Loading

jen-huang commented Mar 4, 2021 •

edited

Loading

skh left a comment •

edited

Loading