Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fleet] use package storage when getting a package #85337

Merged
merged 10 commits into from
Jan 11, 2021
Merged

[Fleet] use package storage when getting a package #85337

merged 10 commits into from
Jan 11, 2021

Conversation

neptunian
Copy link
Contributor

@neptunian neptunian commented Dec 8, 2020 •
edited
Loading

This is the implementation of integrations using the package storage to get assets or install from.

  • adds function getEsPackage to get the package in storage. this function has to create the 'package info' as well.
  • modifies calls to get the package to look in es storage if it doesn't exist in cache

Manual test uploaded package

  • upload a package
  • make a change to the server so the package doesnt exist in the cache
  • GET /api/fleet/epm/packages/{pkg} should return package info

Manual test package installed from registry

  • install package from registry
  • make a change to the server so the package doesnt exist in the cache
  • GET /api/fleet/epm/packages/{pkg} should return package info

Manual test package installed from registry before the storage feature was available

  • install package from registry
  • delete all the items related to this package in elasticsearch storage so it doesn't exist there
  • make a change to the server so the package doesnt exist in the cache
  • GET /api/fleet/epm/packages/{pkg} should return package info by reaching out to the registry

@neptunian neptunian added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes v7.11.0 labels Dec 8, 2020
@neptunian neptunian self-assigned this Dec 8, 2020
jfsiii
jfsiii reviewed Dec 10, 2020
View reviewed changes
x-pack/plugins/fleet/server/services/epm/archive/storage.ts Outdated

// create the packageInfo
// TODO: this is mostly copied from validtion.ts, but we should save packageInfo somewhere
// so we don't need to do this again as this was already done either in registry or through upload
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still getting the full picture of what's going on, but wanted to remind we have get/setPackageInfo methods

kibana/x-pack/plugins/fleet/server/services/epm/archive/cache.ts

Line 49 in 139f0a3

export const setPackageInfo = ({

Same trade-offs as elsewhere re: memory cache

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For packages that are not yet in storage (packages installed before storage was available) or in cache we would need to still create the package info. Also in the case the package IS in storage but not in cache, we don't have the package info. One thing I can do is check first if its in packageInfoCache, if its not, create it from storage as is being done here and add it to packageInfoCache.

neptunian
neptunian commented Dec 11, 2020
View reviewed changes
x-pack/plugins/fleet/server/services/epm/archive/validation.ts
manifestPolicyTemplates.forEach((policyTemplate: any) => {
const { name, title: policyTemplateTitle, description, inputs, multiple } = policyTemplate;
if (!(name && policyTemplateTitle && description && inputs)) {
if (!(name && policyTemplateTitle && description)) {
Copy link
Contributor Author

@neptunian neptunian Dec 11, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@skh I removed input being required in a policy template because it was causing all packages to fail during install or uninstall when getting the endpoint package from storage. endpoint does not have any inputs in its policy_templates in its manifest.yml. Is the package correct or is the spec incorrect?

Screen Shot 2020-12-10 at 7 32 09 PM

inputs will now be an empty array if there are no inputs in the manifest.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you have it right. Let's follow spec as written and fix any bugs there

https://github.com/elastic/package-spec/blob/22bc835bcf0480c0973b2c6bbc86df1314e9c0ef/versions/1/manifest.spec.yml#L223

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the parseAndVerify* methods I followed our existing types to decide which fields are mandatory. Spec and actual working package should take precedence though.

@jfsiii jfsiii mentioned this pull request Dec 15, 2020
Merged
@neptunian neptunian added v7.12.0 and removed v7.11.0 labels Dec 16, 2020
@neptunian neptunian force-pushed the use-package-storage branch from 155b33d to 83b3480 Compare January 5, 2021 14:15
@neptunian neptunian marked this pull request as ready for review January 5, 2021 16:31
@neptunian neptunian requested a review from a team January 5, 2021 16:31
jfsiii
jfsiii reviewed Jan 5, 2021
View reviewed changes
Copy link
Contributor

@jfsiii jfsiii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to apply some of the changes that were in #85711 to this, especially 3f213e5, but want to ensure I don't break/change anything.

Can you add some tests or testing instructions around this to describe the PR behavior so we can be sure it still works as intended?

@botelastic botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Jan 5, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/ingest-management (Team:Ingest Management)

@neptunian
Copy link
Contributor Author

I added some manual steps for testing. We don't currently test getting package info from cache or registry, only a 200 response. I think there should be tests written in a follow up that test when a package is expected to come from registry, cache, or storage.

@jfsiii
Copy link
Contributor

jfsiii commented Jan 8, 2021

@neptunian thanks for listing those.

I'll need to confirm, but I don't believe this is correct:
"make a change to the server so the package doesnt exist in the cache"

I believe the server needs to be restarted (stop & start) vs merely "rebuilt" via a change to a watched file. I'll take a look and update the instructions if they need it.

Thanks again for the PR and additional notes

@neptunian
Copy link
Contributor Author

@elasticmachine merge upstream

@jfsiii jfsiii self-requested a review January 11, 2021 14:36
jfsiii
jfsiii approved these changes Jan 11, 2021
View reviewed changes
Copy link
Contributor

@jfsiii jfsiii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I'd like to follow up with some changes like those in #85711 to break up getEsPackage and remove the duplication of validation code but that can be done in a follow up.

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@neptunian neptunian merged commit 61987df into elastic:master Jan 11, 2021
@neptunian neptunian mentioned this pull request Jan 12, 2021
Merged
neptunian added a commit to neptunian/kibana that referenced this pull request Jan 12, 2021
@neptunian @kibanamachine
[Fleet] use package storage when getting a package (elastic#85337)
35af269 
* getPackageFromSource to use package storage

* fix type

* use bulkGet

* add data streams and policy templates to package info from storage

* fix merge conflict

* comment out policy_templates for now

* add policy_templates to package info, remove required inputs from parseAndVerifyPolicyTemplates

* add storage assets to cache

* tidy up

Co-authored-by: Kibana Machine <[email protected]>
neptunian added a commit that referenced this pull request Jan 12, 2021
@neptunian @kibanamachine
[Fleet] use package storage when getting a package (#85337) (#88014)
6f27956 
* getPackageFromSource to use package storage

* fix type

* use bulkGet

* add data streams and policy templates to package info from storage

* fix merge conflict

* comment out policy_templates for now

* add policy_templates to package info, remove required inputs from parseAndVerifyPolicyTemplates

* add storage assets to cache

* tidy up

Co-authored-by: Kibana Machine <[email protected]>

Co-authored-by: Kibana Machine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@skh skh skh left review comments

@jfsiii jfsiii jfsiii approved these changes

Assignees

@neptunian neptunian

Labels
release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.12.0 v8.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@neptunian @elasticmachine @jfsiii @kibanamachine @skh