[Fleet] Add ?full option to get package info endpoint to return all package fields

[hop-dev]

Summary

Closes #143198.

Add ?full=true parameter to the get packager info endpoint. This param returns the package info from the parsed archive, meaning paramaters such as data_streams.x.elasticsearch.source_mode (which is not supported by the EPR API) will now be returned from the kibana API.

This new param is only used by the create/edit package policy pages as they need the detailed data stream information.

I have made it so that we only ever cache the "full" package info, not the info from the package registry. In the future I want to make it clearer which package info is full and which is not #144640.

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[juliaElastic]

Do we have any concerns on the performance of getting full package info and remove caching of package registry info?
EDIT: I saw you discuss that in the follow-up issue.

[juliaElastic]

@elasticmachine merge upstream

[hop-dev]

@juliaElastic I've been doing some performance testing today and added a testing script. For getting full package info vs getting from the package registry here is the comparison over the 131 non-experimental packages. I think the results are acceptable considering after the first call the results are cached. I am looking to see if there are any quick wins to be had:

Average EPM time (GET /pkg/version: 0.37458015267175565s
Average Archive time (/pkg/version?full): 1.003114503816794s

Top 3 worst response times for the ?full endpoint are:

• ti_util 2.629s
• AWS 2.425s
• Oracle 1.983s

(Note: also looking into why the cached times are so variable, I was expecting them to be consistently quick)

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: e9a6136

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
fleet	861.3KB	861.4KB	+24.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	115.5KB	115.5KB	-2.0B

Unknown metric groups

ESLint disabled in files

id	before	after	diff
osquery	1	2	+1

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
fleet	58	65	+7
osquery	108	113	+5
securitySolution	440	446	+6
total			+20

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
fleet	66	73	+7
osquery	109	115	+6
securitySolution	517	523	+6
total			+21

History

• 💔 Build #85933 failed 786e24956e14dab599197c3299feace68b8701f1
• 💔 Build #85864 failed c15f0eb606811b2c416ff3d385c5b7a5bfa4cf33
• 💔 Build #85676 failed 776d37fa11a1df931a0d73dee04ade399157ab87
• 💚 Build #85557 succeeded 327f13bd488fb2df490e7afc9874a13088399faf
• 💔 Build #85321 failed 9e529f84930939c155082f973f0fcb8008e694f3

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @hop-dev

[juliaElastic]

LGTM, haven't tested locally.

[MiriamAparicio]

Hi @hop-dev, I think this change broke the add APM integration page in fleet.

Screen.Recording.2022-11-10.at.14.31.43.mov