Update ES client to 8.1.0-canary.2 #119791

mshustov · 2021-11-28T10:08:34Z

Summary

Closes #119454
Closes #116111

Important changes:

_type meta field is removed Remove support for _type in searches elasticsearch#68564
aggregations type can be defined with TAggregations generic Aggregations should be a generic in responses elasticsearch-js#1596
product check is performed on 2xx responses only Only check for product headers on 2xx responses elastic-transport-js#32
ES client added asStream configuration option Add asStream support elastic-transport-js#34

Aggregations

In this PR, I declared definitions as

interface MyAgg extends AggregationsMultiBucketAggregateBase {
   buckets: MyType[];
}

const response = client.serch<unknown, MyAgg>(...);

I use extends to enforce buckets compatbility. ES client type definitions define buckets type as

buckets: Record<string, TBucket> | TBucket[];

so we have to specify explicitly the bucket type we expect - object or array-based.
Hopefully, in the future, we will be able to infer a response type from a request type. For now, we declare it explicitly.

stratoula

Viz editors changes LGTM!

ashokaditya

security_solution changes LGTM. I only have a few suggestions for removing redundant union types that use undefined.

ashokaditya · 2021-12-22T08:37:32Z

x-pack/plugins/security_solution/server/lib/detection_engine/signals/build_events_query.ts

-  sortOrder?: estypes.SearchSortOrder;
-  searchAfterSortIds: estypes.SearchSortResults | undefined;
+  sortOrder?: estypes.SortOrder;
+  searchAfterSortIds: estypes.SortResults | undefined;


Suggested change

searchAfterSortIds: estypes.SortResults | undefined;

searchAfterSortIds?: estypes.SortResults;

@ashokaditya Happy to accept those changes. However, IMO, they are not exactly the same:
AFAIK, searchAfterSortIds: estypes.SortResults | undefined; enforces the property to be present but it accepts undefined. Setting a property as optional means that the property might not be provided.

It's a small difference, but depending on where it's used, it can catch some errors.

If you're happy with making the property optional, I'll gladly accept your suggestion :)

EDIT: I noticed about the agreement with Pierre. I also agree that it's better to make these changes in a separate PR.

@afharo yes quite right. At least in these cases, the types look like they are optional. Although it does make sense to explicitly define this so that the param is explicitly assigned undefined when that method is invoked if that is what is intended 👍

x-pack/plugins/security_solution/server/lib/detection_engine/signals/single_search_after.ts

x-pack/plugins/security_solution/server/lib/detection_engine/signals/utils.ts

...ty_solution/server/search_strategy/security_solution/factory/cti/event_enrichment/helpers.ts

dgieselaar · 2021-12-22T09:53:26Z

x-pack/test/apm_api_integration/tests/traces/trace_by_id.spec.ts

@@ -59,6 +59,7 @@ export default function ApiTest({ getService }: FtrProviderContext) {
      `);
      expectSnapshot(
        response.body.traceDocs.map((doc) =>
+          // @ts-expect-error Property 'processor' does not exist on type 'Profile'.


this (and the other @ts-expect-error before) indicate that the request body is not valid. I'll have a look.

dgieselaar

Approving on behalf of APM UI, @walterra will also have a look at the changes in APM's correlations code (thank you Walter!)

afharo · 2021-12-22T13:04:37Z

Relaunching the tests because it looks like they fail due to timeouts in Buildkite (cc @elastic/kibana-operations)

afharo · 2021-12-22T13:05:06Z

@elasticmachine merge upstream

justinkambic

Uptime change LGTM!

x-pack/plugins/security_solution/common/endpoint/data_loaders/index_fleet_server.ts

…index_fleet_server.ts Co-authored-by: Steph Milovic <[email protected]>

stephmilovic

LGTM for threat hunting, thank you

…-upgrade

walterra

Transforms/ML/APM Correlations LGTM

walterra

Transforms/ML/APM Correlations LGTM

afharo · 2021-12-24T11:22:02Z

@elasticmachine merge upstream

afharo · 2021-12-24T11:24:44Z

I double-checked the changes for the 2 pending reviewers. The changes are minimal:

@elastic/kibana-reporting-services: Update some test snapshots because of the removal of _type
@elastic/security-telemetry: Add as const in some string declarations.

I'll merge after green CI.

kibana-ci · 2021-12-24T12:43:17Z

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`core`	997	1000	+3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`discover`	330.6KB	330.5KB	-53.0B
`stackAlerts`	166.6KB	166.6KB	+44.0B
total			-9.0B

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`kibana`	942	956	+14

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`data`	441.1KB	441.0KB	-8.0B

Unknown metric groups

API count

id	before	after	diff
`core`	2326	2329	+3

History

💚 Build #14961 succeeded a3b4059
💔 Build #14901 failed ae5ad9f
💚 Build #14841 succeeded cfdd8b3
💔 Build #14837 failed 1eb5bfd
💚 Build #14792 succeeded 06a3f5b
💔 Build #14760 failed d50c2e9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @afharo

tsullivan

LGTM

pjhampton

🌔 🚀 ✨ LGTM ✨ 🚀 🌔

Security telemetry changes look good!

restrry added 4 commits November 28, 2021 11:07

upgrade es client to canary 37

6c197fc

fix error in core

072392c

mute error in test/

8c0c445

mute incompatible errors

04a49aa

mshustov added chore Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting v8.1.0 labels Nov 28, 2021

unskip request_entity_too_large_exception test

808b5c9

mshustov mentioned this pull request Nov 28, 2021

Un-skip migration batch size test #116111

Closed

restrry added 19 commits November 28, 2021 11:21

commit autofix

0b13e50

unskip batch_size_bytes_exceeds_es_content_length test

e286b8d

Merge branch 'main' into issue-119454-es-upgrade

bfebd9e

fix errors in Core code

227f901

fix or mute errors in data plugin

96cde79

fix data_view_management

95528fd

fix error index_management

64588f3

fix x-pack/test errors

95ef2c9

fix watcher

de5c389

fix event_log

1ad0c55

fix data_enhanced

e7e8cae

fix uptime

5d1823b

fix triggers_actions_ui

c6825bd

fix transform

2d4a432

fix reporting

1b43084

fix rule_registry

ceb7ee8

fix timeline

282d8d5

fix task_manager

ec2dc58

fix security_solution

e7ebd9e

stratoula approved these changes Dec 22, 2021

View reviewed changes

ashokaditya approved these changes Dec 22, 2021

View reviewed changes

dgieselaar reviewed Dec 22, 2021

View reviewed changes

Fix type errors

d50c2e9

dgieselaar approved these changes Dec 22, 2021

View reviewed changes

Import Sort type

31cdf17

Dosant approved these changes Dec 22, 2021

View reviewed changes

Merge branch 'main' into issue-119454-es-upgrade

06a3f5b

justinkambic approved these changes Dec 22, 2021

View reviewed changes

stephmilovic reviewed Dec 22, 2021

View reviewed changes

x-pack/plugins/security_solution/common/endpoint/data_loaders/index_fleet_server.ts Outdated Show resolved Hide resolved

afharo and others added 2 commits December 22, 2021 17:28

Update x-pack/plugins/security_solution/common/endpoint/data_loaders/…

1eb5bfd

…index_fleet_server.ts Co-authored-by: Steph Milovic <[email protected]>

Remove double negation

cfdd8b3

stephmilovic approved these changes Dec 22, 2021

View reviewed changes

Merge branch 'main' of github.com:elastic/kibana into issue-119454-es…

ae5ad9f

…-upgrade

walterra approved these changes Dec 23, 2021

View reviewed changes

Fix snapshot conflict issues after merge from "main"

a3b4059

Merge branch 'main' into issue-119454-es-upgrade

53658ec

afharo disabled auto-merge December 24, 2021 13:43

afharo merged commit 38feafa into elastic:main Dec 24, 2021

mshustov deleted the issue-119454-es-upgrade branch December 25, 2021 15:34

tsullivan reviewed Dec 27, 2021

View reviewed changes

pjhampton reviewed Jan 4, 2022

View reviewed changes

mshustov changed the title ~~Update ES client to canary.37~~ Update ES client to 8.1.0-canary.2 Jan 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update ES client to 8.1.0-canary.2 #119791

Update ES client to 8.1.0-canary.2 #119791

mshustov commented Nov 28, 2021 •

edited

Loading

stratoula left a comment

ashokaditya left a comment

ashokaditya Dec 22, 2021

afharo Dec 22, 2021 •

edited

Loading

ashokaditya Dec 22, 2021

dgieselaar Dec 22, 2021

dgieselaar left a comment

afharo commented Dec 22, 2021 •

edited

Loading

afharo commented Dec 22, 2021

justinkambic left a comment

stephmilovic left a comment

walterra left a comment

walterra left a comment

afharo commented Dec 24, 2021

afharo commented Dec 24, 2021 •

edited

Loading

kibana-ci commented Dec 24, 2021

API count

tsullivan left a comment

pjhampton left a comment

	searchAfterSortIds: estypes.SortResults \| undefined;
	searchAfterSortIds?: estypes.SortResults;

Update ES client to 8.1.0-canary.2 #119791

Update ES client to 8.1.0-canary.2 #119791

Conversation

mshustov commented Nov 28, 2021 • edited Loading

Summary

Aggregations

stratoula left a comment

Choose a reason for hiding this comment

ashokaditya left a comment

Choose a reason for hiding this comment

ashokaditya Dec 22, 2021

Choose a reason for hiding this comment

afharo Dec 22, 2021 • edited Loading

Choose a reason for hiding this comment

ashokaditya Dec 22, 2021

Choose a reason for hiding this comment

dgieselaar Dec 22, 2021

Choose a reason for hiding this comment

dgieselaar left a comment

Choose a reason for hiding this comment

afharo commented Dec 22, 2021 • edited Loading

afharo commented Dec 22, 2021

justinkambic left a comment

Choose a reason for hiding this comment

stephmilovic left a comment

Choose a reason for hiding this comment

walterra left a comment

Choose a reason for hiding this comment

walterra left a comment

Choose a reason for hiding this comment

afharo commented Dec 24, 2021

afharo commented Dec 24, 2021 • edited Loading

kibana-ci commented Dec 24, 2021

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Async chunks

Public APIs missing exports

Page load bundle

API count

History

tsullivan left a comment

Choose a reason for hiding this comment

pjhampton left a comment

Choose a reason for hiding this comment

mshustov commented Nov 28, 2021 •

edited

Loading

afharo Dec 22, 2021 •

edited

Loading

afharo commented Dec 22, 2021 •

edited

Loading

afharo commented Dec 24, 2021 •

edited

Loading