[Maps][ML] Integration: part 1 POC extended #113857

alvarezmelissa87 · 2021-10-04T21:42:38Z

Summary

Extends and updates #90497
Incorporates latest maps changes (#112465, #112333) into initial draft PR (#111228)

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

This was checked for breaking API changes and was labeled appropriately

alvarezmelissa87 · 2021-10-04T21:46:29Z

x-pack/plugins/maps/common/index.ts

@@ -9,14 +9,19 @@ export {
  AGG_TYPE,
  COLOR_MAP_TYPE,
  ES_GEO_FIELD_TYPE,
+  FieldFormatter,
  FIELD_ORIGIN,


@thomasneirynck - looks like we need these exported here so ML can access them

alvarezmelissa87 · 2021-10-04T21:56:21Z

@thomasneirynck - ran into some issues with the changes from #112465

I'm now always getting this failure (and a blank screen when I try to navigate to maps):

Looks like the failure is coming from the lazy load here https://github.com/elastic/kibana/blob/master/x-pack/plugins/maps/public/api/register.ts#L18 when registerSource is run.
The error is caught here https://github.com/elastic/kibana/blob/master/x-pack/plugins/maps/public/lazy_load_bundle/index.ts#L103

Looks like it's occurring when everything is being imported from kibana_services.ts (https://github.com/elastic/kibana/blob/master/x-pack/plugins/maps/public/kibana_services.ts#L19) because of the assumption that coreStart will always be there.

The contents in the file dependent on coreStart can't be accessed in the plugin setup right now.

It would work if we moved these functions back to (or expose them from both) the start function but it does seem they should be in the setup. Happy to discuss other potential solutions but as of now it won't work without some changes on the maps side.

cc @spalger as we discussed some of this.

nreese · 2021-10-06T17:29:11Z

@alvarezmelissa87 Thanks for identifying the issues with registerSource in setup. I have opened #114150 to fix the problem.

kibanamachine · 2021-10-08T01:28:52Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: a9787f5434da7ebf34697778666be27aa07ffb64
Storybooks Preview

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`ml`	1683	1697	+14

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`maps`	201	207	+6

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`maps`	2.7MB	2.7MB	+4.0B
`ml`	3.6MB	3.6MB	-21.8KB
total			-21.8KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`maps`	29	27	-2

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`maps`	41.7KB	41.9KB	+261.0B
`ml`	34.3KB	64.7KB	+30.5KB
total			+30.7KB

Unknown metric groups

API count

id	before	after	diff
`maps`	202	208	+6

async chunk count

id	before	after	diff
`ml`	25	24	-1

References to deprecated APIs

id	before	after	diff
`ml`	704	710	+6

History

💔 Build #157781 failed ed7b2d7b04404746d62ef72d1e70f2ca42306492

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

thomasneirynck

This is really great. The main advantage of course is that it will allow ML-users to actually view their spatial-based ML jobs on a map. This will also help us internally. There is potential (e.g. "impossible travel"). Having the ability to easily map an ML-job will be invaluable to troubleshoot.

The register-source API should provide ordering

The ML-card now shows first. Which is cool :) . But inconsistent with the other soltion-cards, like the RUM or SIEM layers. imho Maps needs to introduce better ordering

^ ML card should be added with the other solution-cards here.

This is on the Maps-plugin end, not the ML-end.

The card should be disabled when the user does not have access to ML

The ML card is always enabled. It should not be when the functionality is not accessible . This check, should be at least the for checking enteprise-license. e.g.

https://github.com/elastic/kibana/blob/a9787f5434da7ebf34697778666be27aa07ffb64/x-pack/plugins/ml/public/maps/anomaly_layer_wizard.tsx#L36-L40

For an example, see

kibana/x-pack/plugins/maps/public/classes/sources/es_geo_line_source/layer_wizard.tsx

Lines 24 to 28 in a9ee83c

    
           disabledReason: REQUIRES_GOLD_LICENSE_MSG, 
        
           icon: TracksLayerIcon, 
        
           getIsDisabled: () => { 
        
             return !getIsGoldPlus(); 
        
           },

This could also include other checks. (e.g. the Observability-layers check whether APM is configured)

Because the card is enabled, there is now a bug after clicking when the user does not hve an enterprise-license.

Uncaught (in promise) Error: Forbidden
    at Fetch.fetchResponse (:5601/dam/9007199254740991/bundles/core/core.entry.js:13463)
    at async interceptResponse (:5601/dam/9007199254740991/bundles/core/core.entry.js:13819)
    at async :5601/dam/9007199254740991/bundles/core/core.entry.js:13360

and an empty wizard.

Are tooltips useful?

The tooltips only include the record-score. I am wondering if there are other fields would be valuable.

e.g.

Document id
Fields used in the config (e.g. influencers, part of detectors arguments (?))
partition-field values

The preconfigured color-ramps is not being picked up by the Maps-UX

^ this is a bug in Maps likely. Maps-team should provide guidance here on how to resolve.

Handle empty state in wizard

When there are no ML jobs available for selection in the wizard, maybe we should not show the combo-box, and instead show some exaplanation/hyperlink. E.g. a link to the ML anomaly explorer.

Enable time filtering

While I don't think full support for KQL is not necessary, it would be nice the ML-layer syncs with the time-picker.

Can we link out from the ML app to a pre-configured Maps app?

Anytime a job is "mappable", can we provide a hyperlink in the ML-app, something like "Explore in maps".

…ized (elastic#120031) * Drop requests for overview page data when the value for the es filters has not been initially set. * Update to not wait when no filters are defined. * Add tests and clean up some files. * Delete unneeded code. * Reduce check condition to single boolean value to avoid triggering effect multiple times. * Simplify implementation, fix race. Update tests and fix types. * Make `useOverviewFilterCheck` consider kuery search as well. Co-authored-by: Kibana Machine <[email protected]>

…1535) * chore(NA): splits types from code on @kbn/rule-data-utils * chore(NA): remove old style imports for this pkg * chore(NA): eslint fix Co-authored-by: Kibana Machine <[email protected]>

…tic#121411) * Fix custom url special characters * Use double quotes instead * Fix to using for automatic links Co-authored-by: Kibana Machine <[email protected]>

…4404) * Added Component Integration Test for Opening A closed Index. * Fixed open index test and changed the structure of the tests to consolidate boilerplate. * Fixed PR based on feedback. Co-authored-by: Kibana Machine <[email protected]>

…stic#121255) Co-authored-by: Ece Ozalp <[email protected]> Co-authored-by: Ying Mao <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

…te (elastic#121662)

…c#121687) Co-authored-by: Kibana Machine <[email protected]>

…lastic#121664)

…elastic#121669)

…2056)

…states (elastic#122238) * Display backbutton for empty states * Remove eui spacer when empty state

…tic#121353) * [ML] File data visualizer reduce chunk size for slow processors * adding constant Co-authored-by: Kibana Machine <[email protected]>

…ment of a task list when empty (elastic#122242) * fix: fixes the issue where the last element of a task list would not render properly * test: update test

* Moved pie_vis to the other. * Removed not used types. * Changed docs. * Fixed labels. * Fixed more translation labels. * Changed types of buildExpressionFunction. * Added limits and extraPublicDir. * Fixed i18n checks. * Fixed translations checks. * Added codeowners to expression_pie. Co-authored-by: Kibana Machine <[email protected]>

Co-authored-by: Kibana Machine <[email protected]>

…r to the policy flyout (elastic#121964) * Displays assignable artifact list and allow selection. Also displays empty state message * Adds mutation to update event filters with new tags * Shows confirm toast when action has ben done correctly * Fix ui/ux loading states and added unit test for flyout * Remove customqueryId * Use new object reference befor update it * Fixes unit test Co-authored-by: Kibana Machine <[email protected]>

Co-authored-by: Kibana Machine <[email protected]>

…stic#122189)

….js test (elastic#122276) * [maps] fix x-pack/test/functional/apps/maps/documents_source/top_hits·js test * eslint

* Implementing the alert count metrics * Fixing type errors * Fixing tests * Building new aggregation function in alert service * Fleshing out compute for alert details * Adding tests for handler * Refactoring the types and adding another test * Removing dead code * Removing unused snapshots * Refactoring to only call handlers once * Refactoring aggregations * some code cleanup * Addressing review feedback Co-authored-by: Kibana Machine <[email protected]>

…a87/kibana into ml-maps-integration-part-1

kibana-ci · 2022-01-12T06:34:42Z

💔 Build Failed

Failed CI Steps

Linting (with types)

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`ml`	1588	1604	+16

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`maps`	206	212	+6

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`maps`	2.6MB	2.6MB	+11.0B
`ml`	3.5MB	3.5MB	+18.4KB
total			+18.4KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`maps`	30	28	-2

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`maps`	38.5KB	38.7KB	+260.0B
`ml`	38.5KB	38.6KB	+122.0B
total			+382.0B

Unknown metric groups

API count

id	before	after	diff
`maps`	207	213	+6

async chunk count

id	before	after	diff
`ml`	25	27	+2

ESLint disabled line counts

id	before	after	diff
`ml`	94	97	+3

References to deprecated APIs

id	before	after	diff
`ml`	61	67	+6

Total ESLint disabled count

id	before	after	diff
`ml`	97	100	+3

History

💔 Build #14347 failed aee2671
💔 Build #13737 failed 38df4baf4d92fc4db79532271932b02e5ee83e2d
💔 Build #13548 failed 47e9e3457a3cac3475d6c8bb2f03846b0c0d72c0
💔 Build #13270 failed b1c80372b2bbfa6d91161cc0c1f1eeeee25dad9e
💔 Build #9796 failed 38a897e53aa323bb62ec303668a7a9de60976cee

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

alvarezmelissa87 · 2022-01-12T18:09:29Z

Closing in lieu of #122862

alvarezmelissa87 mentioned this pull request Oct 4, 2021

[Maps][ML] Integration: part one POC extended #111228

Closed

9 tasks

alvarezmelissa87 requested a review from thomasneirynck October 4, 2021 21:45

alvarezmelissa87 commented Oct 4, 2021

View reviewed changes

nreese mentioned this pull request Oct 6, 2021

[Maps] do not lazy load setup API methods #114150

Merged

alvarezmelissa87 force-pushed the ml-maps-integration-part-1 branch from ed7b2d7 to a9787f5 Compare October 7, 2021 23:22

thomasneirynck reviewed Oct 21, 2021

View reviewed changes

thomasneirynck self-requested a review October 21, 2021 16:00

alvarezmelissa87 force-pushed the ml-maps-integration-part-1 branch from a9787f5 to 6a68aa7 Compare November 1, 2021 18:52

alvarezmelissa87 force-pushed the ml-maps-integration-part-1 branch from 6a68aa7 to af56f4e Compare November 9, 2021 16:59

alvarezmelissa87 force-pushed the ml-maps-integration-part-1 branch from af56f4e to 38a897e Compare November 29, 2021 19:40

elastic deleted a comment from kibanamachine Nov 29, 2021

alvarezmelissa87 force-pushed the ml-maps-integration-part-1 branch 2 times, most recently from b1c8037 to 47e9e34 Compare December 15, 2021 22:11

alvarezmelissa87 added 3 commits December 20, 2021 12:57

wip: create ml anomalies layer in maps

c4f4e1a

incorporate fix

0df156d

update types for map layers

aee2671

alvarezmelissa87 force-pushed the ml-maps-integration-part-1 branch from 38df4ba to aee2671 Compare December 20, 2021 19:57

thomasneirynck mentioned this pull request Dec 20, 2021

[Maps][ML] Add ML data source to Maps POC #90497

Closed

mistic and others added 8 commits December 20, 2021 20:41

chore(NA): splits types from code on @kbn/rule-data-utils (elastic#12…

aa6fe69

…1535) * chore(NA): splits types from code on @kbn/rule-data-utils * chore(NA): remove old style imports for this pkg * chore(NA): eslint fix Co-authored-by: Kibana Machine <[email protected]>

[ML] Fix Anomaly detection's custom url with special characters (elas…

32dc575

…tic#121411) * Fix custom url special characters * Use double quotes instead * Fix to using for automatic links Co-authored-by: Kibana Machine <[email protected]>

[Security Solution] use abortableEsClient in security rule types (ela…

87c994d

…stic#121255) Co-authored-by: Ece Ozalp <[email protected]> Co-authored-by: Ying Mao <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

chore(NA): splits types from code on @kbn/securitysolution-autocomple…

2602459

…te (elastic#121662)

[Reporting] Ensure correct data view is selected for all test (elasti…

17f1d55

…c#121687) Co-authored-by: Kibana Machine <[email protected]>

chore(NA): splits types from code on @kbn/securitysolution-es-utils (e…

198684e

…lastic#121664)

chore(NA): splits types from code on @kbn/securitysolution-hook-utils (…

8678627

…elastic#121669)

lukeelmers and others added 25 commits January 4, 2022 06:43

Update 'Ask Kibana' URL for Cloud users. (elastic#122139)

1435a0c

Fix P12 parsing for certificates with "Basic constraints" (elastic#12…

75ea37b

…2056)

[Security Solution] [Endpoint] Display backbutton for artifact empty …

a08da2e

…states (elastic#122238) * Display backbutton for empty states * Remove eui spacer when empty state

[ML] File data visualizer reduce chunk size for slow processors (elas…

078f176

…tic#121353) * [ML] File data visualizer reduce chunk size for slow processors * adding constant Co-authored-by: Kibana Machine <[email protected]>

Adds test for asTransactionRate method (elastic#122072)

3de247b

[storybook] Enable Emotion in non-EUI components (elastic#122194)

d071e52

[Security Solution][Investigations] Fixes rendering issue of last ele…

96a4cd1

…ment of a task list when empty (elastic#122242) * fix: fixes the issue where the last element of a task list would not render properly * test: update test

Change text in assistant to switch to Elastic Agent (elastic#122240)

2f24b2d

Update nav-kibana-dev.docnav.json (elastic#121915)

d5a9067

Co-authored-by: Kibana Machine <[email protected]>

Adds tests for asInteger and asDecimal methods (elastic#122074)

ee21395

[Metrics UI] Stabilize group header functional test (elastic#122169)

535c058

Co-authored-by: Kibana Machine <[email protected]>

[Fleet] Fix policy count in output edit modal (elastic#122267)

c69aac8

[Logs UI] Remove unnecessary mountReactNode usage (elastic#122173)

8a1f595

Co-authored-by: Kibana Machine <[email protected]>

[Adaptive relevance] Improve handling of errors for bulk update. (ela…

8818278

…stic#122189)

[Maps] gzip _mvt requests

2cb2bd4

[maps] fix x-pack/test/functional/apps/maps/documents_source/top_hits…

6273dcb

….js test (elastic#122276) * [maps] fix x-pack/test/functional/apps/maps/documents_source/top_hits·js test * eslint

[Lens] Hide tooltips while dragging dimensions (elastic#122198)

c2f57dc

wip: create ml anomalies layer in maps

a22c39f

incorporate fix

4113597

update types for map layers

e45eacc

wip: ensure http service initialized for anomalies layer in maps

4ed798a

Merge branch 'ml-maps-integration-part-1' of github.com:alvarezmeliss…

31f7211

…a87/kibana into ml-maps-integration-part-1

alvarezmelissa87 mentioned this pull request Jan 12, 2022

[Maps][ML] Integration part 1: Create anomalies layer in maps #122862

Merged

24 tasks

alvarezmelissa87 closed this Jan 12, 2022

alvarezmelissa87 mentioned this pull request Jan 20, 2022

[Maps][ML] Integration meta issue #123492

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Maps][ML] Integration: part 1 POC extended #113857

[Maps][ML] Integration: part 1 POC extended #113857

alvarezmelissa87 commented Oct 4, 2021

alvarezmelissa87 Oct 4, 2021

alvarezmelissa87 commented Oct 4, 2021 •

edited

Loading

nreese commented Oct 6, 2021 •

edited

Loading

kibanamachine commented Oct 8, 2021

API count

async chunk count

References to deprecated APIs

thomasneirynck left a comment •

edited

Loading

kibana-ci commented Jan 12, 2022 •

edited

Loading

API count

async chunk count

ESLint disabled line counts

References to deprecated APIs

Total ESLint disabled count

alvarezmelissa87 commented Jan 12, 2022

	disabledReason: REQUIRES_GOLD_LICENSE_MSG,
	icon: TracksLayerIcon,
	getIsDisabled: () => {
	return !getIsGoldPlus();
	},

[Maps][ML] Integration: part 1 POC extended #113857

[Maps][ML] Integration: part 1 POC extended #113857

Conversation

alvarezmelissa87 commented Oct 4, 2021

Summary

Checklist

Risk Matrix

For maintainers

alvarezmelissa87 Oct 4, 2021

Choose a reason for hiding this comment

alvarezmelissa87 commented Oct 4, 2021 • edited Loading

nreese commented Oct 6, 2021 • edited Loading

kibanamachine commented Oct 8, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Public APIs missing exports

Page load bundle

API count

async chunk count

References to deprecated APIs

History

thomasneirynck left a comment • edited Loading

Choose a reason for hiding this comment

kibana-ci commented Jan 12, 2022 • edited Loading

💔 Build Failed

Failed CI Steps

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Public APIs missing exports

Page load bundle

API count

async chunk count

ESLint disabled line counts

References to deprecated APIs

Total ESLint disabled count

History

alvarezmelissa87 commented Jan 12, 2022

alvarezmelissa87 commented Oct 4, 2021 •

edited

Loading

nreese commented Oct 6, 2021 •

edited

Loading

thomasneirynck left a comment •

edited

Loading

kibana-ci commented Jan 12, 2022 •

edited

Loading