[ML] Data Frame Analytics creation wizard: add validation step (Part 1) #93478

alvarezmelissa87 · 2021-03-03T18:14:52Z

Summary

Related meta issue #88248
Also addresses: #78637

Adds a validation step to the Data Frame Analytics creation wizard.
Adds functional tests for validation step.
Adds api integration test for validate endpoint.

Dependent variable checks
- is constant
- has more than 30% empty values
Training Percent
- is too low - not enough training docs for model
- is too high - time/resource heavy due to large number of training docs
Analysis fields
- fields included in analysis have more than 30% empty values (TODO: list actual fields up to a limit?)
- too many fields included - time/resource may be increased

Current limitations used to validate

export const TRAINING_DOCS_UPPER = 200000;
export const TRAINING_DOCS_LOWER = 200;
export const INCLUDED_FIELDS_THRESHOLD = 100;
// limits number of fields for MISSING agg in request
export const MINIMUM_NUM_FIELD_FOR_CHECK = 25;
export const FRACTION_EMPTY_LIMIT = 0.3;

Regression/Classification example with warning messages:

Regression/Classification example success:

Outlier detection with warning messages:

Outlier detection success:

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)

elasticmachine · 2021-03-03T18:14:54Z

Pinging @elastic/ml-ui (:ml)

x-pack/test/functional/apps/ml/data_frame_analytics/outlier_detection_creation.ts

peteharverson · 2021-03-04T14:44:14Z

...me_analytics/pages/analytics_creation/components/validation_step/validation_step_details.tsx

+  ];
+  return (
+    <>
+      <EuiFlexGroup style={{ width: '70%' }}>


Is it possible to rerun the validation before returning to the validation step? For example, I set the training percent too low initially, then went back to the Configuration section and increased it, but the warning didn't disappear until I went back into the Validation step.

Adding this to be done in a follow up 👍

x-pack/plugins/ml/common/constants/validation.ts

peteharverson

LGTM. Once this and the follow-ups in #88248 are in, we can continue to test with larger data sets. Currently I don't have a clear idea on whether the thresholds used in the tests need refining, but they worked fine on my local setup.

alvarezmelissa87 · 2021-03-04T20:35:51Z

This part 1 PR has been updated and is ready for a final look when you get a chance cc @jgowdyelastic 🙏

lcawl

Some strings were complete sentences and others were not, so I synched on the former. Otherwise, text LGTM

x-pack/plugins/ml/server/models/data_frame_analytics/validation.ts

jgowdyelastic · 2021-03-05T16:02:52Z

x-pack/plugins/ml/public/application/components/callout/callout.tsx

+  switch (status) {
+    case VALIDATION_STATUS.INFO:
+      return 'primary';
+      break;


nit, the breaks are redundant in these switch statements as you're returning from inside the case

just noticed this is moved code. might be worth updating it still.

Updated in a7e5fc69c41f0b104667516fc53457eebf0a649d

jgowdyelastic · 2021-03-05T16:10:11Z

...data_frame_analytics/pages/analytics_creation/components/validation_step/validation_step.tsx

+import { EuiLoadingSpinner, EuiSpacer } from '@elastic/eui';
+import { i18n } from '@kbn/i18n';
+
+import { ml } from '../../../../../services/ml_api_service';


for new code we should be using the context which provides this.

import { useMlApiContext } from '../../../../../contexts/kibana';

Updated in a7e5fc69c41f0b104667516fc53457eebf0a649d

jgowdyelastic · 2021-03-05T16:20:30Z

x-pack/plugins/ml/public/application/services/ml_api_service/data_frame_analytics.ts

@@ -166,4 +166,12 @@ export const dataFrameAnalytics = {
      method: 'GET',
    });
  },
+  validateDataFrameAnalytics(analyticsConfig: DeepPartial<DataFrameAnalyticsConfig>) {
+    const body = JSON.stringify(analyticsConfig);
+    return http<any>({


can we replace any with the return type here?
should be the return type of validateAnalyticsJob

Updated in 943882adedeeaf1254d355f87d7b8549051e6c37

jgowdyelastic · 2021-03-05T16:27:46Z

x-pack/plugins/ml/server/models/data_frame_analytics/validation.ts

+};
+
+function getTrainingPercentMessage(trainingDocs: number) {
+  let trainingPercentMessage;


trainingPercentMessage probably isn't needed as we could return from within the if statement below.

Updated in a7e5fc69c41f0b104667516fc53457eebf0a649d

jgowdyelastic · 2021-03-05T16:30:29Z

x-pack/plugins/ml/server/models/data_frame_analytics/validation.ts

+  }
+
+  try {
+    const { body }: { body: ValidationSearchResult } = await asCurrentUser.search({


you should be able to use the generic on search here.
something like

const { body } = await asCurrentUser.search<ValidationSearchResult>(

Updated in a7e5fc69c41f0b104667516fc53457eebf0a649d

lcawl

Text strings LGTM!

jgowdyelastic

LGTM

kibanamachine · 2021-03-06T01:32:54Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`ml`	1735	1741	+6

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`ml`	6.4MB	6.4MB	+8.2KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`ml`	69.9KB	70.1KB	+163.0B

History

💚 Build #111467 succeeded 943882adedeeaf1254d355f87d7b8549051e6c37
💚 Build #111456 succeeded a7e5fc69c41f0b104667516fc53457eebf0a649d
💛 Build #111410 was flaky 8458efe8cd2ff4ffe13319473cd357721dc40e89
💚 Build #111164 succeeded a27f210bfc3f981021ba627aa6f32a542b2db70d
💔 Build #111132 failed 64b4e9dd5da186820cd28990b244d906bc3d3964

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @alvarezmelissa87

…1) (elastic#93478) * wip: create validationStep component * wip: trainingPercent check, analysisFields check. Step details * move validation check to server * handle no training percent in validation * move callout component to shared dir * use shared Callout component in AD val and update message headings * update types * adds functional tests for validation * adds api integration test for validate endpoint * consolidate messages for depvar and fields * fix accessibility test * update license * update validation messages * update types in validation model * add jobValidationReturnType

…1) (#93478) (#93869) * wip: create validationStep component * wip: trainingPercent check, analysisFields check. Step details * move validation check to server * handle no training percent in validation * move callout component to shared dir * use shared Callout component in AD val and update message headings * update types * adds functional tests for validation * adds api integration test for validate endpoint * consolidate messages for depvar and fields * fix accessibility test * update license * update validation messages * update types in validation model * add jobValidationReturnType

alvarezmelissa87 added release_note:enhancement :ml Feature:Data Frame Analytics ML data frame analytics features v7.13.0 labels Mar 3, 2021

alvarezmelissa87 requested a review from a team as a code owner March 3, 2021 18:14

alvarezmelissa87 self-assigned this Mar 3, 2021

alvarezmelissa87 requested review from lcawl, peteharverson and jgowdyelastic March 3, 2021 18:15

alvarezmelissa87 force-pushed the ml-dfa-validation-step branch from fadfb1e to a3bf34b Compare March 3, 2021 21:16

peteharverson reviewed Mar 4, 2021

View reviewed changes

x-pack/test/functional/apps/ml/data_frame_analytics/outlier_detection_creation.ts Show resolved Hide resolved

peteharverson reviewed Mar 4, 2021

View reviewed changes

x-pack/plugins/ml/common/constants/validation.ts Outdated Show resolved Hide resolved

alvarezmelissa87 changed the title ~~[ML] Data Frame Analytics creation wizard: add validation step~~ [ML] Data Frame Analytics creation wizard: add validation step (Part 1) Mar 4, 2021

peteharverson approved these changes Mar 4, 2021

View reviewed changes

lcawl reviewed Mar 4, 2021

View reviewed changes

jgowdyelastic reviewed Mar 5, 2021

View reviewed changes

lcawl approved these changes Mar 5, 2021

View reviewed changes

jgowdyelastic approved these changes Mar 5, 2021

View reviewed changes

alvarezmelissa87 added 9 commits March 5, 2021 14:18

wip: create validationStep component

fec8bb1

wip: trainingPercent check, analysisFields check. Step details

2d1fc0f

move validation check to server

f5db3c3

handle no training percent in validation

352c8b8

move callout component to shared dir

805adc6

use shared Callout component in AD val and update message headings

e497225

update types

a7ba485

adds functional tests for validation

e84e032

adds api integration test for validate endpoint

7e8c4ae

alvarezmelissa87 added 6 commits March 5, 2021 14:18

consolidate messages for depvar and fields

69112ad

fix accessibility test

79d3c10

update license

fdc7d5c

update validation messages

bb6ee9b

update types in validation model

7b4ff16

add jobValidationReturnType

9e15668

alvarezmelissa87 force-pushed the ml-dfa-validation-step branch from 943882a to 9e15668 Compare March 5, 2021 22:19

alvarezmelissa87 merged commit cac26b8 into elastic:master Mar 6, 2021

alvarezmelissa87 mentioned this pull request Mar 6, 2021

[7.x] [ML] Data Frame Analytics creation wizard: add validation step (Part 1) (#93478) #93869

Merged

alvarezmelissa87 deleted the ml-dfa-validation-step branch March 6, 2021 02:50

alvarezmelissa87 mentioned this pull request Mar 6, 2021

[ML]DF Analytics creation: Validate/gently nudge user to better train/test split percentage when dataset is large #78637

Closed

peteharverson mentioned this pull request Mar 16, 2021

Failing test: X-Pack Accessibility Tests.x-pack/test/accessibility/apps/ml·ts - ml for user with full ML access with data loaded data frame analytics create job create step for outlier job #94666

Closed

peteharverson added release_note:feature Makes this part of the condensed release notes and removed release_note:enhancement labels May 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Data Frame Analytics creation wizard: add validation step (Part 1) #93478

[ML] Data Frame Analytics creation wizard: add validation step (Part 1) #93478

alvarezmelissa87 commented Mar 3, 2021 •

edited

Loading

elasticmachine commented Mar 3, 2021

peteharverson Mar 4, 2021

alvarezmelissa87 Mar 4, 2021

peteharverson left a comment

alvarezmelissa87 commented Mar 4, 2021

lcawl left a comment

jgowdyelastic Mar 5, 2021

jgowdyelastic Mar 5, 2021

alvarezmelissa87 Mar 5, 2021

jgowdyelastic Mar 5, 2021

alvarezmelissa87 Mar 5, 2021

jgowdyelastic Mar 5, 2021

alvarezmelissa87 Mar 5, 2021

jgowdyelastic Mar 5, 2021

alvarezmelissa87 Mar 5, 2021

jgowdyelastic Mar 5, 2021

alvarezmelissa87 Mar 5, 2021

lcawl left a comment

jgowdyelastic left a comment

kibanamachine commented Mar 6, 2021

[ML] Data Frame Analytics creation wizard: add validation step (Part 1) #93478

[ML] Data Frame Analytics creation wizard: add validation step (Part 1) #93478

Conversation

alvarezmelissa87 commented Mar 3, 2021 • edited Loading

Summary

Current limitations used to validate

Regression/Classification example with warning messages:

Regression/Classification example success:

Outlier detection with warning messages:

Outlier detection success:

Checklist

elasticmachine commented Mar 3, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peteharverson left a comment

Choose a reason for hiding this comment

alvarezmelissa87 commented Mar 4, 2021

lcawl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcawl left a comment

Choose a reason for hiding this comment

jgowdyelastic left a comment

Choose a reason for hiding this comment

kibanamachine commented Mar 6, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

History

alvarezmelissa87 commented Mar 3, 2021 •

edited

Loading