[ML] Use indices options in anomaly detection job wizards #91830

jgowdyelastic · 2021-02-18T12:47:04Z

In the advanced job wizard, it is now possible to add indices_options to the datafeed, and will now be used for all searches performed during the job creation UI process (e.g. the datafeed preview, loading the start and end times for the index, bucket span and model memory estimation)

Moves the datafeed preview function to the server side, behind the endpoint /api/ml/jobs/datafeed_preview

Also updates the JSON editor save function to recalculate the start and end times for the index which is used for all searches. The start and end times of the index are loaded when the wizard opens, but after a datafeed change these times may now be incorrect or the original times may have been incorrect if there was a problem with the search (e.g. a problem with CCS)

Closes #91983

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
This was checked for breaking API changes and was labeled appropriately

-- edit

For indices/data streams, doing a ccs search via *:my-index requires that my-index exists on all remote clusters being queried. If my-index is missing in one of the remote clusters then the index_not_found_exception error is returned. Unless ignore_unavailable is set to true.

jgowdyelastic · 2021-03-01T13:02:21Z

@elasticmachine merge upstream

jgowdyelastic · 2021-03-08T16:39:15Z

@elasticmachine merge upstream

jgowdyelastic · 2021-03-09T15:39:04Z

@elasticmachine merge upstream

jgowdyelastic · 2021-03-11T11:31:25Z

@elasticmachine merge upstream

elasticmachine · 2021-03-11T16:41:11Z

Pinging @elastic/ml-ui (:ml)

jgowdyelastic · 2021-03-15T09:17:06Z

@elasticmachine merge upstream

peteharverson · 2021-03-15T11:47:02Z

Testing locally, I create a job in the advanced wizard and set indices_options in the datafeed JSON (attempting to use "ignore_unavailable": true,)

{
  "datafeed_id": "datafeed-fq_advanced",
  "job_id": "fq_advanced",
  "indices": [
    "farequote-2019"
  ],
  "indices_options": {
    "expand_wildcards": [
      "open"
    ],
    "ignore_unavailable": true,
    "allow_no_indices": true,
    "ignore_throttled": true
  },
  "query": {
    "bool": {
      "must": [
        {
          "match_all": {}
        }
      ]
    }
  }
}

and when I hit Estimate bucket span, I see this error, which implies the wrong arguments are getting sent for runtime_mappings in one of the calls:

peteharverson

Tested latest edit and LGTM

x-pack/plugins/ml/server/models/job_service/datafeeds.ts

qn895 · 2021-03-16T13:24:09Z

x-pack/plugins/ml/server/routes/job_service.ts

@@ -548,7 +551,9 @@ export function jobServiceRoutes({ router, routeGuard }: RouteInitialization) {
          aggFieldNamePairs,
          splitFieldName,
          splitFieldValue,
-          runtimeMappings
+          runtimeMappings,
+


Should we get rid of this extra line?

qn895 · 2021-03-16T13:24:30Z

x-pack/plugins/ml/server/routes/job_service.ts

@@ -603,7 +609,9 @@ export function jobServiceRoutes({ router, routeGuard }: RouteInitialization) {
          query,
          aggFieldNamePairs,
          splitFieldName,
-          runtimeMappings
+          runtimeMappings,
+


Same for this extra line here as well

qn895 · 2021-03-16T13:26:11Z

x-pack/plugins/ml/server/routes/schemas/fields_service_schema.ts

@@ -29,4 +29,6 @@ export const getTimeFieldRangeSchema = schema.object({
  timeFieldName: schema.maybe(schema.string()),
  /** Query to match documents in the index(es). */
  query: schema.maybe(schema.any()),
+  /** Additional search options. */
+  indicesOptions: schema.maybe(schema.any()),


Since this indicesOptions is used in a lot of end points, should we define the schema for it?

qn895 · 2021-03-16T13:39:12Z

LGTM. Just had a few minor comments about the extra lines and indicesOptions schema.

) * [ML] WIP Datafeed preview refactor * adding indices options to ad job creation searches * update datafeed preview schema * updating types * recalculating wizard time range on JSON edit save * updating endpoint docs * fixing types * more type fixes * fixing missing runtime fields * using isPopulatedObject * adding indices options schema * fixing test * fixing schema Co-authored-by: Kibana Machine <[email protected]>

kibanamachine · 2021-03-16T19:58:02Z

💚 Backport successful

✅ 7.x / #94755

Successful backport PRs will be merged automatically after passing CI.

…94755) * [ML] WIP Datafeed preview refactor * adding indices options to ad job creation searches * update datafeed preview schema * updating types * recalculating wizard time range on JSON edit save * updating endpoint docs * fixing types * more type fixes * fixing missing runtime fields * using isPopulatedObject * adding indices options schema * fixing test * fixing schema Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: James Gowdy <[email protected]>

…-action * 'master' of github.com:elastic/kibana: (44 commits) Migrate the optimizer mixin to core (#94272) Replace EuiCodeBlock with JsonCodeEditor in DiscoverGrid (#92442) [ML] Anomaly Detection: Migrate validation messages links to use docLinks. (#94568) [Lists][Exceptions] - Adding basic linting, i18n and storybook support (#94772) [Fleet] Add test/fix for invalid/missing ids in bulk agent reassign (#94632) [Security Solution] [Cases] Move create page components and dependencies to Cases (#94444) [ML] Data Frame Analytics accessibility tests: fix flaky outlier creation test (#94735) [Security Solutions] Remove commented out old linter rules (#94753) [App Search] Role mappings migration part 2 (#94461) [CI] Update Backport action inputs to match updated ones (#94721) [chore] Remove the infra team from CODEOWNERS (#94740) [Connectors UI] Make UI use new connector APIs (#94180) [ML] Use indices options in anomaly detection job wizards (#91830) Remove `string` as a valid registry var type value (#94174) [Alerts] Replaces legacy es client with the ElasticsearchClient for alerts and triggers_actions_ui plugins. (#93364) [Reporting-CSV Export] Re-write CSV Export using SearchSource (#88303) chore(NA): upgrade bazel rules nodejs to v3.2.2 (#94726) [APM] Settings: Update layout and update/add descriptions (#94398) skip flaky suite (#94666) [XY axis] Integrates legend color picker with the eui palette (#90589) ... # Conflicts: # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/components/phases/cold_phase/cold_phase.tsx # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/components/phases/shared_fields/searchable_snapshot_field/searchable_snapshot_field.tsx # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/constants.ts # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/edit_policy.tsx # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/form/configuration_context.tsx # x-pack/plugins/index_lifecycle_management/public/shared_imports.ts

kibanamachine · 2021-05-26T10:18:11Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 74e02d1
Storybooks not built
Pipeline Steps (look for red circles / failed steps)
Interpreting CI Failures

Failed CI Steps

Metrics [docs]

‼️ ERROR: no builds found for mergeBase sha [ed5bbda]

History

💚 Build #113464 succeeded 74e02d1
💔 Build #113387 failed ddcf29d
💔 Build #113364 failed 76a4479
💚 Build #113051 succeeded feb5443

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jgowdyelastic

[ML] WIP Datafeed preview refactor

6ec8ea2

kibanamachine and others added 3 commits March 1, 2021 08:02

Merge branch 'master' into datafeed-preview-refactor

c0e65d0

adding indices options to ad job creation searches

9582557

update datafeed preview schema

05fd651

jgowdyelastic changed the title ~~[ML] WIP Datafeed preview refactor~~ [ML] WIP Use indices options in anomaly detection job wizards Mar 2, 2021

updating types

380b57e

Merge branch 'master' into datafeed-preview-refactor

7603ab7

Merge branch 'master' into datafeed-preview-refactor

699c60e

kibanamachine and others added 2 commits March 11, 2021 06:31

Merge branch 'master' into datafeed-preview-refactor

3e23790

recalculating wizard time range on JSON edit save

916aab1

jgowdyelastic requested review from peteharverson and qn895 March 11, 2021 16:40

jgowdyelastic changed the title ~~[ML] WIP Use indices options in anomaly detection job wizards~~ [ML] Use indices options in anomaly detection job wizards Mar 11, 2021

jgowdyelastic marked this pull request as ready for review March 11, 2021 16:40

jgowdyelastic requested a review from a team as a code owner March 11, 2021 16:40

jgowdyelastic self-assigned this Mar 11, 2021

jgowdyelastic added :ml auto-backport Deprecated - use backport:version if exact versions are needed Feature:Anomaly Detection ML anomaly detection release_note:fix review v7.13.0 v8.0.0 labels Mar 11, 2021

jgowdyelastic added release_note:enhancement and removed release_note:fix labels Mar 11, 2021

jgowdyelastic added 2 commits March 11, 2021 17:00

updating endpoint docs

b0225ad

fixing types

37595c1

kibanamachine and others added 2 commits March 15, 2021 05:17

Merge branch 'master' into datafeed-preview-refactor

109f454

more type fixes

e2224f9

fixing missing runtime fields

feb5443

peteharverson approved these changes Mar 15, 2021

View reviewed changes

qn895 reviewed Mar 16, 2021

View reviewed changes

x-pack/plugins/ml/server/models/job_service/datafeeds.ts Outdated Show resolved Hide resolved

qn895 reviewed Mar 16, 2021

View reviewed changes

using isPopulatedObject

8ca21bb

qn895 approved these changes Mar 16, 2021

View reviewed changes

jgowdyelastic added 3 commits March 16, 2021 13:43

adding indices options schema

76a4479

fixing test

ddcf29d

fixing schema

74e02d1

jgowdyelastic merged commit 4914acc into elastic:master Mar 16, 2021

jgowdyelastic deleted the datafeed-preview-refactor branch March 16, 2021 19:55

kibanamachine mentioned this pull request Mar 16, 2021

[7.x] [ML] Use indices options in anomaly detection job wizards (#91830) #94755

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Use indices options in anomaly detection job wizards #91830

[ML] Use indices options in anomaly detection job wizards #91830

jgowdyelastic commented Feb 18, 2021 •

edited by sophiec20

Loading

jgowdyelastic commented Mar 1, 2021

jgowdyelastic commented Mar 8, 2021

jgowdyelastic commented Mar 9, 2021

jgowdyelastic commented Mar 11, 2021

elasticmachine commented Mar 11, 2021

jgowdyelastic commented Mar 15, 2021

peteharverson commented Mar 15, 2021

peteharverson left a comment

qn895 Mar 16, 2021

jgowdyelastic Mar 16, 2021

qn895 Mar 16, 2021

jgowdyelastic Mar 16, 2021

qn895 Mar 16, 2021

jgowdyelastic Mar 16, 2021

qn895 commented Mar 16, 2021

kibanamachine commented Mar 16, 2021

kibanamachine commented May 26, 2021 •

edited

Loading

[ML] Use indices options in anomaly detection job wizards #91830

[ML] Use indices options in anomaly detection job wizards #91830

Conversation

jgowdyelastic commented Feb 18, 2021 • edited by sophiec20 Loading

jgowdyelastic commented Mar 1, 2021

jgowdyelastic commented Mar 8, 2021

jgowdyelastic commented Mar 9, 2021

jgowdyelastic commented Mar 11, 2021

elasticmachine commented Mar 11, 2021

jgowdyelastic commented Mar 15, 2021

peteharverson commented Mar 15, 2021

peteharverson left a comment

Choose a reason for hiding this comment

qn895 Mar 16, 2021

Choose a reason for hiding this comment

jgowdyelastic Mar 16, 2021

Choose a reason for hiding this comment

qn895 Mar 16, 2021

Choose a reason for hiding this comment

jgowdyelastic Mar 16, 2021

Choose a reason for hiding this comment

qn895 Mar 16, 2021

Choose a reason for hiding this comment

jgowdyelastic Mar 16, 2021

Choose a reason for hiding this comment

qn895 commented Mar 16, 2021

kibanamachine commented Mar 16, 2021

💚 Backport successful

kibanamachine commented May 26, 2021 • edited Loading

💔 Build Failed

Failed CI Steps

Metrics [docs]

History

jgowdyelastic commented Feb 18, 2021 •

edited by sophiec20

Loading

kibanamachine commented May 26, 2021 •

edited

Loading