[Logs UI] Create screen to set up analysis ML jobs #43413

Zacqary · 2019-08-15T23:03:51Z

Summary

Closes #41877
Based on #43050

Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

This was checked for cross-browser compatibility, including a check against IE11
Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
This was checked for keyboard-only and screenreader accessibility

For maintainers

This was checked for breaking API changes and was labeled appropriately
This includes a feature addition or change that requires a release note and was labeled appropriately

…uirement filtering check

…dule-setup-hook

…hook' into 41877-ml-setup-screen

elasticmachine · 2019-08-15T23:03:53Z

Pinging @elastic/infra-logs-ui

elasticmachine · 2019-08-15T23:59:55Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

Kerry350 · 2019-08-16T11:28:33Z

@Zacqary

I've taken a look at the request failures you were seeing. For the most part these were just small syntax errors (with bucketSpan and timeField), which I've fixed in this commit.

This was the error coming back in relation to that:

{
   "error":{
      "root_cause":[
         {
            "type":"status_exception",
            "reason":"Datafeed aggregations are not parsable"
         }
      ],
      "type":"x_content_parse_exception",
      "reason":"[1:486] [datafeed_config] failed to parse field [aggregations]",
      "caused_by":{
         "type":"status_exception",
         "reason":"Datafeed aggregations are not parsable",
         "caused_by":{
            "type":"illegal_argument_exception",
            "reason":"failed to parse [{bucketSpan}ms]",
            "caused_by":{
               "type":"number_format_exception",
               "reason":"For input string: \"{bucketspan}\""
            }
         }
      }
   },
   "status":400
}

However, this did lead me to uncover a potential problem once the syntax issues were fixed. I started to get this error:

{
   "error":{
      "root_cause":[
         {
            "type":"status_exception",
            "reason":"No node found to start datafeed [datafeed-kibana-logs-ui-kerry-default-log-entry-rate], allocation explanation [cannot start datafeed [datafeed-kibana-logs-ui-kerry-default-log-entry-rate] because index [filebeat-*,kibana_sample_data_logs*] does not exist, is closed, or is still initializing.]"
         }
      ],
      "type":"status_exception",
      "reason":"No node found to start datafeed [datafeed-kibana-logs-ui-kerry-default-log-entry-rate], allocation explanation [cannot start datafeed [datafeed-kibana-logs-ui-kerry-default-log-entry-rate] because index [filebeat-*,kibana_sample_data_logs*] does not exist, is closed, or is still initializing.]"
   },
   "status":409
}

this is with the API sending a string of filebeat-*,kibana_sample_data_logs* using the multiple indices syntax, which is the default in the source configuration (and users can of course configure this to cover as many indices as they need). Back over in #42872, the indexPatternName was set to filebeat-* in the "Testing hints". If I swap this over to a hardcoded filebeat-* everything works as expected, with the jobs and datafeed returning success: true and started: true.

@jgowdyelastic does indexPatternName have to be a singular index pattern? Is there a way we can send multiple indices? I followed through to updateDatafeedIndices() from setupModuleItems() and it looked like it got passed through as is, but I don't know the code well past that point.

x-pack/legacy/plugins/infra/public/pages/logs/analysis/page_setup_content.tsx

elasticmachine · 2019-08-16T12:28:17Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

…o 41877-ml-setup-screen

elasticmachine · 2019-08-19T17:44:00Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

…p-screen

elasticmachine · 2019-08-19T21:28:28Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

sophiec20 · 2019-08-20T15:21:40Z

Looking at the elasticsearch ML APIs, a datafeed error occurs on _start if the indices are supplied as a single string rather than an array. Both of the following examples can be PUT however only the first example will _start without error.

PUT _ml/datafeeds/job1
{
    "job_id": "job1",
    "indices": [ "index1-*", "index2-*" ]
}

PUT _ml/datafeeds/job2
{
    "job_id": "job2",
    "indices": [ "index1-*,index2-*" ]
}

# error on _start:
# "type" : "resource_not_found_exception",
# "reason" : "datafeed [job2] cannot retrieve data because index [index1-*,index2-*] does not exist"

It seems to me that the Kibana ML setup process should split the string it receives for INDEX_PATTERN_NAME if it contains commas and create the datafeed correctly using an array of indices.

Note, in order for the ML analysis results to be meaningful for an event rate analysis, for this single search the resulting documents require the same time field throughout.

cc @jgowdyelastic

jasonrhodes · 2019-08-21T12:09:59Z

Both of the following examples can be PUT however only the first example will _start without error.
PUT _ml/datafeeds/job1
{
    "job_id": "job1",
    "indices": [ "index1-*", "index2-*" ]
}

@sophiec20 if this example works, can we just split our string on commas before we send it? That would be a very easy thing for us to do, even if ML eventually splits on commas too.

jasonrhodes · 2019-08-21T12:47:27Z

can we just split our string on commas before we send it? That would be a very easy thing for us to do, even if ML eventually splits on commas too.

Ah I think I understand now -- we don't call the datafeeds endpoints, we call the module endpoint, so this isn't something we can fix on our end.

jgowdyelastic · 2019-08-21T17:25:24Z

#43686 created to enable the use of a comma separated index pattern in the modules/setup endpoint

jasonrhodes · 2019-08-22T01:31:57Z

x-pack/legacy/plugins/infra/public/containers/logs/log_analysis/api/ml_setup_module_api.ts

+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+import * as rt from 'io-ts';


Can someone explain to me why io-ts gets called rt?

Kerry350 · 2019-08-22T08:27:17Z

@jgowdyelastic Awesome, thank you!

…reen

Kerry350 · 2019-08-22T11:43:04Z

Just pushed the change here for ensuring the sample data index doesn't get passed to the ML setup.

jasonrhodes · 2019-08-22T12:08:24Z

x-pack/legacy/plugins/infra/public/containers/logs/log_analysis/api/ml_setup_module_api.ts

+  } else {
+    return indexPattern;
+  }
+};


Would this work with a filter() too? If so that might be easier to follow later when we want to remember what this does and safely rip it out ... maybe it's just me but I always forget how indices and -1 and Array#splice all work :P

return indices.filter(index => index !== SAMPLE_DATA_INDEX).join(',')

👍 I'll change this on the "cleanup / bug fixing" PR so I don't need to trigger another build cycle.

jasonrhodes

Everything here LGTM, great work @Kerry350 and @Zacqary

elasticmachine · 2019-08-22T12:53:34Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

* Add empty analysis tab * Add ml capabilities check * Add job status checking functionality * Add a loading page for the job status check * Change types / change method for deriving space ID / change setup requirement filtering check * Use new structure * Add module setup to log analysis jobs hook * Add ID to path * [Logs UI] Add analyis setup landing screen * Add function to set up ML module on click * Use partial type for start and end props * Add start and end time selection * Fix syntax * Change seconds timestamp to ms * Update wording * Use FormControlLayout to clear datepickers * Update wording about earlier start date * Remove specific point in time wording * Fix typechecking * Reload analysis page on successful job creation * Add error handling for setup failure * Update description ton of feature to reflect 7.4 feature set * Add toggleable default message * Revert to EuiFormControlLayout until eui changes are pushed * Remove sample data index if user has it set

weltenwort and others added 15 commits August 8, 2019 14:37

Add empty analysis tab

de0499e

Merge branch 'master' into logs-ui-add-analysis-tab

8d06a54

Add ml capabilities check

0ba9cca

Add job status checking functionality

84248c2

Add a loading page for the job status check

31be96e

Change types / change method for deriving space ID / change setup req…

6c0bac5

…uirement filtering check

Use new structure

98a9fa9

Add module setup to log analysis jobs hook

83330d7

Merge remote-tracking branch 'upstream/master' into logs-ui-add-ml-mo…

916c167

…dule-setup-hook

Add ID to path

6951bde

Merge remote-tracking branch 'weltenwort/logs-ui-add-ml-module-setup-…

fe9c76a

…hook' into 41877-ml-setup-screen

[Logs UI] Add analyis setup landing screen

705577a

Add function to set up ML module on click

1beb78c

Use partial type for start and end props

f6594e3

Add start and end time selection

fb139df

Zacqary added release_note:enhancement Feature:Logs UI Logs UI feature Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v7.4.0 labels Aug 15, 2019

Zacqary requested a review from a team August 15, 2019 23:03

Kerry350 self-requested a review August 16, 2019 08:55

Fix syntax

c36586f

Kerry350 requested changes Aug 16, 2019

View reviewed changes

jasonrhodes mentioned this pull request Aug 16, 2019

[Logs UI] Add ML module setup hook #43050

Closed

Zacqary added 2 commits August 16, 2019 12:09

Change seconds timestamp to ms

2569444

Merge branch '41877-ml-setup-screen' of github.com:Zacqary/kibana int…

c4a2c0a

…o 41877-ml-setup-screen

Add toggleable default message

4908922

Revert to EuiFormControlLayout until eui changes are pushed

c7a33dc

Zacqary marked this pull request as ready for review August 19, 2019 20:25

Merge branch 'master' of github.com:elastic/kibana into 41877-ml-setu…

4871bad

…p-screen

Kerry350 mentioned this pull request Aug 21, 2019

[Logs UI] [Discussion] Remove kibana_sample_data_logs* from the default source configuration #43671

Closed

Zacqary added the v8.0.0 label Aug 21, 2019

jasonrhodes reviewed Aug 22, 2019

View reviewed changes

Kerry350 added 2 commits August 22, 2019 11:37

Merge remote-tracking branch 'upstream/master' into 41877-ml-setup-sc…

c13eb2f

…reen

Remove sample data index if user has it set

41ca5e4

jasonrhodes reviewed Aug 22, 2019

View reviewed changes

jasonrhodes approved these changes Aug 22, 2019

View reviewed changes

Kerry350 approved these changes Aug 22, 2019

View reviewed changes

Kerry350 merged commit 40b28c7 into elastic:master Aug 22, 2019

Kerry350 mentioned this pull request Aug 22, 2019

[LogsUI] Cleanup / bug fixing / error handling for phase 1 of ML analysis project #43766

Closed

4 tasks

Zacqary deleted the 41877-ml-setup-screen branch August 22, 2019 15:16

Zacqary mentioned this pull request Aug 22, 2019

[7.x] [Logs UI] Create screen to set up analysis ML jobs (#43413) #43782

Merged

weltenwort added release_note:skip Skip the PR/issue when compiling release notes and removed release_note:enhancement labels Sep 17, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Logs UI] Create screen to set up analysis ML jobs #43413

[Logs UI] Create screen to set up analysis ML jobs #43413

Zacqary commented Aug 15, 2019 •

edited

Loading

elasticmachine commented Aug 15, 2019

elasticmachine commented Aug 15, 2019

Kerry350 commented Aug 16, 2019 •

edited

Loading

elasticmachine commented Aug 16, 2019

elasticmachine commented Aug 19, 2019

elasticmachine commented Aug 19, 2019

sophiec20 commented Aug 20, 2019 •

edited

Loading

jasonrhodes commented Aug 21, 2019

jasonrhodes commented Aug 21, 2019

jgowdyelastic commented Aug 21, 2019

jasonrhodes Aug 22, 2019

Kerry350 commented Aug 22, 2019

Kerry350 commented Aug 22, 2019

jasonrhodes Aug 22, 2019 •

edited

Loading

Kerry350 Aug 22, 2019

jasonrhodes left a comment

elasticmachine commented Aug 22, 2019

[Logs UI] Create screen to set up analysis ML jobs #43413

[Logs UI] Create screen to set up analysis ML jobs #43413

Conversation

Zacqary commented Aug 15, 2019 • edited Loading

Summary

Checklist

For maintainers

elasticmachine commented Aug 15, 2019

elasticmachine commented Aug 15, 2019

💔 Build Failed

Kerry350 commented Aug 16, 2019 • edited Loading

elasticmachine commented Aug 16, 2019

💔 Build Failed

elasticmachine commented Aug 19, 2019

💔 Build Failed

elasticmachine commented Aug 19, 2019

💚 Build Succeeded

sophiec20 commented Aug 20, 2019 • edited Loading

jasonrhodes commented Aug 21, 2019

jasonrhodes commented Aug 21, 2019

jgowdyelastic commented Aug 21, 2019

jasonrhodes Aug 22, 2019

Choose a reason for hiding this comment

Kerry350 commented Aug 22, 2019

Kerry350 commented Aug 22, 2019

jasonrhodes Aug 22, 2019 • edited Loading

Choose a reason for hiding this comment

Kerry350 Aug 22, 2019

Choose a reason for hiding this comment

jasonrhodes left a comment

Choose a reason for hiding this comment

elasticmachine commented Aug 22, 2019

💚 Build Succeeded

Zacqary commented Aug 15, 2019 •

edited

Loading

Kerry350 commented Aug 16, 2019 •

edited

Loading

sophiec20 commented Aug 20, 2019 •

edited

Loading

jasonrhodes Aug 22, 2019 •

edited

Loading