[Logs UI Meta Issue] Logs + ML integration

[jasonrhodes]

Summary

This tracks the tasks associated with the Logs UI / ML categorization feature.

Task Breakdown

Note: boxes are checked below when the corresponding issue has been created and linked here. 👍

Design

• Log rate issue ([Logs UI] [Design] Machine learning analysis screens #41497 and [Logs UI/Design] Improve log rate visualization #46399)
• Categorization (Log categories logs#3)

Implementation

• Turn source configuration into a tab ([Infra/Logs UI] Turn source configuration into a tab #39071)
• Create API routes to manage ML categorization jobs ([Logs UI Meta Issue] Create API routes to manage ML categorization jobs #41509)
  • Create API route to create ML categorization jobs ([Logs UI] Create API route to create ML categorization jobs  #42017)
  • Create API route to check the status of ML jobs ([Logs UI] Create API route to check the status of ML jobs #42018)
  • Create API route to remove ML categorization jobs ([Logs UI] Create API route to remove ML categorization jobs #42019)
• Create API routes to access ML categorization results ([Logs UI] Create API routes to access ML categorization results #41805)
  • Create API route to access log rate analysis results ([Logs UI] Create API route to access log rate analysis results #42057)
  • Create API route to access rare log category analysis results ([Logs UI] Create API route to access rare log category analysis results #42061)
  • Create API route to access frequent log category analysis results ([Logs UI] Create API route to access common log category analysis results #42065)
• Add analysis tab to the tab bar ([Logs UI] Add analysis tab #41879)
• Create ML module for log analysis ([Logs UI] Create ML module for log analysis #42872)
• Create screen to set up analysis ML jobs ([Logs UI] Create screen to set up analysis ML jobs #41877)
• Create screen to show results of the analysis ML jobs ([Logs UI] Create screen to show results of the analysis ML jobs #41881)
  • Implement the Log Rate portion of the results screen ([Logs UI] Implement the Log Rate portion of the results screen #42771)
  • Implement the Rare Messages portion of the results screen ([Logs UI] Implement the Rare Messages portion of the results screen #42773)
  • Implement the Log Categories portion of the results screen ([Logs UI] Add the log categories result screen #42776)
• ❓ Add ML analysis to settings screen, with links to ML UI and ability to disable ML analysis (no editing is allowed)
• Phase 1 cleanup / bug fixing / tweaks ([LogsUI] Cleanup / bug fixing / error handling for phase 1 of ML analysis project #43766)

Open questions:

• If a user deletes any of the ML jobs created for the in progress screen, how do we handle this? Note: for simplicity I think we should treat categorization as an all-or-nothing enable/disable from our point of view, but a user is free to visit the ML UI and remove any job they like. That's the edge (?) case we need to worry about no matter what.
• How do we determine if a user has the right permissions? In the past with APM, this has been difficult since parts of ML are now available in Basic, but I don't think the parts we need are. (In APM, we always show the "Enable anomaly detection" link, and if it fails due to permissions, we handle that error -- not the best solution IMO so it'd be nice if we could do better.)
• Is there an existing article or blog post about ML categorization? If not, can we commission one so it's ready to be linked to?

[elasticmachine]

Pinging @elastic/infra-logs-ui

[jgowdyelastic]

How do we determine if a user has the right permissions?

There is an ML kibana endpoint which will give you all the necessary ml permissions. This is being used by SIEM for their ML integration, so might be worth looking at how they are using it.
api/ml/ml_capabilities
More details here: #39749

[weltenwort]

Add ML analysis to settings screen, with links to ML UI and ability to disable ML analysis (no editing is allowed)

@jasonrhodes is the settings screen really the right place for this? the setup doesn't happen there either, so I would have expected this to be located on the ml analysis page itself 🤔

[jasonrhodes]

is the settings screen really the right place for this? the setup doesn't happen there either, so I would have expected this to be located on the ml analysis page itself 🤔

Open to discussion for sure. If we don't move "settings" to the tab called settings, I think we may want to re-think what that tab is called. But my preference would be to use it for all of these type of things that don't need to always be in your face and usually are only interacted with once, rather than trying to figure out how to fit that into the ML Analysis UI. We can see how it goes as we build out the page design with @hbharding

[jasonrhodes]

• Represent active tab selection in the URL

@weltenwort do you think this should be in a separate task or should it be done in the task where we add the tabs to the pages?

[weltenwort]

My first instinct was to split it up to keep the steps small. Wiring it up to the router could then be done in parallel to the creation of the setup and result screens. But OTOH just using the router immediately wouldn't be too much extra work either.

[jasonrhodes]

Following up on ML permissions, here is what I've confirmed:

$ curl --user admin:{redacted} localhost:5601/xuc/api/ml/ml_capabilities | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   661  100   661    0     0   1078      0 --:--:-- --:--:-- --:--:--  1078
{
  "capabilities": {
    "canGetJobs": true,
    "canCreateJob": true,
    "canDeleteJob": true,
    "canOpenJob": true,
    "canCloseJob": true,
    "canForecastJob": true,
    "canGetDatafeeds": true,
    "canStartStopDatafeed": true,
    "canUpdateJob": true,
    "canUpdateDatafeed": true,
    "canPreviewDatafeed": true,
    "canGetCalendars": true,
    "canCreateCalendar": true,
    "canDeleteCalendar": true,
    "canGetFilters": true,
    "canCreateFilter": true,
    "canDeleteFilter": true,
    "canFindFileStructure": true,
    "canGetDataFrameJobs": true,
    "canDeleteDataFrameJob": true,
    "canPreviewDataFrameJob": true,
    "canCreateDataFrameJob": true,
    "canStartStopDataFrameJob": true
  },
  "upgradeInProgress": false,
  "isPlatinumOrTrialLicense": true,
  "mlFeatureEnabledInSpace": true
}

vs a "No ML" space:

$ curl --user admin:{redacted} localhost:5601/xuc/s/no-ml/api/ml/ml_capabilities | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   685  100   685    0     0   1364      0 --:--:-- --:--:-- --:--:--  1364
{
  "capabilities": {
    "canGetJobs": false,
    "canCreateJob": false,
    "canDeleteJob": false,
    "canOpenJob": false,
    "canCloseJob": false,
    "canForecastJob": false,
    "canGetDatafeeds": false,
    "canStartStopDatafeed": false,
    "canUpdateJob": false,
    "canUpdateDatafeed": false,
    "canPreviewDatafeed": false,
    "canGetCalendars": false,
    "canCreateCalendar": false,
    "canDeleteCalendar": false,
    "canGetFilters": false,
    "canCreateFilter": false,
    "canDeleteFilter": false,
    "canFindFileStructure": false,
    "canGetDataFrameJobs": false,
    "canDeleteDataFrameJob": false,
    "canPreviewDataFrameJob": false,
    "canCreateDataFrameJob": false,
    "canStartStopDataFrameJob": false
  },
  "upgradeInProgress": false,
  "isPlatinumOrTrialLicense": true,
  "mlFeatureEnabledInSpace": false
}

vs running yarn es snapshot --license=basic and requesting from the default space:

$ curl --user elastic:changeme localhost:5601/api/ml/ml_capabilities | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   679  100   679    0     0   1054      0 --:--:-- --:--:-- --:--:--  1052
{
  "capabilities": {
    "canGetJobs": false,
    "canCreateJob": false,
    "canDeleteJob": false,
    "canOpenJob": false,
    "canCloseJob": false,
    "canForecastJob": false,
    "canGetDatafeeds": false,
    "canStartStopDatafeed": false,
    "canUpdateJob": false,
    "canUpdateDatafeed": false,
    "canPreviewDatafeed": false,
    "canGetCalendars": false,
    "canCreateCalendar": false,
    "canDeleteCalendar": false,
    "canGetFilters": false,
    "canCreateFilter": false,
    "canDeleteFilter": false,
    "canFindFileStructure": true,
    "canGetDataFrameJobs": true,
    "canDeleteDataFrameJob": true,
    "canPreviewDataFrameJob": true,
    "canCreateDataFrameJob": true,
    "canStartStopDataFrameJob": true
  },
  "upgradeInProgress": false,
  "isPlatinumOrTrialLicense": false,
  "mlFeatureEnabledInSpace": true
}

[jasonrhodes]

So I think we probably want to base it on capabilities.canCreateJob here.

[weltenwort]

I tried to visualize a rough user flow in the following graph:

[jasonrhodes]

Closing this in favor of the working group meta issue for "Logs/ML integration" (220)

@weltenwort @Kerry350 if there's anything here you think should move there, feel free -- thanks!