[Fleet] Transform installation - order of dependencies

[susan-shu-c]

Kibana version:

Elasticsearch version:

  "version": {
    "number": "8.3.3",
    "build_flavor": "default",
    "build_type": "docker",
    "build_hash": "",
    "build_date": "2022-07-23T19:30:09.227964828Z",
    "build_snapshot": false,
    "lucene_version": "9.2.0",
    "minimum_wire_compatibility_version": "7.17.0",
    "minimum_index_compatibility_version": "7.0.0"
  },

Server OS version:

Browser version:

Browser OS version:

Original install method (e.g. download page, yum, from source, etc.):

Using Elastic Package v0.65.0 (Oct 5, 2022) - elastic-package stack up

Describe the bug:

While installing a (work in progress) Fleet integration, the destination index is not created.

• Create Host Risk Score package integrations#4019

Using the new transform spec

• [Fleet] Implement new transform installation mechanism #134321
• [ML] Update installation mechanism for Transforms in Fleet according to new specifications #140046

Steps to reproduce:

1. Set up a Security rule to [create an alert](https://[YOUR KIBANA URL]:5601/app/security/rules) any time there is @timestamp: * and running every 5 minutes. This is to create the .alerts-security.alerts-default index.
2. Use Elastic Package to build this package
3. On a local Kibana instance, use Fleet to install the package or use elastic-package install

Expected behavior:

The package is intended to do the following:

1. Read from .alerts-security.alerts-default
2. Create pivot transform and write results to destination index 1
3. Create latest transform based on destination index 1, and write results to destination index 2

_diagram by @ajosh0504 _

The following errors may appear, and could be different each time:

user@users-MacBook-Pro host_risk_score % elastic-package install
Install the package
Error: can't install the package: can't install the package: could not install package; API status code = 500; response body = {"statusCode":500,"error":"Internal Server Error","message":"validation_exception: [validation_exception] Reason: Validation Failed: 1: no such index [.alerts-security.host-risk-score];"}

user@users-MacBook-Pro host_risk_score % elastic-package install
Install the package
Error: can't install the package: can't install the package: could not install package; API status code = 500; response body = {"statusCode":500,"error":"Internal Server Error","message":"resource_not_found_exception: [resource_not_found_exception] Reason: Transform with id [host_risk_score.pivot_transform-transform-0.0.1] could not be found"}

user@users-MacBook-Pro host_risk_score % elastic-package install
Install the package
Error: can't install the package: can't install the package: could not install package; API status code = 500; response body = {"statusCode":500,"error":"Internal Server Error","message":"resource_not_found_exception: [resource_not_found_exception] Reason: Transform with id [host_risk_score.latest_transform-transform-0.0.1] could not be found"}

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

Since this package intends to have 2 transforms; the second one which relies on the output of the first one, it's possible that there's no strict install order of both transforms, so then the 2nd transform's install could error if the 1st transform didn't create the 1st destination index yet. (See diagram image above)

However, since both transforms error out randomly when I repeatedly install it via elastic-package install, I'm not totally sure.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[susan-shu-c]

Fixed with #142920, thank you @qn895 !