Move ILM out of legacy

[cjcenizal]

Other changes

• Removed unused "get template" API route.
• Removed unnecessary custom styles.

Testing

Run ES with a custom node attribute so you can configure allocation and test the node API.

yarn es snapshot --license=trial -E node.attr.rack=r1

Settings

xpack.ilm.ui.enabled: false

With the above setting, verify that ILM isn't visible in the UI and you can't filter by lifecycle status or phase in Index Management.

xpack.ilm.enabled: false

With the above setting, verify the same effect.

xpack.ilm.filteredNodeAttributes: ["rack"]

With the above setting, verify that the r1 node attribute isn't available for selection under the warm phase's shard allocation when creating a policy.

Create index lifecycle policy

Create variations of an index lifecycle policy, ensuring that the API validation is appropriately lenient and correctly typed.

Phase variations

• Create policy with just hot phase
• Create policy with all phases: hot, warm, cold, and delete

Hot phase variations

• Create policy with a hot phase configuration that has rollover enabled, maximum index size set, maximum documents set, maximum age set, and index priority set.
• Create policy with a hot phase configuration that has rollover enabled and only maximum index size set.
• Create policy with a hot phase configuration that has rollover enabled and only maximum documents set.
• Create policy with a hot phase configuration that has rollover enabled and only maximum age set.
• Create policy with rollover disabled and without index priority set.

Warm phase variations

Note: All of these variations assume the hot phase is configured with rollover enabled.

• Create policy with a warm phase configuration that has rollover enabled, default allocation selected, replicas set, shrink enabled and its fields set, force merge enabled and its fields set, and index priority set.
• Create the same as above but with the r1 node attribute selected for shard allocation. Before you save this policy, click the "View list of nodes attached to this configuration" button and verify that information shows up in the flyout.
• Create policy with a warm phase configuration that has rollover disabled, default allocation selected, without replicas set, shrink disabled, force merge disabled, and without index priority set. Note: if you're editing an existing policy and clearing the index priority, this won't be persisted. This is a known bug.

Cold phase variations

Note: All of these variations assume the hot phase is configured with rollover enabled.

• Create policy with a cold phase configuration that has default allocation selected, replicas set, freeze enabled, and index priority set.
• Create the same as above but with the r1 node attribute selected for shard allocation.
• Create policy with a cold phase configuration that has default allocation selected, without replicas set, freeze disabled, and without index priority set. Note: if you're editing an existing policy and clearing the index priority, this won't be persisted. This is a known bug.

Delete phase variations

None, because this phase can't be configured. As long as you test that a policy can be created with the phase enabled, we're good.

Actions

• Delete one of the policies you created.
• Batch delete multiple policies.

Index Management

Test that the Frozen badge, phase filtering, and lifecycle information is surfaced in Index Management by running this series of requests in Console:

PUT /_ilm/policy/full
{
  "policy": {
    "phases" : {
      "hot" : {
        "min_age" : "0ms",
        "actions" : {
          "rollover" : {
            "max_docs" : 1
          }
        }
      },
      "warm" : {
        "min_age" : "15s",
        "actions" : {
          "forcemerge" : {
            "max_num_segments" : 1
          },
          "shrink" : {
            "number_of_shards" : 1
          }
        }
      },
      "cold" : {
        "min_age" : "30s",
        "actions" : {
          "freeze": {}
        }
      },
      "delete" : {
        "min_age" : "1d",
        "actions" : {
          "delete" : { }
        }
      }
    }
  }
}

PUT _template/test
{
  "index_patterns": ["test-*"],
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 0,
    "index.lifecycle.name": "full",
    "index.lifecycle.rollover_alias": "test-alias"
  }
}

PUT /test-000001
{
  "aliases": {
    "test-alias": {
      "is_write_index": true
    }
  }
}

PUT test-alias/_doc/1
{
  "a": "a"
}

PUT /_cluster/settings
{
  "transient": {
    "logger.org.elasticsearch.xpack.core.indexlifecycle": "TRACE",
    "logger.org.elasticsearch.xpack.indexlifecycle": "TRACE",
    "logger.org.elasticsearch.xpack.core.ilm": "TRACE",
    "logger.org.elasticsearch.xpack.ilm": "TRACE",
    "indices.lifecycle.poll_interval": "10s"
  }
}

Then go into Index Management and, after about 1 minute, verify that there's a frozen index and that you can filter by the various lifecycle phases and statuses.

Next, add the Kibana sample data and attach the full policy to the index that gets created. After about a minute, there should be an error on this index. Click the index, verify that there's ILM information in the detail panel as well as an error, and click Manage > Retry lifecycle step. Make sure this action dismisses the error.

Index and index template actions

• We've already tested the actions for adding a policy for an index. Now remove the policy from the Kibana sample data index.
• Create an index template. In Index Lifecycle Policies, click the "Actions" button next to one of the policies and add the policy to the template, without an alias specified.
• Repeat the above, but with an alias specified.

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[cchaos]

A couple things:

1. Selectors should not be directly in index files. You'll need another file that the index imports.
2. You don't need the styling constants imports when importing the SASS in the plugin .ts file. That's automatically added.

[cjcenizal]

Thanks @cchaos! Changes are in-flight to address your comments.

Selectors should not be directly in index files. You'll need another file that the index imports.

Could you explain the problem this solves?

[cchaos]

Yes, it solves the problem of scaleability of the styles. If more UI components and files are added to this plugin, each would get it's own SASS file and get imported via the index file. Then if any are also removed, it's easy to remove the entire folder and a single import line vs having to comb through a list of selectors to find which ones should be removed (or risk keeping dead code).

[cjcenizal]

Thanks for the context! That is very helpful.

[cjcenizal]

@cchaos I took a closer look at the styles and they weren't needed so I removed them.

[cchaos]

Sounds good, thx

[jloleysens]

Thanks for the detailed walkthrough @cjcenizal ! I encountered a few anomalies as I was testing and left a number of code comments, none of which need to be blocked on.

Also I am really happy to see a lot of code being removed but it is not super clear where most of the deletions originated from - do you have an idea?

Ran through your instructions and was able to perform almost all the actions you described without running into an issue.

Batch delete multiple policies.

Not sure where in the UI I am able to do this? I expected 👇🏻but I was not able to select multiple policies to delete

click Manage > Retry lifecycle step

For this step, the first click resulted in a 400 Bad Request response - subsequent clicks resulted in a 200 that dismissed the error state. Not sure what caused the transitory error state.

I saw the following EUI warning in regarding MinAgeInput:

[jloleysens]

I think it is best to leave explicit annotations for return values, it ensures that in future we don't accidentally return something else from this function.

[jloleysens]

I think we should change this to:

this.config$.pipe(first()).toPromise().then(() => ...)

Then we can make the setup function synchronous. Also, we will probably need a way for the routes to call getConfig so they can request config more lazily.

[cjcenizal]

I synced with @joshdover and you're right that we should be using sync lifecycle methods, but the platform team expects to introduce a sync config API. It will be easier to migrate the code to use it with the way things are currently set up, so he suggested we leave this as-is for now.

[jloleysens]

Great job on thoroughly documenting these!

[cjcenizal]

@jloleysens Thanks for the review!

It is not super clear where most of the deletions originated from

I'm a little hazy on this, but I seem to recall the removed styles were originally an attempt to let the table scroll horizontally within its container div, when the window was made narrow enough to occlude it. I tested this and the styles don't have a discernible effect now:

So I decided to remove them. Without them, the table responds nicely to the narrow window. It's missing labels for the content in each card, so I created #63189 to track this bug.

I discovered that the API tests used the getTemplate API. I have to assume that either this endpoint was created to support this tests or it was written in anticipation of a use case that never materialized, and along the way we ended up using it in the test. These are just guesses. Bill originally built ILM and he said it was probably dead code when I asked him about it.

Batch delete multiple policies.

Great catch! This isn't possible via the UI, but the API is written in a way that implies it's possible. 🤷‍♂️

click Manage > Retry lifecycle step

I will dig into this, thank you for calling it out.

EUI warning emitted from MinAgeInput

I was able to repro this in master so I'm going to exclude it from this PR's scope.

[cjcenizal]

For this step, the first click resulted in a 400 Bad Request response - subsequent clicks resulted in a 200 that dismissed the error state. Not sure what caused the transitory error state.

I was able to reproduce this and the full error message indicates that the index no longer has an ILM error associated with it when you retry the action:

{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "[illegal_argument_exception] cannot retry an action for an index [kibana_sample_data_ecommerce] that has not encountered an error when running a Lifecycle Policy"
}

It seems like the error is transitory. If I repeatedly reload the indices, the error will appear, disappear, repeat. So my guess is that the UI state is just stale when we click the "Retry" button. I was able to reproduce the behavior you mentioned and this transitory behavior in master, so I don't think we need to worry about it in this PR.

[cjcenizal]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 767267e

History

• 💚 Build #39746 succeeded 6fa30e1
• 💚 Build #39033 succeeded 6338e84
• 💚 Build #38238 succeeded bf90bc2
• 💚 Build #38103 succeeded 48eaad8
• 💚 Build #37907 succeeded 184e374b7927f9dbe00f612ea8aad29d6d368611

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream