[8.0 only] [eventLog] make newly created and existing eventLog indices, hidden indices

[pmuellr]

In elasticsearch >= 7.7, a new feature will be available to make indices "hidden". To some extent, it's just a formalism of the "dot-prefix" convention we use for Kibana-internal indices.

For more info on hidden indices, see: PR elastic/elasticsearch#50452

We should probably make the eventLog indices be hidden. Especially since we don't know if we could "change" eventLog from a "normal" index to a "hidden" index, if we first ship eventLog that writes to normal indices but want to switch later to hidden.

I don't think this needs to happen until we switch on eventLog writing to indices as default true - currently it's default false. No ES resources will be touched if it's false. Current thinking is that we'll switch to default true after the 7.7 freeze, or shortly thereafter, to get a long burn-in on it's use during 7.8 development.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[mikecote]

Some examples how ML implemented the change:

• [ML] Make ml internal indices hidden ([ML] Make ml internal indices hidden elasticsearch#52423)
• Make ML index aliases hidden (Make ML index aliases hidden elasticsearch#53160)

[mikecote]

Moving from 7.x - Candidates to 7.13 after the latest 7.x planning session.

[mikecote]

See #93515. We may have to wait for 8.0, or we may be able to push to 8.x.

[mikecote]

Based on the comments (#93515 (comment)), it seems best to work on this issue in 8.0 so I will move it to 7.15/8.0 - Tentative.

[gmmorris]

Given that we have not been able to prioritise work for an event Log UI prior to 8.0, should we consider keeping these as hidden indices rather than system indices?
I'm worries about power users not being able to debug issues using the event log.

Perhaps we can add some kind of deprecation notice on these indices that will allow us to switch them to system indices in 8.x?

[pmuellr]

I don't think we want to change the event log to a system index - they are more constrained than hidden indices. And I'm worried, like Gidi, that since we're now using these indices for diagnostic purposes with users, we'll need to make sure they can still use these in ways we'd like them to be used. For example, you should be able to create a Kibana index pattern for the event log, and then create a Lens visualization using that index pattern. Make sure ILM still works, and what-not ...

So, we'll likely need to do a little experimenting here, and ping ES folks more about hidden index caveats, etc.

I still don't know of any reason why this would need to be done prior to 8.0.

[mikecote]

Sounds like we'll be able to defer this issue for higher priority items. Thanks for the input!

[mikecote]

I modified the issue's title. After talking with @kobelb, we will go with the approach of creating new indices as hidden in 8.0 or higher and leave existing indices (created in 7.x) as is.

I believe the same should apply for aliases. Since they are created per-version, we should be ok to mark 8.0+ aliases as hidden as well.

[mikecote]

Reverting title back to new + existing indices after further discussion with @kobelb and @stacey-gammon for consistency. There isn't a strong push to migrate existing indices in 8.0, but it sounded fairly easy (single PUT HTTP request, probably in migration assistant) and might as well if it's easy.

[ymao1]

For a brand new event log index, we are able to set index.hidden index setting and set the is_hidden attribute on the index alias on index creation. By doing both of these, the index is properly hidden, and does not show up when you perform _cat/indices.

For existing event log indices, this is less straightforward. According to the docs, only the dynamic index settings can be updated after an index has been created. index.hidden is a static index setting which can only be set at index creation or on a closed index. We are able to update the index.hidden setting on the index template and the is_hidden setting on the index alias but updating these settings does not apply to existing indices. We would have to reindex to get these new settings applied. Given that the event log indices can potentially be very large and they age off after 90 days, do we feel like it's worth it to add something to upgrade assistant to update the settings and reindex?

[mikecote]

do we feel like it's worth it to add something to upgrade assistant to update the settings and reindex?

@kobelb, from #58035 (comment), I recall our prior conversation you mentioned we can "migrate" the existing event log indices to hidden via a single PUT HTTP request (via the upgrade assistant?). Do you have an example of how to do so?

[kobelb]

Very interesting... I did not read the documentation as closely as you did @ymao1, and I was able to use the ES APIs against 7.14 to change indices from non-hidden to hidden:

PUT .foo_1
{
  "mappings": {
    "properties": {
      "message": {
        "type": "keyword"
      }
    }
  }
}

POST .foo_1/_doc
{
  "message": "hello world"
}

PUT .foo_*/_settings
{
  "index.hidden": true
}

GET .foo_1/_settings

## Results
## 
## {
##   ".foo_1" : {
##     "settings" : {
##       "index" : {
##         "routing" : {
##           "allocation" : {
##             "include" : {
##               "_tier_preference" : "data_content"
##             }
##           }
##         },
##         "hidden" : "true",
##         "number_of_shards" : "1",
##         "provided_name" : ".foo_1",
##         "creation_date" : "1630526221853",
##         "number_of_replicas" : "1",
##         "uuid" : "F81M2mrmTxuVbYqUJ_Z6HQ",
##         "version" : {
##           "created" : "7140099"
##         }
##       }
##     }
##   }
## }

@elastic/es-data-management can you all confirm whether we should be able to update existing indices to be hidden?

[ymao1]

@kobelb That is interesting! I was using the ES Client through Kibana to update the settings and unable to have the updates show up, which is why I dug into the docs. It obviously works though, so I must have missed something. Thanks!

[ymao1]

Yeah, I wonder if it is different via the ES Client.

I'm using

await esClient.indices.putSettings({
	index: indexName,
	body: {
		settings: {
			...currentIndexSettings.settings,
			index: {
				...currentIndexSettings.settings.index,
				hidden: true,
			}
		}
	}
});

which sends

{
	"index": {
		"lifecycle": {
			"name": "kibana-event-log-policy",
			"rollover_alias": ".kibana-event-log-7.15.0"
		},
		"routing": {
			"allocation": {
				"include": {
					"_tier_preference": "data_content"
				}
			}
		},
		"number_of_shards": "1",
		"auto_expand_replicas": "0-1",
		"provided_name": ".kibana-event-log-7.15.0-000001",
		"creation_date": "1630439186791",
		"number_of_replicas": "0",
		"uuid": "Ure4d9edQbCMtcmyy0ObrA",
		"version": {
			"created": "7150099"
		},
		"hidden": true
	}
}

and the changes are not reflected in the index settings. I did try it via Dev Tools using the same command you did and that did work though, although the index still shows up when you do a _cat/indices

[jakelandis]

@elastic/es-data-management can you all confirm whether we should be able to update existing indices to be hidden?

@williamrandolph / @gwbrown - Can you assist with this question ?

[gwbrown]

The index.hidden setting is dynamic and you should be able to set it via API. It started out as a static setting, which explains the incorrect docs (which I'll fix very soon), but it was changed to be dynamic in elastic/elasticsearch#52772 (in 7.7.0).

I just tried it quickly (in a BC of 7.15.0, as that's what I had handy), and updating index.hidden from to true from the default on a test index worked for me - the new value was visible when retrieving the settings for the index and it behaved as expected (i.e. doesn't show up without expand_wildcards=all) in GET _cat/indices and GET _all.

[gwbrown]

@ymao1 I'm pretty sure I know what's wrong: If that's the body of the settings update request, that request will error out 100% of the time. A number of settings there are static and cannot be updated via API, such as index.creation_date.

When I try to issue that command against a test cluster, I get a response like this:

{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "unknown setting [index.creation_date] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
            }
        ],
        "type": "illegal_argument_exception",
        "reason": "unknown setting [index.creation_date] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
        "suppressed": [
            {
                "type": "illegal_argument_exception",
                "reason": "unknown setting [index.provided_name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
            },
            {
                "type": "illegal_argument_exception",
                "reason": "unknown setting [index.uuid] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
            },
            {
                "type": "illegal_argument_exception",
                "reason": "can not update private setting [index.version.created]; this setting is managed by Elasticsearch"
            }
        ]
    },
    "status": 400
}

When the update settings request errors out like this, none of the settings will be applied.

Fortunately, I think this should be easy to solve: Just don't put the existing settings into the settings update request. PUT my_index/_settings will only update the settings given, so:

PUT my_index/_settings
{
  "index": {
    "hidden": true
  }
}

Is all that's required to update index.hidden on my_index, none of the other index settings will be changed.

[ymao1]

@gwbrown That makes so much sense. I was trying to make sure I didn't overwrite any existing settings but the API is smarter than that. Thank you!

[gwbrown]

Following up, I've opened a PR to correct the docs for index.hidden at elastic/elasticsearch#77218.