Integrity checks for read-only indices

[jpountz]

The checksum verification that we perform on merge or relocation has proved very useful to detect corruption but it is only triggered on merge. So static indices or segments that already reached the maximum segment size are never checked, even though they are as likely to get corrupted.

It would be nice to have an API, a command-line tool or even to figure out a way to run integrity checks automatically (anything that is easier than running CheckIndex on the shard directories)?

[clintongormley]

+1 - should probably integrate with the task management api

[martijnvg]

When discussing whether the experimental tag should be removed from index.shard.check_on_startup index setting in #19798, we came to the conclusion that performing a checkindex when a shard is starting is maybe not the best time. It can slow down recovery significantly.

Instead like this issue suggested an api is better, because there is full control when a check index is performed. The api should mimic the checkindex tool as close as possible. Exposing its verbose, fast, segment and crossCheckTermVectors options:

POST /{index}/_check_index?verbose=true|false&fast=true|false&segement=[segment_id]&crossCheckTermVectors=true|false

The exorcise option should not be exposed as it modifies the index by removing segments that have issues.

The check index api should be executed on a closed index. Making this api work on open indices is tricky, because the underlying Lucene index can change while check index is running. However I think we can make this api work on read only indices.

The check index api should replace the index.shard.check_on_startup index setting.

@s1monw If I remember correctly you prefer a command line utility instead of an api for things like check index. However I think in this case check index I think an api is preferred? The current proposal doesn't modify shards and checking the integrity of an index via an api can be useful for curator like tools. Do you agree?

[elasticmachine]

Pinging @elastic/es-core-infra

[s1monw]

@s1monw If I remember correctly you prefer a command line utility instead of an api for things like check index. However I think in this case check index I think an api is preferred? The current proposal doesn't modify shards and checking the integrity of an index via an api can be useful for curator like tools. Do you agree?

I was suggesting a disk utility if you repair the index by dropping corrupted segments. I think for your use-case and API is just fine.

[dakrone]

This has been open for quite a while, and hasn't had a lot of interest. For now I'm going to close this as something we aren't planning on implementing. We can re-open it later if needed.