[BUG] Shrink action can't work properly

[Angie-Zhang]

What is the bug?
When we use shrink action after rollover action, or apply shrink action on data stream's index, the shrink action can't be executed correctly. It throws:

Shrink failed when sending shrink request.
<some index> must be read-only to resize index. use "index.blocks.write=true"

After several retries, the cluster has no available node to execute shrink action, all indexes for shrink have:

{
    "message": "There are no available nodes to move to to execute a shrink. Delaying until node becomes available."
}

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Created an ISM policy with shrink action after rollover, e.g.

{
    "policy": {
        "policy_id": "rollover_shrink_test",
        "description": "Rollover after 3 min then shrink and delete after 1 day",
        "error_notification": null,
        "default_state": "hot",
        "states": [
            {
                "name": "hot",
                "actions": [
                    {
                        "rollover": {
                            "min_doc_count": 1
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "search"
                    }
                ]
            },
            {
                "name": "search",
                "actions": [
                    {
                        "shrink": {
                            "num_new_shards": 1,
                            "aliases": [
                                {
                                    "rollover-shrink-target": {
                                        "is_write_index": false
                                    }
                                }
                            ],
                            "force_unsafe": true
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "1d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "rollover-shrink-test*"
                ],
                "priority": 100,
            }
        ]
    }
}

2. Shrink action fails after rollover, keep getting error message:

{
    "message": "There are no available nodes to move to execute a shrink. Delaying until node becomes available."
}

Even though there are available nodes for shrink, ISM still throws above error message.

What is the expected behavior?
Shrink action should work as expected.

What is your host/environment?

• OS: AWS & MacOS Monterey
• Version: 2.3 in AWS; 2.5.0 docker image in local MacOS Monterey.
• Plugins: Index Management

[getsaurabh02]

@Angie-Zhang Can we also share the setup used while replicating this issue. Wondering if this issue replicable on any specific setup or configuration of indices/cluster.
Also, do we have any insights on the possible reasons for it yet?

[petardz]

I made IT with this configuration and everything is working fine.

Can you please paste here datastream info:

1. Response from /my_data_stream_name
2. index template for this datastream