[BUG] Non remote indices can get allocated to nodes with search role and no data role

[dgilling]

Describe the bug
A clear and concise description of what the bug is.
Using OS 2.4.1 with searchable snapshots enabled, if a cluster has a set of nodes with data role and a set of nodes with search role, non remote indices can get allocated to the search nodes.

To Reproduce
Steps to reproduce the behavior:

1. Create cluster containing both search and data nodes (i.e one has node.roles=search and one has node.roles=data)
2. Have no other allocation filters other than defaults
3. Restore both local and remote indices initially to the respective nodes
4. Reallocation will gradually rebalance non remote indices from the data nodes to the search role.

Expected behavior
A clear and concise description of what you expected to happen.
If a node only has the role search and does not also contain data, only remote indices should allocate to it.
While the search node can service the non remote indices fine, it's not well documented or expected this occurs. A user may have defined lower memory/hardware requirements for search nodes.

** As a workaround, one can set a custom attribute on search nodes and use that for index level allocation.**

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS] Ubuntu
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[kotwanikunal]

Hey @dgilling! This issue was fixed with #6689 and backported to 2.x with #6718.
Please reopen if you still come across the issue post these fixes.