Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] [Remote Store] Wildcards and regex are not inferred by _remotestore/_restore API #8480

Closed
BhumikaSaini-Amazon opened this issue Jul 6, 2023 · 0 comments · Fixed by #8922
Assignees
Labels
bug Something isn't working enhancement Enhancement or improvement to existing feature or request Storage Issues and PRs relating to data and metadata storage

Comments

@BhumikaSaini-Amazon
Copy link
Contributor

Describe the bug
Restoring from remote store should offer functionality similar to restoring from a snapshot. But passing a wildcard index name pattern to the remote store restore API fails to restore indices matching the pattern. Other missing functionalities include:

  1. Restoring all remote-enabled indices by default
  2. Excluding selected indices from getting restored
  3. Further enhancements based on https://opensearch.org/docs/1.2/opensearch/snapshot-restore/#restore-snapshots

To Reproduce
Steps to reproduce the behavior:

  1. Create multiple remote store-enabled indices matching a pattern - for example remote-store-test-idx-1,remote-store-test-idx-2,my-index-1,my-index-2. A set of wildcards that can be used to refer to these indices would be: remote-store-test-idx-* => (remote-store-test-idx-1, remote-store-test-idx-2), my-index-* => (my-index-1, my-index-2).
  2. Index some data in all indices.
  3. Turn the indices red by terminating the nodes housing the primary/replica shards.
  4. Bring up new nodes to house the primary/replica shards, unless there are other nodes remaining.
  5. Close the red indices.
  6. Trigger restore from remote store for the red indices by using a wildcard pattern. For example, [remote-store-test-idx-*,my-index-*]
  7. The following error shows up, failing the restore for the matching indices:
[WARN ][o.o.s.RestoreService     ] [node_t0] Remote store restore is not supported for non-existent index. Skipping: remote-store-test-idx-*

Expected behavior
There should be support for wildcard and regex inference for a better user experience. To work around this, currently the user would need to explicitly spell out the full index names, which could be cumbersome if count of indices is too high.

Additional context

  1. Remote store restore API: https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/remote-store/index/#restoring-from-a-backup
  2. Snapshot restore API: https://opensearch.org/docs/1.2/opensearch/snapshot-restore/#restore-snapshots:~:text=Description-,indices,-The%20indices%20you%20want%20to%20restore
@BhumikaSaini-Amazon BhumikaSaini-Amazon added bug Something isn't working untriaged labels Jul 6, 2023
@anasalkouz anasalkouz added enhancement Enhancement or improvement to existing feature or request and removed untriaged labels Jul 18, 2023
@Bukhtawar Bukhtawar added the Storage Issues and PRs relating to data and metadata storage label Jul 27, 2023
@linuxpi linuxpi moved this from 🆕 New to 👀 In review in Storage Project Board Mar 5, 2024
@rramachand21 rramachand21 added the v2.13.0 Issues and PRs related to version 2.13.0 label Mar 8, 2024
@gbbafna gbbafna removed the v2.13.0 Issues and PRs related to version 2.13.0 label Apr 4, 2024
@github-project-automation github-project-automation bot moved this from 👀 In review to ✅ Done in Storage Project Board May 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement Enhancement or improvement to existing feature or request Storage Issues and PRs relating to data and metadata storage
Projects
Status: ✅ Done
Development

Successfully merging a pull request may close this issue.

8 participants
@linuxpi @Bukhtawar @anasalkouz @rramachand21 @gbbafna @minalsha @BhumikaSaini-Amazon and others