Compatibility with segment replication

[dreamer-89]

Summary

With 2.9.0 release, there are lot of enhancements going in for segment replication[1][2] feature (went GA in 2.7.0), we need to ensure different plugins are compatible with current state of this feature. Previously, we ran tests on plugin repos to verify this compatibility but want plugin owners to be aware of these changes so that required updates (if any) can be made. With 2.10.0 release, remote store feature is going GA which internally uses SEGMENT replication strategy only i.e. it enforces all indices to use SEGMENT replication strategy. So, it is important to validate plugins are compatible with segment replication feature.

What changed

1. Refresh policy behavior

1. RefreshPolicy.IMMEDIATE will only refresh primary shards but not replica shards immediately. Instead post refresh, primary will start a round of segment replication to update the replica shard copies leading to eventual consistency.
2. RefreshPolicy.WAIT_UNTIL ensures the indexing operation is searchable in your cluster i.e. RAW (Read after write guarantee). With segment replication, this guarantee is not promised due to delay in replica shared updates from asynchronous background refreshes.

2. Refresh lag on replicas

With segment replication, there is inherent delay in documents to be searchable on replica shard copies. This is due to the fact that replica shard copies over data (segment) files from primary. Thus, compared to document replication, there will be on average increase in amount of time the replica shards are consistent with primaries.

3. System/hidden indices support

With opensearch-project/OpenSearch#8200, system and hidden indices are now supported with SEGMENT replication strategy. We need to ensure there are no bottlenecks which prevents system/hidden indices with segment replication.

Next steps

With segment replication strong reads are not guaranteed. Thus, if the plugin needs strong reads guarantees specially as alternative to change in behavior of refresh policy and lag on replicas (point 1 and 2 above), we need to update search requests to target primary shard only. With opensearch-project/OpenSearch#7375, core now supports primary shards only based search. Please follow documentation for examples and details

Open questions

In case of any questions or issues, please post it in core issue

Reference

[1] Design
[2] Documentation

[dreamer-89]

Request owners to add v2.9.0 label on this issue.

[macohen]

this plugin should not require any changes for this case. @sejli can you please verify?

[sejli]

dashboards-search-relevance currently uses calls to /_cat/indices to populate an index dropdown in the Search Comparison Tool. We also have calls to _search which uses indices from the dropdown. I believe these do not have any interaction with segment replication, @dreamer-89 could you confirm?

Taking a look at other dashboards plugins like ml-commons-dashboards, we only need to check if there is interaction with indices.

[dreamer-89]

@sejli : If you are using _search API calls and need read after write consistency, you may want to redirect search to primary shards as mentinoed in documentation

[sejli]

This plugin currently does not do any writing to OpenSearch. I believe the only time there may potentially be reads after writes is when a user has DevTools open and writes to an index, and then immediately opens up the Search Comparison Tool to search the above index. The Search Comparison Tool simply runs a DSL query and lists the documents returned in a row format. Discussed with @macohen offline, I don't think there is a need for read after write consistency for this use case, and I don't think that the segment replication change is breaking for the Search Comparison Tool... Is the delay in which the replicas are consistent with the primaries long enough to justify redirecting search to primary shards? Would love to hear some other opinions if any @noCharger @mingshl

[sejli]

Discussed with other maintainers, we've decided to not move forward with a code change since the segment replication feature updates are not breaking. While the aforementioned situation may create read after write consistency, it's also possible that if a user was doing some heavy searches on the Search Comparison Tool with all searches directed to primaries, it may be possible that it puts too big of a load on the primaries themselves. We'll keep this in consideration for feature updates. Closing this issue... Thanks!