[Segment Replication] Support realtime TermVector requests with Segment Replication

[Rishikesh1159]

Description

This PR supports realtime reads for TermVectors requests when segment replication is enabled.
If there are no preferences mentioned for the incoming TermVectors requests then we route it to primary shards

Related Issues

Resolves #9563

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/23631/
• CommitID: 4012c16

[codecov]

Codecov Report

Merging #9585 (4e43c57) into main (4ccecd6) will decrease coverage by 0.07%.
Report is 5 commits behind head on main.
The diff coverage is 26.13%.

@@             Coverage Diff              @@
##               main    #9585      +/-   ##
============================================
- Coverage     71.16%   71.10%   -0.07%     
- Complexity    58115    58125      +10     
============================================
  Files          4831     4831              
  Lines        273999   274062      +63     
  Branches      39920    39925       +5     
============================================
- Hits         195005   194880     -125     
- Misses        62604    62890     +286     
+ Partials      16390    16292      -98     

Files Changed	Coverage Δ
...arch/index/recovery/RemoteStoreRestoreService.java	12.03% <0.00%> (ø)
server/src/main/java/org/opensearch/node/Node.java	85.54% <0.00%> (ø)
...arch/gateway/remote/RemoteClusterStateService.java	65.37% <23.37%> (-13.05%)	⬇️
...action/termvectors/TransportTermVectorsAction.java	51.35% <33.33%> (-3.20%)	⬇️
...search/gateway/remote/ClusterMetadataManifest.java	96.95% <100.00%> (+0.03%)	⬆️

... and 468 files with indirect coverage changes

📢 Have feedback on the report? Share it here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 6f429b6

Incompatible components

Skipped components

Compatible components

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/23698/
• CommitID: 6f429b6
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25033/
• CommitID: 89a4588
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25043/
• CommitID: 2648fd4
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Compatibility status:

Checks if related components are compatible with change 2648fd4

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25062/
• CommitID: 2648fd4
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/25070/
• CommitID: 2648fd4

[github-actions]

Compatibility status:

Checks if related components are compatible with change 4e43c57

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/25135/
• CommitID: 4e43c57

[andrross]

@Rishikesh1159 I don't think the primary override here is necessary. In this case, an "artificial" document has been supplied and the user is asking to compute term vectors as if this was a real document. I don't think the "real time" parameter is applicable here since it doesn't need to look up any index data.

@msfroh Does that sound right to you?

[msfroh]

That does sound right to me.

I think the override condition should be:

        if (request.request().realtime()
            && preference == null
            && state.getMetadata().isSegmentReplicationEnabled(request.concreteIndex())
            && request.request().doc() == null) {

Or you could revert the edit to this line and continue to use request.request().preference() for the artificial doc case.

[andrross]

Interestingly the code seems to unconditionally attempt to fetch the document even if an artificial document is supplied. Not sure if that complicates things here.

[msfroh]

I think that's probably worth fixing -- from the looks of it, that GetResult is ignored, which seems a little wasteful.

That said, I don't think it hurts anything. In particular, if the doc parameter is specified, then it GETs a document with ID based on an autoincrementing (not really random) integer:

OpenSearch/server/src/main/java/org/opensearch/action/termvectors/TermVectorsRequest.java

Lines 315 to 323 in aca2e9d

	public TermVectorsRequest doc(BytesReference doc, boolean generateRandomId, MediaType mediaType) {
	// assign a random id to this artificial document, for routing
	if (generateRandomId) {
	this.id(String.valueOf(randomInt.getAndAdd(1)));
	}
	this.doc = doc;
	this.mediaType = mediaType;
	return this;
	}

(generateRandomId is always true on the REST parsing code path.)

[msfroh]

Still, there is no value in routing to a primary shard if an artificial doc is specified.