SessionTokenMismatchRetryPolicy optimization through customer supplied region switch hints

[jeet1995]

Background

There have been customer reported incidents, where the customer has provided diagnostics wherein there is a storm of 404 / 1002 (NOT_FOUND / READ_SESSION_NOT_AVAILABLE) errors returned by the service.

Upon investigation of our SDK-internal retry policies, whenever the above error occurs, the SDK sends the request to each replica for the physical partition first for a specific region and this cycle of sending a request to each replica of the physical partition for the same region repeats courtesy the SessionTokenMismatchRetryPolicy in an exponential backoff manner over a certain time period. Only after the retry window elapses on the SessionTokenMismatchRetryPolicy, is the request retried on a different region courtesy the ClientRetryPolicy (specific to read requests).

This leads to the same replica being sent requests to multiple times which can cause additional latency before of leveraging a different region.

Why do READ_SESSION_NOT_AVAILABLE errors occur?

• The session token in the request has a higher GlobalLSN value or a higher LocalLSN value (specific to multi-write accounts) than that of the session token in the response. This points to either a lagging region or a lagging replica.
• A loss of quorum between replicas due to network partition between them.

Possible workarounds

The ask now becomes, to see if different regions can be leveraged quicker or if it is necessary to cycle through all replicas first in a given region.

What this PR adds

This PR will allow application developers to configure hints through a SessionRetryOptions instance which will signal to the SDK whether to pin retries on the local region or move quicker to a remote region especially when READ_SESSION_NOT_AVAILABLE errors are thrown.

Public API surface changes

1. First build the SessionRetryOptions instance.

// if local region retries are prioritzed
SessionRetryOptions sessionRetryOptions = new SessionRetryOptionsBuilder()
                .setRegionSwitchHint(CosmosRegionSwitchHint.LOCAL_REGION_PREFERRED)
                .build();

// if remote region retries are prioritized
SessionRetryOptions sessionRetryOptions = new SessionRetryOptionsBuilder()
                .setRegionSwitchHint(CosmosRegionSwitchHint.REMOTE_REGION_PREFERRED)
                .build();

2. Set the built SessionRetryOptions instance on the CosmosClientBuilder instance.

CosmosAsyncClient clientWithPreferredRegions = new CosmosClientBuilder()
    .endpoint("<account URL goes here>")
    .key("<account key goes here>")
    .sessionRetryOptions(sessionRetryOptions)
    .directMode()
    .buildAsyncClient();

System config provided

1. In order to control the no. of retries in the local region when REMOTE_REGION_PREFERRED is set as the region switch hint, set the following JVM config as below:

int maxRetriesInLocalRegion= 5;
System.setProperty("COSMOS.MAX_RETRIES_IN_LOCAL_REGION_WHEN_REMOTE_REGION_PREFERRED", String.valueOf(maxRetriesInLocalRegion));

Reasoning about region switch hints

Account type	Operation type	First request region	Hint	Impact when READ_SESSION_NOT_AVAILABLE errors occur
Multi-write	Read	Read region	LOCAL_REGION_PREFERRED	It may not be beneficial to stick to local region (read region) since some write region will have more updated data.
Multi-write	Read	Read region	REMOTE_REGION_PREFERRED	It will be beneficial to switch to a write region since a write region will have updated data.
Multi-write	Read	Write region	LOCAL_REGION_PREFERRED	It may not be beneficial to stick to the local write region.
Multi-write	Read	Write region	REMOTE_REGION_PREFERRED	It maybe beneficial to switch to a remote write region which has more updated data.
Multi-write	Write	Write region	LOCAL_REGION_PREFERRED	It may not be beneficial to stick to the local write region.
Multi-write	Write	Write region	REMOTE_REGION_PREFERRED	It maybe beneficial to switch to a different write region.
Single-write	Read	Read region	LOCAL_REGION_PREFERRED	It won't be beneficial to stick to local region (read region) since the write region will have updated data.
Single-write	Read	Read region	REMOTE_REGION_PREFERRED	It will be beneficial to switch to the write region since a write region will have updated data.
Single-write	Read	Write region	LOCAL_REGION_PREFERRED	No impact since a read request to a write region will be retried again on the same region after region switch.
Single-write	Read	Write region	REMOTE_REGION_PREFERRED	No impact since a read request to a write region will be retried again on the same region after region switch.
Single-write	Write	Write region	N/A	No impact since a write request to a single-write region account will never face READ_SESSION_NOT_AVAILABLE errors.

[azure-sdk]

API change check

APIView has identified API level changes in this PR and created following API reviews.

azure-cosmos

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[xinlian12]

LGTM, thanks

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[kushagraThapar]

LGTM, thanks @jeet1995