Using time out in cluster state observer as we are reusing the observer

[gbbafna]

Description

This makes waitForNextChange wait till time out value every time it is called. Without this change, the cluster state observer doesn't update cso.startTimeMS . So it waits for total timeout across multiple calls . For ex : 60 sec on first time and after that since cso.startTimeMS is not updated , the waitForNextChange returns immediately . This results in unnecessary CPU cycles and log flood as well.

Issues Resolved

#207

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[gbbafna]

Logs after fix

[2021-10-27T10:00:05,559][INFO ][o.o.r.t.i.IndexReplicationTask] [followCluster-0] [grab] In restoring state
[2021-10-27T10:00:05,569][DEBUG][o.o.c.c.PublicationTransportHandler] [followCluster-0] received diff cluster state version [10] with uuid [wGnixzedSougu_Atbb-PSQ], diff size [598]
[2021-10-27T10:00:05,789][DEBUG][o.o.c.c.C.CoordinatorPublication] [followCluster-0] publication ended successfully: Publication{term=1, version=10}
[2021-10-27T10:01:05,800][INFO ][o.o.r.t.i.IndexReplicationTask] [followCluster-0] [grab] Timed out while waiting for restore to complete.
[2021-10-27T10:02:05,803][INFO ][o.o.r.t.i.IndexReplicationTask] [followCluster-0] [grab] Timed out while waiting for restore to complete.
[2021-10-27T10:03:05,807][INFO ][o.o.r.t.i.IndexReplicationTask] [followCluster-0] [grab] Timed out while waiting for restore to complete.
[2021-10-27T10:04:05,811][INFO ][o.o.r.t.i.IndexReplicationTask] [followCluster-0] [grab] Timed out while waiting for restore to complete.

Logs before fix

[2021-10-13T07:49:33,977][INFO ][c.a.e.r.t.i.IndexReplicationTask] [a133cc96509ad479b0161b22d54d2199] [abcd] In restoring state

[2021-10-13T07:50:33,170][INFO ][c.a.e.r.t.i.IndexReplicationTask] [a133cc96509ad479b0161b22d54d2199] [abcd] Timed out while waiting for restore to complete.

[2021-10-13T07:50:33,170][INFO ][c.a.e.r.t.i.IndexReplicationTask] [a133cc96509ad479b0161b22d54d2199] [abcd] Timed out while waiting for restore to complete.
[2021-10-13T07:50:33,170][INFO ][c.a.e.r.t.i.IndexReplicationTask] [a133cc96509ad479b0161b22d54d2199] [abcd] Timed out while waiting for restore to complete.
[2021-10-13T07:50:33,170][INFO ][c.a.e.r.t.i.IndexReplicationTask] [a133cc96509ad479b0161b22d54d2199] [abcd] Timed out while waiting for restore to complete.
[2021-10-13T07:50:33,171][INFO ][c.a.e.r.t.i.IndexReplicationTask] [a133cc96509ad479b0161b22d54d2199] [abcd] Timed out while waiting for restore to complete.
[2021-10-13T07:50:33,171][INFO ][c.a.e.r.t.i.IndexReplicationTask] [a133cc96509ad479b0161b22d54d2199] [abcd] Timed out while waiting for restore to complete.
[2021-10-13T07:50:33,171][INFO ][c.a.e.r.t.i.IndexReplicationTask] [a133cc96509ad479b0161b22d54d2199] [abcd] Timed out while waiting for restore to complete.

[saikaranam-amazon]

If the timeout is not set, Is the default value under cluster service not taken into account?

[gbbafna]

it is taken, but it is measured since startTimeMS which is initialized only once across multiple waitForNextChange.

[saikaranam-amazon]

The global object cso object in index replication task seems to be creating the issue. Can we check the previous cluster state tracker in this observer?

[ankitkala]

Correct fix here would be to set startTimeMS and timeoutTimeLeftMS irrespective of whether the timeOutValue is null or not. That'll require changes in OS repo though.

[gbbafna]

yes @ankitkala , but that will change behavior for other use cases and may break their use case as well.

@saikaranam-amazon : not sure i understand your point. will sync up offline and update it here.

[saikaranam-amazon]

As discussed, Concern was around the reference to the previous cluster state. Based on the logic from observer code, it is updating the latest state.

[ankitkala]

LGTM.