Distributed-index changes feed requests has high latency as compared with old sync gateway

[ashvindersingh]

The changes feed requests with Distributed index feature takes longer time to finish as compared with older SG releases. We should have comparable response times or better than previous releases.

With Distributed index enabled:

1000 docs w 10 revisions:

• The changes feed returns with results.
• Round trip time to complete changes requests: 12.63 sec, 12.46 secs, 12.7 secs
• Total time to finish test: 263 secs

10 docs w 1000 revisions:

• Got results with changes feed:
• Round trip time to complete changes requests: 1.58 sec, 1.09 sec, 0.5 secs
• Total time to finish test: 179 secs

With Old Sync Gateway:

1000 docs w 10 revisions:

• Got results from changes feed
• Round trip time to complete changes requests: 0.12 secs, 0.12 secs, 0.10 secs
• Total time to finish test: 167.2 secs

10 docs w 1000 revisions:

• Got results from changes feed
• Round trip time to complete changes requests: 0.02 secs, 0.005 secs, 0.007 secs
• Total time to finish test: 164.5 secs

Observations:
With distributed index it takes lot longer to get changes feed.
Surprised to see that changing docs and rev count did not make much difference in old SG.

[adamcfraser]

Based on discussion with Ashvinder, this was based on a test where Sync Gateway and the index bucket weren't colocated (SG in MV, index bucket in Santa Clara). That appears to be the reason the test is taking 150+ seconds in both cases. I expect that would be a contributing factor to the slower changes response, but it still shouldn't be taking 10+ seconds to get the results.

[zgramana]

Want to document the requirement that index service nodes run very closely (in hops) to the CBS nodes that they are indexing.

[adamcfraser]

Lowering priority on this - not critical for 1.2. @ashvindersingh - let me know what you find when you retest.

[ashvindersingh]

Here are the results after retest.

Significant improvement as compared to previous run! though channel cache mode is faster than DI mode.

Tested with sg build: 1.2.0-53.
Test script: https://github.com/couchbaselabs/sync-gateway-testcluster/blob/master/functional_tests/test_single_user_single_channel_doc_updates.py

With Distributed index enabled:

1000 docs w 10 revisions:

The changes feed returns with results.
Round trip time to complete changes requests: 442, 478, 487 (all in ms)
Total time to finish test: 213 secs

10 docs w 1000 revisions:

Got results with changes feed:
Round trip time to complete changes requests: 340, 398, 328 (ms)
Total time to finish test: 223 secs

With channel cache mode:

1000 docs w 10 revisions:

Got results from changes feed
Round trip time to complete changes requests: 111, 88, 175 (ms)
Total time to finish test: 204 secs

10 docs w 1000 revisions:

Got results from changes feed
Round trip time to complete changes requests: 40, 53, 35 (ms)
Total time to finish test: 203 secs

[adamcfraser]

Closing, as the difference isn't significant enough to warrant particular attention.