Changes feed for * channel missing entries

[adamcfraser]

While testing distributed index, QE identified cases where the number of entries returned by a _changes request didn't match the expected count (and also differed from the all_docs results).

API results:
_all_docs (1000 docs): https://gist.github.com/adamcfraser/d51690b9d53080f13f2a
_changes (998 docs): https://gist.github.com/adamcfraser/9a9c299eb83d8738c615

[adamcfraser]

Comparing the two, the missing entries are:
36a31201-bc03-43e7-afe6-edcd29b5c4a0 (vbucket 313)
422f91d7-2e2c-4fab-8cac-a843e3cf94b5 (vbucket 24)

The index entry docs (_idx_entry:313:1 and _idx_entry:24:1) are present in the index bucket, with the correct data.

However, the channel clock for the * channel doesn't include entries for vbuckets 313 and 24 - this would be the root of the problem. (https://gist.github.com/adamcfraser/717f1dd74422f1f046d8)

Need to determine why the channel clock didn't get updated. There aren't any write errors appearing in the sync gateway logs for the test.

[adamcfraser]

Based on followup run with additional diagnostics, there's no write errors happening during clock processing.

A few additional bits of information from the new test:

1. Ran with 10000 docs, 9 were missing from _changes. _idx_entry docs were present for all - but weren't present in the * channel doc.
2. Missing docs were always highest docs in the vbucket. e.g. [567:9] was missing, channel clock showed [567:8]. No cases where [567:2] was missing when the channel clock showed [567:8]. Suggests that block writes are fine, and it's primarily a clock issue.
3. Some missing docs appeared in the ABC channel clock, but not the * channel clock (e.g. 518354dd-16a5-43de-9084-63a739789d8e, which is vb 381 seq 12. * channel clock has [381:11], ABC channel clock has [381:12], stable clock has [381:12]

[adamcfraser]

Missing entries are also present in the index blocks - looks like strictly a clock issue.

[adamcfraser]

Root cause is partitioning of channel updates by a single writer. The write logic was still assuming partitioned channel clocks, even though we've deferred that for now to improve read performance.

Fix is to move channel clock write up to the change index, to ensure it's done once per batch (instead of once per partition per batch).