_changes with doc_ids filter does not work with feed=longpoll

[snej]

If you send a changes request with both filter=_doc_ids and feed=longpoll, the filter is ignored and all changed documents are returned.

This will break client continuous pull of a set of docIDs, because the replicator will end up pulling every doc int the database, not just the ones requested. (We could add a workaround to Couchbase Lite and ignore the returned docIDs that don't match, but this would still be a waste of bandwidth.)

[adamcfraser]

The current implementation of the doc_ids filter doesn't support continuous/longpoll replication - it's strictly for one-shot replication. The missing piece is a document-level notification mechanism - currently our changes notification process only triggers for channel or user/role changes. It might be possible to add doc-level notification tags, but we'd need to evaluate the performance impact associated with that.

To test this out, we'd need to:

• When a changes feed with a doc id filter, register a wait listener with keys like _d/[doc_id] (prefix needed to ensure uniqueness between doc ids and channel names
• During channel index processing, send notification for each document seen on the feed, in the same format

A bigger challenge would be supporting this for sg-accel - since each node isn't listening to the full feed, clients would have to poll for updates to the list of doc ids, which has the potential for significant performance overhead when many active changes feeds are listening for docs.

[snej]

What about just having the change handler look up every incoming change's docID against the set being filtered on, and ignore it if it's not in the set? Clearly this has scalability problems, but it at least produces correct results, so I think it's better than what's there now.

In any case, until this is fixed, I think the REST handler should return an error with this combination of options. It would be better for the client to get an error than to start getting updates for docs it didn't want to sync.

[adamcfraser]

I agree about returning an error for this combination of options - filed #1712 for that change.

We can use this ticket to track non-one-shot w/ doc_id filtering. I think we need to come up with a solution that scales - at least something that's compatible with sg-accel - before we deliver it.

[adamcfraser]

Moved to CBG-793