_all_docs still slow

[jnordberg]

Refs #326

On a bucket with 1 742 612 items fetching _all_docs logged in as a user who has 1 channel with 10 documents in it takes more than 60 seconds (our LB cuts the request off at that point so I'm unable to see the response, if there is any).

Requesting _changes?since=0 as the same user on the same bucket only takes ~20ms.

[jnordberg]

Using Couchbase Sync Gateway/1.0.2-9(commit 95ef60ab)

[tleyden]

@lichenyang2 assigning to you for now -- are there any performance tests that cover this?

[amazkovoi]

Hi Guys,

My name is Anton Mazkovoi and I have recently joined SafetyCulture, the company for whom jnordberg raised this issue.

We are now using Sync Gateway 1.0.3, but fetching _all_docs is still very slow.

Would you be able to indicate whether this is something you are planning to fix in the short term? An ETA for this would greatly help us understand when we can take Sync Gateway into production.

Cheers,
Anton

[ajres]

@amazkovoi, this is the next ticket in my current sprint.

Initial review suggests the issue is that the _all_docs, passes every document in the underlying bucket through a channel filter for the calling user. Whereas the _changes feed gets the channels for the user and then retrieves the revisions only in those channels.

Since the most common use cases will have each user only see a small subset of the documents in a bucket, _all_docs performance would probably be faster if we take a similar approach to the _changes feed.

Andy

[amazkovoi]

Hi Andy,

That's great news! Thank you!!!

You are absolutely right. We find that each user has access to only a subset of documents, so getting _all_docs to use a similar approach to _changes makes a lot of sense.

Cheers,
Anton

[ajres]

For authenticated users on the public API, _all_docs will now generate the list of documents based on the changes feed. On the admin port the _all_docs feed is still generated by iterating over all the documents in the underlying bucket.