First-time sync optimized backfill mode

[zgramana]

Proposal

Add a new option to the style parameter of the _changes endpoint: all_active. This would be a similar to the existing all_docs behavior, but would exclude _removed entries as well as tombstoned documents.

Motivations

During a first-time replication, where Couchbase Lite is backfilling up to the current last_sequence, current all_docs requests will include a number of items unlikely to be of interest, such as deleted documents and _removed revisions. For a long running database, this can result in more data being transmitted to Couchbase Lite and is desired. This proposal allows a more streamlined, concise first-time backfill.

[adamcfraser]

In addition to filtering deleted/removed out of the set of active revisions, we should also remove them from the set of non-winning leaf revisions returned.

[zgramana]

Proposed change to Couchbase Lite needed to support this would look something like this (using CBL.NET as the example)

if (includeConflicts)
{
       if (lastSequenceID == null) 
       {
            path.Append("&style=all_active");
       }
       else
       {
            path.Append("&style=all_docs");
       }
}

[snej]

Looks good to me!

[snej]

CBL could also add &include_docs=true, and get the doc bodies in the same stream, which would eliminate the need to send the bulk_get. Does SG implement that option?

[adamcfraser]

Yes - include_docs is implemented by SG, and would work with the proposed style=all_active.

[snej]

I noticed a problem with the proposal: If we send ?style=all_active to a server that doesn't understand it (SG 1.1, CouchDB, ...), the server will probably ignore the parameter and use the default style value. That means we won't get the all_docs behavior, so we won't get info about conflicts.

Unfortunately we can't do a version check of the server before sending the feed, because (due to recent pull optimizations) _changes is the first request we send the server on a first-time pull.

As an alternative, I suggest we add a new parameter, something like ?include_deleted=false. That way if the server doesn't recognize it, we still have style=all_docs in the URL and will get the conflicts.

[adamcfraser]

Good point Jens. I agree that we need to have a backwards-compatible approach, and an additional parameter makes that possible without intrusive client changes.

I'd suggest that we use the parameter name ?active_only=true, in order to better describe the fact that both deleted and removed revisions will be excluded.

[ethanfrey]

@adamcfraser

I compiled this commit from the current master 1444ebc locally and unfortunately, it didn't seem to fix the problem. Basically, http://localhost:4985/default/_changes?style=all_docs&active_only=true returns the exact same results as http://localhost:4985/default/_changes I have some "resolved" conflicts, where all but one leaf nodes were deleted and an unresolved conflict, where there are two leaf nodes, not deleted.

The first case works well, all deleted nodes are not shown in the changes feed. In the second case, I would expect both exisiting (conflicting) leaf revisions to appear in http://localhost:4985/default/_changes?style=all_docs&active_only=true Here is an example:

http://localhost:4985/default/_changes?style=all_docs&active_only=true:

{
seq: 67,
id: "actions:tab:32383a4d-89d3-455e-807e-a2dfa73d9108",
changes: [
{
rev: "30-3c34652c89532fae4a987ee36b98864c"
}
]
},...

http://localhost:4985/default/_changes?style=all_docs:

...
{
seq: 67,
id: "actions:tab:32383a4d-89d3-455e-807e-a2dfa73d9108",
changes: [
{
rev: "30-3c34652c89532fae4a987ee36b98864c"
},
{
rev: "8-601096f71316c503b57aadaa4cc976bb"
}
]
},...

And now for the two revisions:
http://localhost:4985/default/actions:tab:32383a4d-89d3-455e-807e-a2dfa73d9108?rev=30-3c34652c89532fae4a987ee36b98864c:

{
_id: "actions:tab:32383a4d-89d3-455e-807e-a2dfa73d9108",
_rev: "30-3c34652c89532fae4a987ee36b98864c",
... doc data here...
}

http://localhost:4985/default/actions:tab:32383a4d-89d3-455e-807e-a2dfa73d9108?rev=8-601096f71316c503b57aadaa4cc976bb:

{
_id: "actions:tab:32383a4d-89d3-455e-807e-a2dfa73d9108",
_rev: "8-601096f71316c503b57aadaa4cc976bb",
... doc data here ...
}

Neither of these two leaf node has a _deleted: true element

[ethanfrey]

I took a look at the fix and think I found the issue. At least it now provides correct results on my local data set. Simply a small error in the conditional logic. Making a pull request now, please review it.