Attachments fail to sync if revpos ancestor on server has been compacted

[alex7egli]

If a document with an attachment in it gets out of sync so that the client is a revision or two behind the server, and before it resyncs the ancestor that is the revpos of the attachment is compacted away on the server, then the document will not be able to sync again. Newer revisions from the server are not synced to the client and the newer revisions from the client are not synced to the server. Only solution is to delete client DB and resync completely, losing any data in the db since the last successful sync. Error shown in the logs is:

BulkDocs: Doc "[DOC ID]" --> 400 Missing digest in stub attachment "[attachment name]" (400 Missing digest in stub attachment "[attachment name]")

Steps to Reproduce:

1. Create a document
2. Attach something to the document
3. Sync the document to a client device
4. Turn off wi-fi on the client device
5. Update the document on the client device
6. Update the document a few times on the server
7. Compact the bucket the document is in on the server (make sure it deletes the revpos of the attachment, e.g. if you attached to revision 2 then make sure revision 2 is deleted). You may have to create lots of revisions, or have a very active db, for this to actually get rid of the revision with the attachment.
8. Turn wi-fi back on for the client device, try to sync
   RESULTS: You see a 400 Missing Digest error for the document in the sync gateway logs. The document does not sync the client updates to the server.

CAUSE:
The client does not send the digest of the attachment to the server in a _bulk_docs POST. It just sends the revpos and stub=true. E.g. In the doc on the server the attachment metadata object looks like:
"_attachments" : {"name":{"content_type":"image/jpg","digest":"sha1-RD3Ax1G1gPfqWrCnMmh/OzNLVDY=","length":134124,"revpos":2,"stub":true}}
But in the _bulk_docs POST request it just has:
"_attachments":{"name":{"stub":true,"revpos":2}}
Since the client and the server are out of sync Sync gateway tries to walk up the tree of the document to the revision that the attachment came from to get the full _attachment object and insert the client revisions into the tree, but since the original attachment revision is gone it fails.

Walking up the tree:

sync_gateway/db/crud.go

Line 308 in 3aa3c25

func (db *Database) getAvailableRev(doc *document, revid string) (Body, error) {

Called in here, which actually generates the 400 error because digest is missing because it failed to find the parent revision when walking up the tree:

sync_gateway/db/attachment.go

Line 40 in 3aa3c25

// Given a CouchDB document body about to be stored in the database, goes through the _attachments

Tries to integrate the attachment:

sync_gateway/db/crud.go

Line 409 in 3aa3c25

func (db *Database) PutExistingRev(docid string, body Body, docHistory []string) error {

[zgramana]

Related to issue #1034

[jprosevear]

Its somewhat related to #1034, however the solution for that, walking up the ancestor tree, will not work here. There is no ancestor revision available.

[adamcfraser]

My feeling is that it's actually the same as #1034, where the root cause of #1034 is also unavailable ancestor (as here), since the pre-tombstone revisions would get pruned immediately.

[snej]

Well, “revpos”:2 means “this attachment hasn’t changed since generation 2”. So even if the ancestor with revID 2-* is gone, the same attachment can be found in any later revision in that branch.

In fact SG should be able to look at the current revision on the branch (the latest common ancestor) and find that same attachment, with the same “revpos”:2. If it can’t, then something’s wrong with the request.

[adamcfraser]

@snej Makes sense - will make that change in the SG handling.

[adamcfraser]

@snej We're already handling the first case you described - we're checking the revpos on the immediate parent for a non-conflicting update, and don't require the revpos revision to still exist.

I'm only able to reproduce the filer's issue if the caller doesn't include the revision id corresponding to revpos in the _revisions property on the bulk_docs request.

In this case the client is pushing a revision that doesn't have a common ancestor on the server (that ancestor has been pruned away).

[adamcfraser]

I was able to reproduce by doing a PUT to /db/doc?new_edits=false with both of these payloads, where the doc already exists on the server, and rev 1-c325a14e17ef08fb34c37b965d0ca3a0 has been pruned away.

{"_rev":"2-foo", "_attachments": {"hello.txt": {"stub":true, "revpos":1}}}

{"_revisions":{"ids":["foo", "c325a14e17ef08fb34c37b965d0ca3a0"],"start":2}, "_attachments": {"hello.txt": {"stub":true, "revpos":1}}}

So the scenario appears to be:

1. Revpos revision has been pruned on the server
2. Client attempts to push a new (conflicting) revision for the doc as a child of the pruned revision.

The active revision has the same revpos, but since we don't have the history back to that generation on the server, I don't think the server can validate that the revision at revpos=1 that it knows about is actually the same revision at revpos=1 that the request is referencing (i.e. the server could be referencing an attachment added at 1-foo, and the client referencing an attachment on a conflicting branch at 1-bar).

@snej, @pasin Based on this, I feel like Sync Gateway is doing the right thing, and the client needs to resubmit the conflicting revision with the full attachment. Let me know what you think.

[adamcfraser]

@alex7egli Can you let us know what platform of Couchbase Lite you're using?

[snej]

I don't think the scenario you describe can happen, unless there's a bug somewhere. Since the server doesn't have revision 1-c325 anymore, the _revs_diff response for the doc would have included an empty/missing possible_ancestors list. In that case, CBL should include all attachments in the PUT or bulk_docs request, i.e. "hello.txt" would have a "data" not a "stub":true.
(In the iOS codebase see CBL_RestPusher.m, lines 304-318, i.e. starting at // Add the revision history:.)

[jprosevear]

Its iOS 1.1.1. Given the number of reports, I think its reasonable to suggest there is a bug. The POST request would need to be scrubbed to upload fully, but the relevant snippets are:

     "_rev":"16-6bca1a3dc2f468bf77400da7b07540f2",

     "_attachments":{
        "profileImage":{
           "stub":true,
       "revpos":3
        }
     },

     "_revisions":{
        "ids":[
           "6bca1a3dc2f468bf77400da7b07540f2",
           "d9c7eefd468c6d932a44bf92f8c67fb0"
        ],
        "start":16
     },

"new_edits":false

[adamcfraser]

@jprosevear Are you able to provide a scrubbed version of the same document as viewed through the Couchbase Server Admin Console (i.e. showing the _sync metadata)?

[jprosevear]

@alex7egli should be able to

[alex7egli]

Here is the current _sync object for that document:

{
    "_attachments": {
        "profileImage": {
            "content_type": "image/jpg",
            "digest": "sha1-IIbtV/OFrSpl0Dj9rI5HKHCyxxE=",
            "length": 102620,
            "revpos": 3,
            "stub": true
        }
    },
    "_sync": {
        "rev": "27-96a10d41bf71b39479312a73a97faa5e",
        "flags": 24,
        "sequence": 6451044,
        "recent_sequences": [
            5795619,
            5795664,
            5795665,
            5795720,
            5795721,
            5795722,
            5795740,
            5795743,
            5795788,
            6158173,
            6197302,
            6197526,
            6197527,
            6197528,
            6212048,
            6217413,
            6268398,
            6451044
        ],
        "history": {
            "revs": [
                "2-159105a56d837a946358f7633a0a8795",
                "10-bb636160ac6bf1d3b0d906249bd0717d",
                "7-bdc07aad4568e6081abee41fbc2b691b",
                "14-4b7138ade24aa4c54bcca17d9a4ea715",
                "24-812bb5671990fe77b6cbf67a0e92dd1c",
                "8-ff69fbf17da5a2daab99121d58b833e5",
                "19-54450b8187d99a6feda37e5d85b3c432",
                "21-77803b64abab37ddbfbb8fcdb6a1f1ba",
                "3-e8a7d12976a63f6be68febf2456bad6d",
                "4-41ae1ce4396904b27753f285ac781ae0",
                "5-2a384f20e180eca6a5cd804933b18a4a",
                "9-ec50e831fe5584eb900d152e900bdc71",
                "23-b73b8b910a8c2b690a2c45acaa923a85",
                "1-24ed6d88f74da77e06cda5ef4c64eacf",
                "16-6cce60e7927d703ff8dd816ae6a6b95a",
                "22-7cb598f5a1a9cb542b40b2166bef10fd",
                "12-c66a540a7119f91a22f09f8c6484b787",
                "25-09cb714677c914a815622a0708a2a3f9",
                "17-c0846c81b43775aca8801efc3e2ef700",
                "6-c5d63a5ebda9da51a3afeb188fd53134",
                "15-d9c7eefd468c6d932a44bf92f8c67fb0",
                "27-96a10d41bf71b39479312a73a97faa5e",
                "8-1cf420679d46354fc5ce79829d4d6ea2",
                "7-82cfd33a912fb20d2ee2eefaa7b321ac",
                "20-51d998ae87a379077cd5706c789b1702",
                "11-0afdf8e8f13088affbfd9497831df010",
                "18-ebecef5b389cad05d6192396c0249679",
                "26-b5988fb6a4ba33b86f1c8affc891cd73",
                "13-afa13aac72f206ef1c5a89a18ffe25c1"
            ],
            "parents": [
                13,
                11,
                19,
                28,
                12,
                2,
                26,
                24,
                0,
                8,
                9,
                5,
                15,
                -1,
                20,
                7,
                25,
                4,
                14,
                10,
                3,
                27,
                23,
                19,
                6,
                1,
                18,
                17,
                16
            ],
            "bodies": [
                "",
                "",
                "",
                "",
                "",
                "",
                "",
                "",
                "",
                "",
                "",
                "",
                "",
                "",
                "",
                "",
                "",
                "",
                "",
                "",
                "",
                "",
                "{\"_attachments\":{\"profileImage\":{\"content_type\":\"image/jpg\",\"digest\":\"sha1-IIbtV/OFrSpl0Dj9rI5HKHCyxxE=\",\"length\":102620,\"revpos\":3,\"stub\":true}},\"createdAt\":\"2016-03-14T18:16:01Z\",\"dateOfBirth\":\"1981-03-14\",\"demo\":false,\"email\":\"[email protected]\",\"familyName\":\"Vaz\",\"favouriteExercises\":[],\"favouriteTeamId\":\"-CBdIFrrwwByACu3ham-1it\",\"gender\":\"m\",\"givenName\":\"Luís\",\"heightUnit\":\"ft\",\"isEndOfRestTimeBeeping\":false,\"isExerciseLoadOptionalForPowerOutputDisplayed\":false,\"isManualEntryHintDisplayed\":false,\"isManualEntryWalkthroughDisplayed\":false,\"isRealTimeInProductionMessageDisplayed\":true,\"isShowPeaks\":false,\"isVelocityLossFormulaChangeMessageDisplayed\":false,\"lastPortalLogin\":1.457979750985e+12,\"mobileDevicesUsed\":[\"IPHONE_9.2.1\"],\"portalLogins\":[1.457979750985e+12],\"realTimeDisplayMetric\":0,\"realTimeOn\":true,\"realTimeOnBeeping\":false,\"recommendationOn\":true,\"role\":\"coach\",\"seenWalkthrough\":true,\"skillLevel\":\"intermediate\",\"type\":\"user\",\"updatedAt\":\"2016-03-14T18:22:32.057Z\",\"updatedBy\":\"6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd\",\"userId\":\"6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd\",\"weightUnit\":\"lbs\"}",
                "",
                "",
                "",
                "",
                "",
                ""
            ],
            "channels": [
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ],
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ],
                null,
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ],
                null,
                null,
                null,
                null,
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ],
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ],
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ],
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ],
                null,
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ],
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ],
                null,
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ],
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ],
                null,
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ],
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ],
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ],
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ],
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ],
                null,
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ],
                null,
                null,
                [
                    "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync",
                    "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd",
                    "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd"
                ]
            ]
        },
        "channels": {
            "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync": null,
            "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd": null,
            "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd": null
        },
        "access": {
            "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd": {
                "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_results": 5795619,
                "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd_sync": 5795619,
                "all": 5795619,
                "user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd": 5795619,
                "write_user_6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd": 5795619
            }
        },
        "role_access": {
            "6b3bd10a-af9f-4e55-90f9-bbbeb3e6b4bd": {
                "user": 5795619
            }
        },
        "time_saved": "2016-04-20T17:20:51.003877Z"
    }
}

[adamcfraser]

Thanks @alex7egli that's very helpful. It looks like the server and the client doc share a common ancestor that hasn't been pruned (15-d9c7eefd468c6d932a44bf92f8c67fb0), so Sync Gateway should be able to recognize the attachment. I'll look into this today and try to work out what's going on.

[adamcfraser]

Fixed by #1725.