Large revs with no common parent. Panic serving, can't find rev

[hegra]

I'm running an android app(couchbase-lite-android:1.1.0-532) where multiple clients can make a new document with the same specific document id. They are then all pushed to the server (these clients never pull).
After running for some days another android client pulls the data and tries to flag all but one revision as deleted. Then after a sync to the server I get the crash report as seen below. In addition only some of the entries in the bodymap of the current revision are flagged as deleted when inspecting the document on the server side.

This looks similar to issue #807 but we are running master(bfb402f) in which the revision pruning issue should have been corrected?

2015/07/28 14:00:03 http: panic serving x.x.x.x:48698: can't find rev: 4-998bcf4e7acf256af85f46f3a8d31e75
goroutine 7660 [running]:
net/http.func·011()
        /usr/lib/golang/src/net/http/server.go:1130 +0xbb
github.com/couchbase/sync_gateway/db.RevTree.getInfo(0xc23eadb470, 0xc23eadb380, 0x22, 0xc214256ff0)
        /home/ec2-user/sync_gateway/sync_gateway/src/github.com/couchbase/sync_gateway/db/revtree.go:132 +0xd8
github.com/couchbase/sync_gateway/db.RevTree.getHistory(0xc23eadb470, 0xc23eadb380, 0x22, 0x0, 0x0, 0x0)
        /home/ec2-user/sync_gateway/sync_gateway/src/github.com/couchbase/sync_gateway/db/revtree.go:148 +0x184
github.com/couchbase/sync_gateway/db.(*Database).updateDoc(0xc22bb24700, 0xc23eadb290, 0x24, 0x100, 0xc239c72dc0, 0x0, 0x0, 0x0, 0x0)
        /home/ec2-user/sync_gateway/sync_gateway/src/github.com/couchbase/sync_gateway/db/crud.go:665 +0x8c0
github.com/couchbase/sync_gateway/db.(*Database).PutExistingRev(0xc22bb24700, 0xc23eadb290, 0x24, 0xc23eadb230, 0xc22bb25100, 0x2, 0x2, 0x0, 0x0)
        /home/ec2-user/sync_gateway/sync_gateway/src/github.com/couchbase/sync_gateway/db/crud.go:445 +0x447
github.com/couchbase/sync_gateway/rest.(*handler).handleBulkDocs(0xc239c6d5f0, 0x0, 0x0)
        /home/ec2-user/sync_gateway/sync_gateway/src/github.com/couchbase/sync_gateway/rest/bulk_api.go:397 +0xd45
github.com/couchbase/sync_gateway/rest.(*handler).invoke(0xc239c6d5f0, 0xd8b4a8, 0x0, 0x0)
        /home/ec2-user/sync_gateway/sync_gateway/src/github.com/couchbase/sync_gateway/rest/handler.go:159 +0x4b8
github.com/couchbase/sync_gateway/rest.func·015(0x7f24a5b0d2a8, 0xc239c76280, 0xc23d6355f0)
        /home/ec2-user/sync_gateway/sync_gateway/src/github.com/couchbase/sync_gateway/rest/handler.go:86 +0x7d
net/http.HandlerFunc.ServeHTTP(0xc2083774e0, 0x7f24a5b0d2a8, 0xc239c76280, 0xc23d6355f0)
        /usr/lib/golang/src/net/http/server.go:1265 +0x41
github.com/gorilla/mux.(*Router).ServeHTTP(0xc20803b450, 0x7f24a5b0d2a8, 0xc239c76280, 0xc23d6355f0)
        /home/ec2-user/sync_gateway/sync_gateway/src/github.com/gorilla/mux/mux.go:86 +0x29e
github.com/couchbase/sync_gateway/rest.func·017(0x7f24a5b0d2a8, 0xc239c76280, 0xc23d6355f0)
        /home/ec2-user/sync_gateway/sync_gateway/src/github.com/couchbase/sync_gateway/rest/routing.go:236 +0x32f
net/http.HandlerFunc.ServeHTTP(0xc2081cbe60, 0x7f24a5b0d2a8, 0xc239c76280, 0xc23d6355f0)
        /usr/lib/golang/src/net/http/server.go:1265 +0x41
net/http.serverHandler.ServeHTTP(0xc2081f20c0, 0x7f24a5b0d2a8, 0xc239c76280, 0xc23d6355f0)
        /usr/lib/golang/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc21fdf45a0)
        /usr/lib/golang/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
        /usr/lib/golang/src/net/http/server.go:1751 +0x35e

[tleyden]

@hegra are you using bucket shadowing? Asking in case it's related to #959

Also, can you post your sync gateway config, or at least let me know what you are setting your revs_limit to, or whether you are leaving it to the default value?

[hegra]

First I was running default config. Then I changed to "revs_limit":5. No bucket shadowing.

[tleyden]

@adamcfraser it's interesting that people are seeing this error without bucket shadowing.

[adamcfraser]

@tleyden Also interesting that, based on the stack trace, it looks like the doc update is successful, and it's panicking when trying to add the rev to the rev cache. Would help to understand whether it's an issue with rev cache handling (it's expecting a parent where we don't actually need one), or an actual problem with the rev tree for this doc.

[hegra]

Looked through the issues you referenced to and one thing that perhaps can be in effect here is that I restart sync gateway every second hour because the memory usage is too large. Now with #966 closed I will see if I can avoid restarting the sync gateway.

[tleyden]

@hegra can you post:

• Sync gateway config
• Sync gateway logs

to github gists? With logs, please make sure there is no sensitive information.

[tleyden]

It would also be useful if you could post your android logs.

[zgramana]

Looks like we have enough information to try some speculative testing.

[hegra]

Made a gist here with a unit test for android and complete logs and configs.

https://gist.github.com/hegra/227c490e406c027f9c31

[ajres]

Reproduced the panic from both Java core and Android test clients.

Now ported to unit test for ongoing automated testing.

[tleyden]

Action item from discussion:

• RevTree.getInfo() should return an error instead of panicking
• RevTree.getHistory() will log a warning, but other callers of RevTree.getInfo() could still panic
• Retest against Sync Gw 1.0.4 and see if it's a regression

Btw some notes on rev tree pruning that might be useful: https://github.com/couchbase/sync_gateway/wiki/Rev-Tree-Storage-on-Couchbase-Server

[ajres]

Ran test against Sync Gateway 1.0.4, no panics were logged.

SG logs from the test runs against 1.04 and 1.1.1 have been uploaded as gists

1.0.4

1.1.1

[ajres]

Issue was caused by updateDocument callback function pruning the revision history and removing the newRevision. A subsequent attempt to insert the new revision into the revision cache would fail.

Now the revision will not be inserted into the revision cache when it has been pruned by the callback function.