Sync Gateway not starting first time

[cavalier123]

When trying to start the Sync Gateway (in a DOS window, under Windows 8.1), it almost never starts first time. Instead, it will fail will with an error message like the following:

09:09:49.350987 WARNING: Error from Incr in _reserveSequences(0): MCResponse status=DELTA_BADVAL, opcode=INCREMENT, opaque=0, msg: Non-numeric server-side value for incr or decr -- db.(*sequenceAllocator)._reserveSequences() at sequence_allocator.go:59
09:09:49.350987 FATAL: Error opening database: MCResponse status=DELTA_BADVAL, opcode=INCREMENT, opaque=0, msg: Non-numeric server-side value for incr or decr -- rest.RunServer() at config.go:415

Typically, this will happen 5-10 times before a successful restart.

We are using the latest versions of the Couchbase Server and Sync Gateway (versions 3.0.2 and 1.0.3, respectively), downloaded from the Couchbase web site. Our application uses an app bucket and a corresponding shadow bucket. The Sync Gateway is running on a Windows 8.1 (Enterprise Edition, 64bit) PC with an Intel Xeon E5 processor and 40Gb of RAM. The Couchbase server is running on a second, similarly specced, PC.

I have 'Googled' this issue and several other other users have experienced the same problem. Those users that have had the problem usually report that it has just 'gone away' after installing a new server version (or carrying out a similar action). Indeed, a bug report on this forum (#302) ends with the comment "I guess we can say this was just a weird problem with the old server installation.". BTW, I have (as suggested on this report) checked the contents of the _sync:seq document, and it does contain the correct contents (a whole number {only, no curly braces})

Also this - https://issues.couchbase.com/browse/NCBC-720 - seems relevant, but is about the .NET client software specifically.

[cavalier123]

Interestingly, we have just seen a similar error (given below) in our mobile application that runs Couchbase on Android. This application is connected through a Sync Gateway (the same sync Gateway referred to in my first posting) to a Couchbase server. The error has not caused any detectable problem at the mobile application or server.

02-19 08:37:08.662 2040-2265/uk.co.exel.fieldservice E/CBLite﹕ Exception getting status from {error=502, id=FieldServiceVisit:12, reason=Non-numeric server-side value for incr or decr (DELTA_BADVAL), status=502}
java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String
at com.couchbase.lite.replicator.ReplicationInternal.statusFromBulkDocsResponseItem(ReplicationInternal.java:1270)
at com.couchbase.lite.replicator.PusherInternal$5.onCompletion(PusherInternal.java:497)
at com.couchbase.lite.support.RemoteRequestRetry$1.completed(RemoteRequestRetry.java:178)
at com.couchbase.lite.support.RemoteRequestRetry$1.onCompletion(RemoteRequestRetry.java:189)
at com.couchbase.lite.support.RemoteRequest.respondWithResult(RemoteRequest.java:310)
at com.couchbase.lite.support.RemoteRequest.executeRequest(RemoteRequest.java:260)
at com.couchbase.lite.support.RemoteRequest.run(RemoteRequest.java:104)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:390)
at java.util.concurrent.FutureTask.run(FutureTask.java:234)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:153)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
at java.lang.Thread.run(Thread.java:841)

[JaiL81]

Hi,

I have exactly the same problem with a W7 machine and couchbase server 3.01 and sync gateway 1.0.4 (also happened with 1.0.3)

I am not using the android client of CBLite but the phonegapp plugin so I don't know if I get the same stacktrace that you get.

[adamcfraser]

I reproduced this issue once in the performance environment when testing a 3-SG cluster (using non-Windows CBS) - one of the three nodes got an error back for the initial Incr call to initialize the sequence. Will look into better error/retry handling in the SG codebase to address this.

[tleyden]

I just saw this on the AWS perfcluster as well:

2015/04/13 19:12:42.601070 Enabling logging: [HTTP HTTP+ CRUD DCache]
2015/04/13 19:12:42.601188 ==== Couchbase Sync Gateway/feature/distributed_cache(10a2cdc) ====
2015/04/13 19:12:42.601212 Configured Go to use all 2 CPUs; setenv GOMAXPROCS to override this
2015/04/13 19:12:42.601224 Configured process to allow 4096 open file descriptors
2015/04/13 19:12:42.601246 Opening db /db as bucket "bucket-1", pool "default", server <http://10.36.81.228:8091/>
2015/04/13 19:12:42.601326 Opening Couchbase database bucket-1 on <http://10.36.81.228:8091/>
2015/04/13 19:12:43.215266 Opening cache bucket "bucket-2", pool "default", server <http://10.36.81.228:8091/>
2015/04/13 19:12:43.215364 Opening Couchbase database bucket-2 on <http://10.36.81.228:8091/>
2015/04/13 19:12:43.318851 Opened cache bucket successfully, writer=false
2015/04/13 19:12:43.327153 WARNING: Error from Incr in _reserveSequences(0): MCResponse status=TMPFAIL, opcode=INCREMENT, opaque=0, msg: Temporary failure -- db.(*sequenceAllocator)._reserveSequences() at sequence_allocator.go:59
2015/04/13 19:12:43.327183 FATAL: Error opening database: MCResponse status=TMPFAIL, opcode=INCREMENT, opaque=0, msg: Temporary failure -- rest.RunServer() at config.go:488

[tleyden]

@jamesmauss have you seen this error before? Any ideas why this would happen?

WARNING: Error from Incr in _reserveSequences(0): MCResponse status=TMPFAIL, opcode=INCREMENT, opaque=0, msg: Temporary failure -- db.(*sequenceAllocator)._reserveSequences() at sequence_allocator.go:59

[adamcfraser]

Added retry support (#786) for failed incr calls.