You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have a replica set with 2 replicas + arbiter. This morning, one of the replicas crashed (and won't start again, will file a separate issue), and the other one works mostly fine, but I can't rebuild the replica set because listDatabases makes the thing crash with [1];
<rant>even more awesomer, there is absolutely no repair option I can find, because this code can't ever be wrong and filesystems can't ever get corrupted...</rant>
[1]
Fri Mar 27 11:06:06.395 nssize is a deprecated parameter
Fri Mar 27 11:06:06.397 [initandlisten] TokuMX starting : pid=29832 port=13456 dbpath=atk2/tokumx-atk2.bak/ 64-bit host=toku-bk1
Fri Mar 27 11:06:06.397 [initandlisten] TokuMX mongod server v2.0.0-mongodb-2.4.10, using TokuKV rev 668f1118593ba0976b6ec68768062f64d418ec83
Fri Mar 27 11:06:06.397 [initandlisten] git version: c7f2e017eb71d93ca51d5073eb1570f6c9ce0ba1
Fri Mar 27 11:06:06.397 [initandlisten] build info: Linux c1ad9bbe58cd 3.11.0-26-generic #45-Ubuntu SMP Tue Jul 15 04:02:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux BOOST_LIB_VERSION=1_49
Fri Mar 27 11:06:06.397 [initandlisten] options: { dbpath: "atk2/tokumx-atk2.bak/", port: 13456 }
Fri Mar 27 11:06:06.417 [initandlisten] [tokumx] startup
Fri Mar 27 11:06:06.968 [initandlisten]
Fri Mar 27 11:06:06.968 [initandlisten] ** WARNING: mongod started without --replSet yet 1 documents are present in local.system.replset
Fri Mar 27 11:06:06.968 [initandlisten] ** Restart with --replSet unless you are doing maintenance and no other clients are connected.
Fri Mar 27 11:06:06.968 [initandlisten] ** The TTL collection monitor will not start because of this.
Fri Mar 27 11:06:06.968 [initandlisten] ** For more info see http://dochub.mongodb.org/core/ttlcollections
Fri Mar 27 11:06:06.968 [initandlisten]
Fri Mar 27 11:06:06.968 [initandlisten] waiting for connections on port 13456
Fri Mar 27 11:06:06.969 [websvr] admin web console waiting for connections on port 14456
Fri Mar 27 11:06:10.712 [initandlisten] connection accepted from 127.0.0.1:57521 #1 (1 connection now open)
Fri Mar 27 11:06:12.496 [conn1] lock status: R recursive:2 otherCount:0
Fri Mar 27 11:06:12.496 [conn1] Assertion: 16105:expected to be write locked for maindb.system.users
Fri Mar 27 11:06:12.502 [conn1] maindb.system.namespaces Assertion failure !Lock::isReadLocked() /data/release_build-linux-c-opt/build/src/mongo/db/txn_complete_hooks.cpp 67
Fri Mar 27 11:06:12.508
Fri Mar 27 11:06:12.508 ================================================================================
Fri Mar 27 11:06:12.508 Fatal error detected
Fri Mar 27 11:06:12.508 ================================================================================
Fri Mar 27 11:06:12.508
Fri Mar 27 11:06:12.508 About to gather debugging information, please include all of the following along
Fri Mar 27 11:06:12.508 with logs from other servers in the cluster in a bug report.
Fri Mar 27 11:06:12.508
Fri Mar 27 11:06:12.508 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.508 Version info:
Fri Mar 27 11:06:12.508
Fri Mar 27 11:06:12.508 tokumxVersion: 2.0.0
Fri Mar 27 11:06:12.508 gitVersion: c7f2e017eb71d93ca51d5073eb1570f6c9ce0ba1
Fri Mar 27 11:06:12.508 tokukvVersion: 668f1118593ba0976b6ec68768062f64d418ec83
Fri Mar 27 11:06:12.508 sysInfo: Linux c1ad9bbe58cd 3.11.0-26-generic #45-Ubuntu SMP Tue Jul 15 04:02:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux BOOST_LIB_VERSION=1_49
Fri Mar 27 11:06:12.508 loaderFlags:
Fri Mar 27 11:06:12.508 compilerFlags: -fPIC -fno-strict-aliasing -ggdb -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -pipe -Wnon-virtual-dtor -Woverloaded-virtual -Wno-unused-local-typedefs -fno-builtin-memcmp -O3
Fri Mar 27 11:06:12.508 debug: false
Fri Mar 27 11:06:12.508
Fri Mar 27 11:06:12.508 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.508 Simple stacktrace:
Fri Mar 27 11:06:12.508
0xc4efe3 0x953177 0x955fbb 0x776900 0x7f05ac0f9836 0x7f05ac0f88f9 0x7f05ac0f94aa 0x7f05abb8eff3 0x7f05abb8f517 0x86b3a6 0x86b3d5 0x967a31 0x980974 0x980b1d 0xa090bc 0xa2dccf 0xa1b829 0xa1efc7 0xa1f8ca 0x962c35
/opt/tokumx/bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xc4efe3]
/opt/tokumx/bin/mongod() [0x953177]
/opt/tokumx/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x2b) [0x955fbb]
/opt/tokumx/bin/mongod(_ZN5mongo11myterminateEv+0x20) [0x776900]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836) [0x7f05ac0f9836]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5d8f9) [0x7f05ac0f88f9]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(__gxx_personality_v0+0x52a) [0x7f05ac0f94aa]
/lib/x86_64-linux-gnu/libgcc_s.so.1(+0xfff3) [0x7f05abb8eff3]
/lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_Resume+0x57) [0x7f05abb8f517]
/opt/tokumx/bin/mongod() [0x86b3a6]
/opt/tokumx/bin/mongod(_ZN5mongo6Client11Transaction5abortEv+0x25) [0x86b3d5]
/opt/tokumx/bin/mongod(_ZN5boost10scoped_ptrIN5mongo6Client11TransactionEED1Ev+0x11) [0x967a31]
/opt/tokumx/bin/mongod(_ZN5mongo13CollectionMap7open_nsERKNS_10StringDataEb+0x394) [0x980974]
/opt/tokumx/bin/mongod(_ZN5mongo13CollectionMap13getCollectionERKNS_10StringDataE+0xad) [0x980b1d]
/opt/tokumx/bin/mongod(_ZN5mongo8Database8diskSizeERmS1_+0x6c) [0xa090bc]
/opt/tokumx/bin/mongod(_ZN5mongo16CmdListDatabases3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x60f) [0xa2dccf]
/opt/tokumx/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x39) [0xa1b829]
/opt/tokumx/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xcd7) [0xa1efc7]
/opt/tokumx/bin/mongod(_ZN5mongo12_runCommandsEPKcRKNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x3aa) [0xa1f8ca]
/opt/tokumx/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x962c35]
Fri Mar 27 11:06:12.513
Fri Mar 27 11:06:12.513 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.513 TokuKV engine backtrace:
Fri Mar 27 11:06:12.513
Fri Mar 27 11:06:12.513 Backtrace: (Note: toku_do_assert=0x0x7f05acbe68b0)
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/../lib64/libtokuportability.so(_Z27db_env_do_backtrace_errfuncPFvPKviPKczES0_+0x1f) [0x7f05acbe7c9f]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/../lib64/libtokufractaltree.so(+0x1437d) [0x7f05ace0137d]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod() [0x953248]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x2b) [0x955fbb]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo11myterminateEv+0x20) [0x776900]
Fri Mar 27 11:06:12.518 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836) [0x7f05ac0f9836]
Fri Mar 27 11:06:12.518 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5d8f9) [0x7f05ac0f88f9]
Fri Mar 27 11:06:12.518 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(__gxx_personality_v0+0x52a) [0x7f05ac0f94aa]
Fri Mar 27 11:06:12.518 /lib/x86_64-linux-gnu/libgcc_s.so.1(+0xfff3) [0x7f05abb8eff3]
Fri Mar 27 11:06:12.518 /lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_Resume+0x57) [0x7f05abb8f517]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod() [0x86b3a6]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo6Client11Transaction5abortEv+0x25) [0x86b3d5]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5boost10scoped_ptrIN5mongo6Client11TransactionEED1Ev+0x11) [0x967a31]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo13CollectionMap7open_nsERKNS_10StringDataEb+0x394) [0x980974]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo13CollectionMap13getCollectionERKNS_10StringDataE+0xad) [0x980b1d]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo8Database8diskSizeERmS1_+0x6c) [0xa090bc]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo16CmdListDatabases3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x60f) [0xa2dccf]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x39) [0xa1b829]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xcd7) [0xa1efc7]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo12_runCommandsEPKcRKNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x3aa) [0xa1f8ca]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x962c35]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x6e3) [0x9663b3]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x6cb) [0x993a5b]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x8b) [0x78cd3b]
Fri Mar 27 11:06:12.518 /opt/tokumx/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x3dc) [0x89649c]
Fri Mar 27 11:06:12.518 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f05ac7c8182]
Fri Mar 27 11:06:12.518 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f05ab8b400d]
Fri Mar 27 11:06:12.518 BUILD_ID = 107540753
Fri Mar 27 11:06:12.518 time of environment creation: Tue Jan 13 07:26:51 2015
Fri Mar 27 11:06:12.518 time of engine startup: Fri Mar 27 11:06:06 2015
Fri Mar 27 11:06:12.518 time now: Fri Mar 27 11:06:12 2015
Fri Mar 27 11:06:12.518 db opens: 440
Fri Mar 27 11:06:12.518 db closes: 3
Fri Mar 27 11:06:12.518 num open dbs now: 437
Fri Mar 27 11:06:12.518 max open dbs: 440
Fri Mar 27 11:06:12.518 period, in ms, that recovery log is automatically fsynced: 100
Fri Mar 27 11:06:12.518 dictionary inserts: 0
Fri Mar 27 11:06:12.518 dictionary inserts fail: 0
Fri Mar 27 11:06:12.518 dictionary deletes: 1
Fri Mar 27 11:06:12.518 dictionary deletes fail: 0
Fri Mar 27 11:06:12.518 dictionary updates: 0
Fri Mar 27 11:06:12.518 dictionary updates fail: 0
Fri Mar 27 11:06:12.518 dictionary broadcast updates: 0
Fri Mar 27 11:06:12.518 dictionary broadcast updates fail: 0
Fri Mar 27 11:06:12.518 dictionary multi inserts: 1
Fri Mar 27 11:06:12.518 dictionary multi inserts fail: 0
Fri Mar 27 11:06:12.518 dictionary multi deletes: 2
Fri Mar 27 11:06:12.518 dictionary multi deletes fail: 0
Fri Mar 27 11:06:12.518 dictionary updates multi: 0
Fri Mar 27 11:06:12.518 dictionary updates multi fail: 0
Fri Mar 27 11:06:12.518 le: max committed xr: 1
Fri Mar 27 11:06:12.518 le: max provisional xr: 2
Fri Mar 27 11:06:12.518 le: expanded: 0
Fri Mar 27 11:06:12.518 le: max memsize: 883
Fri Mar 27 11:06:12.518 le: size of leafentries before garbage collection (during message application): 0
Fri Mar 27 11:06:12.518 le: size of leafentries after garbage collection (during message application): 0
Fri Mar 27 11:06:12.518 le: size of leafentries before garbage collection (outside message application): 0
Fri Mar 27 11:06:12.518 le: size of leafentries after garbage collection (outside message application): 0
Fri Mar 27 11:06:12.518 checkpoint: period: 60
Fri Mar 27 11:06:12.518 checkpoint: footprint: 0
Fri Mar 27 11:06:12.518 checkpoint: last checkpoint began : Fri Mar 27 11:06:06 2015
Fri Mar 27 11:06:12.518 checkpoint: last complete checkpoint began : Fri Mar 27 11:06:06 2015
Fri Mar 27 11:06:12.518 checkpoint: last complete checkpoint ended: Fri Mar 27 11:06:06 2015
Fri Mar 27 11:06:12.518 checkpoint: time spent during checkpoint (begin and end phases): 0
Fri Mar 27 11:06:12.518 checkpoint: time spent during last checkpoint (begin and end phases): 0
Fri Mar 27 11:06:12.518 checkpoint: last complete checkpoint LSN: 2565863732
Fri Mar 27 11:06:12.518 checkpoint: checkpoints taken : 2
Fri Mar 27 11:06:12.518 checkpoint: checkpoints failed: 0
Fri Mar 27 11:06:12.518 checkpoint: waiters now: 0
Fri Mar 27 11:06:12.518 checkpoint: waiters max: 0
Fri Mar 27 11:06:12.518 checkpoint: non-checkpoint client wait on mo lock: 0
Fri Mar 27 11:06:12.518 checkpoint: non-checkpoint client wait on cs lock: 0
Fri Mar 27 11:06:12.518 checkpoint: checkpoint begin time: 15
Fri Mar 27 11:06:12.518 checkpoint: long checkpoint begin time: 0
Fri Mar 27 11:06:12.518 checkpoint: long checkpoint begin count: 0
Fri Mar 27 11:06:12.518 cachetable: miss: 12
Fri Mar 27 11:06:12.518 cachetable: miss time: 644
Fri Mar 27 11:06:12.518 cachetable: prefetches: 0
Fri Mar 27 11:06:12.518 cachetable: size current: 406407
Fri Mar 27 11:06:12.518 cachetable: size limit: 17334624256
Fri Mar 27 11:06:12.518 cachetable: size writing: 0
Fri Mar 27 11:06:12.518 cachetable: size nonleaf: 0
Fri Mar 27 11:06:12.518 cachetable: size leaf: 403919
Fri Mar 27 11:06:12.518 cachetable: size rollback: 2488
Fri Mar 27 11:06:12.518 cachetable: size cachepressure: 0
Fri Mar 27 11:06:12.518 cachetable: size currently cloned data for checkpoint: 0
Fri Mar 27 11:06:12.518 cachetable: evictions: 0
Fri Mar 27 11:06:12.518 cachetable: cleaner executions: 3
Fri Mar 27 11:06:12.518 cachetable: cleaner period: 2
Fri Mar 27 11:06:12.518 cachetable: cleaner iterations: 5
Fri Mar 27 11:06:12.518 cachetable: number of waits on cache pressure: 0
Fri Mar 27 11:06:12.518 cachetable: time waiting on cache pressure: 0
Fri Mar 27 11:06:12.518 cachetable: number of long waits on cache pressure: 0
Fri Mar 27 11:06:12.518 cachetable: long time waiting on cache pressure: 0
Fri Mar 27 11:06:12.518 locktree: memory size: 576
Fri Mar 27 11:06:12.518 locktree: memory size limit: 1679775334
Fri Mar 27 11:06:12.518 locktree: number of times lock escalation ran: 0
Fri Mar 27 11:06:12.518 locktree: time spent running escalation (seconds): 0.000000
Fri Mar 27 11:06:12.518 locktree: latest post-escalation memory size: 0
Fri Mar 27 11:06:12.518 locktree: number of locktrees open now: 440
Fri Mar 27 11:06:12.518 locktree: number of pending lock requests: 0
Fri Mar 27 11:06:12.518 locktree: number of locktrees eligible for the STO: 5
Fri Mar 27 11:06:12.518 locktree: number of times a locktree ended the STO early: 0
Fri Mar 27 11:06:12.518 locktree: time spent ending the STO early (seconds): 0.000000
Fri Mar 27 11:06:12.518 locktree: number of wait locks: 0
Fri Mar 27 11:06:12.518 locktree: time waiting for locks: 0
Fri Mar 27 11:06:12.518 locktree: number of long wait locks: 0
Fri Mar 27 11:06:12.518 locktree: long time waiting for locks: 0
Fri Mar 27 11:06:12.518 locktree: number of lock timeouts: 0
Fri Mar 27 11:06:12.518 locktree: number of waits on lock escalation: 0
Fri Mar 27 11:06:12.518 locktree: time waiting on lock escalation: 0
Fri Mar 27 11:06:12.518 locktree: number of long waits on lock escalation: 0
Fri Mar 27 11:06:12.518 locktree: long time waiting on lock escalation: 0
Fri Mar 27 11:06:12.518 ft: dictionary updates: 0
Fri Mar 27 11:06:12.518 ft: dictionary broadcast updates: 0
Fri Mar 27 11:06:12.518 ft: descriptor set: 0
Fri Mar 27 11:06:12.518 ft: messages ignored by leaf due to msn: 0
Fri Mar 27 11:06:12.518 ft: total search retries due to TRY_AGAIN: 0
Fri Mar 27 11:06:12.518 ft: searches requiring more tries than the height of the tree: 0
Fri Mar 27 11:06:12.518 ft: searches requiring more tries than the height of the tree plus three: 0
Fri Mar 27 11:06:12.518 ft: leaf nodes flushed to disk (not for checkpoint): 4
Fri Mar 27 11:06:12.518 ft: leaf nodes flushed to disk (not for checkpoint) (bytes): 35840
Fri Mar 27 11:06:12.518 ft: leaf nodes flushed to disk (not for checkpoint) (uncompressed bytes): 187286
Fri Mar 27 11:06:12.518 ft: leaf nodes flushed to disk (not for checkpoint) (seconds): 0.000023
Fri Mar 27 11:06:12.518 ft: nonleaf nodes flushed to disk (not for checkpoint): 0
Fri Mar 27 11:06:12.518 ft: nonleaf nodes flushed to disk (not for checkpoint) (bytes): 0
Fri Mar 27 11:06:12.518 ft: nonleaf nodes flushed to disk (not for checkpoint) (uncompressed bytes): 0
Fri Mar 27 11:06:12.518 ft: nonleaf nodes flushed to disk (not for checkpoint) (seconds): 0.000000
Fri Mar 27 11:06:12.518 ft: leaf nodes flushed to disk (for checkpoint): 0
Fri Mar 27 11:06:12.518 ft: leaf nodes flushed to disk (for checkpoint) (bytes): 0
Fri Mar 27 11:06:12.518 ft: leaf nodes flushed to disk (for checkpoint) (uncompressed bytes): 0
Fri Mar 27 11:06:12.518 ft: leaf nodes flushed to disk (for checkpoint) (seconds): 0.000000
Fri Mar 27 11:06:12.518 ft: nonleaf nodes flushed to disk (for checkpoint): 0
Fri Mar 27 11:06:12.518 ft: nonleaf nodes flushed to disk (for checkpoint) (bytes): 0
Fri Mar 27 11:06:12.518 ft: nonleaf nodes flushed to disk (for checkpoint) (uncompressed bytes): 0
Fri Mar 27 11:06:12.518 ft: nonleaf nodes flushed to disk (for checkpoint) (seconds): 0.000000
Fri Mar 27 11:06:12.518 ft: uncompressed / compressed bytes written (leaf): UNKNOWN STATUS TYPE: 6
Fri Mar 27 11:06:12.518 ft: uncompressed / compressed bytes written (nonleaf): UNKNOWN STATUS TYPE: 6
Fri Mar 27 11:06:12.518 ft: uncompressed / compressed bytes written (overall): UNKNOWN STATUS TYPE: 6
Fri Mar 27 11:06:12.519 ft: nonleaf node partial evictions: 0
Fri Mar 27 11:06:12.519 ft: nonleaf node partial evictions (bytes): 0
Fri Mar 27 11:06:12.519 ft: leaf node partial evictions: 0
Fri Mar 27 11:06:12.519 ft: leaf node partial evictions (bytes): 0
Fri Mar 27 11:06:12.519 ft: leaf node full evictions: 0
Fri Mar 27 11:06:12.519 ft: leaf node full evictions (bytes): 0
Fri Mar 27 11:06:12.519 ft: nonleaf node full evictions: 0
Fri Mar 27 11:06:12.519 ft: nonleaf node full evictions (bytes): 0
Fri Mar 27 11:06:12.519 ft: leaf nodes created: 0
Fri Mar 27 11:06:12.519 ft: nonleaf nodes created: 0
Fri Mar 27 11:06:12.519 ft: leaf nodes destroyed: 0
Fri Mar 27 11:06:12.519 ft: nonleaf nodes destroyed: 0
Fri Mar 27 11:06:12.519 ft: bytes of messages injected at root (all trees): 0
Fri Mar 27 11:06:12.519 ft: bytes of messages flushed from h1 nodes to leaves: 0
Fri Mar 27 11:06:12.519 ft: bytes of messages currently in trees (estimate): 0
Fri Mar 27 11:06:12.519 ft: messages injected at root: 0
Fri Mar 27 11:06:12.519 ft: broadcast messages injected at root: 0
Fri Mar 27 11:06:12.519 ft: basements decompressed as a target of a query: 1
Fri Mar 27 11:06:12.519 ft: basements decompressed for prelocked range: 0
Fri Mar 27 11:06:12.519 ft: basements decompressed for prefetch: 0
Fri Mar 27 11:06:12.519 ft: basements decompressed for write: 0
Fri Mar 27 11:06:12.519 ft: buffers decompressed as a target of a query: 0
Fri Mar 27 11:06:12.519 ft: buffers decompressed for prelocked range: 0
Fri Mar 27 11:06:12.519 ft: buffers decompressed for prefetch: 0
Fri Mar 27 11:06:12.519 ft: buffers decompressed for write: 0
Fri Mar 27 11:06:12.519 ft: pivots fetched for query: 0
Fri Mar 27 11:06:12.519 ft: pivots fetched for query (bytes): 0
Fri Mar 27 11:06:12.519 ft: pivots fetched for query (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: pivots fetched for prefetch: 0
Fri Mar 27 11:06:12.519 ft: pivots fetched for prefetch (bytes): 0
Fri Mar 27 11:06:12.519 ft: pivots fetched for prefetch (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: pivots fetched for write: 4
Fri Mar 27 11:06:12.519 ft: pivots fetched for write (bytes): 35840
Fri Mar 27 11:06:12.519 ft: pivots fetched for write (seconds): 0.000009
Fri Mar 27 11:06:12.519 ft: basements fetched as a target of a query: 0
Fri Mar 27 11:06:12.519 ft: basements fetched as a target of a query (bytes): 0
Fri Mar 27 11:06:12.519 ft: basements fetched as a target of a query (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: basements fetched for prelocked range: 0
Fri Mar 27 11:06:12.519 ft: basements fetched for prelocked range (bytes): 0
Fri Mar 27 11:06:12.519 ft: basements fetched for prelocked range (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: basements fetched for prefetch: 0
Fri Mar 27 11:06:12.519 ft: basements fetched for prefetch (bytes): 0
Fri Mar 27 11:06:12.519 ft: basements fetched for prefetch (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: basements fetched for write: 4
Fri Mar 27 11:06:12.519 ft: basements fetched for write (bytes): 35840
Fri Mar 27 11:06:12.519 ft: basements fetched for write (seconds): 0.000002
Fri Mar 27 11:06:12.519 ft: buffers fetched as a target of a query: 0
Fri Mar 27 11:06:12.519 ft: buffers fetched as a target of a query (bytes): 0
Fri Mar 27 11:06:12.519 ft: buffers fetched as a target of a query (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: buffers fetched for prelocked range: 0
Fri Mar 27 11:06:12.519 ft: buffers fetched for prelocked range (bytes): 0
Fri Mar 27 11:06:12.519 ft: buffers fetched for prelocked range (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: buffers fetched for prefetch: 0
Fri Mar 27 11:06:12.519 ft: buffers fetched for prefetch (bytes): 0
Fri Mar 27 11:06:12.519 ft: buffers fetched for prefetch (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: buffers fetched for write: 0
Fri Mar 27 11:06:12.519 ft: buffers fetched for write (bytes): 0
Fri Mar 27 11:06:12.519 ft: buffers fetched for write (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: leaf compression to memory (seconds): 0.002101
Fri Mar 27 11:06:12.519 ft: leaf serialization to memory (seconds): 0.000085
Fri Mar 27 11:06:12.519 ft: leaf decompression to memory (seconds): 0.000469
Fri Mar 27 11:06:12.519 ft: leaf deserialization to memory (seconds): 0.000131
Fri Mar 27 11:06:12.519 ft: nonleaf compression to memory (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: nonleaf serialization to memory (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: nonleaf decompression to memory (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: nonleaf deserialization to memory (seconds): 0.000000
Fri Mar 27 11:06:12.519 ft: promotion: roots split: 0
Fri Mar 27 11:06:12.519 ft: promotion: leaf roots injected into: 11
Fri Mar 27 11:06:12.519 ft: promotion: h1 roots injected into: 0
Fri Mar 27 11:06:12.519 ft: promotion: injections at depth 0: 0
Fri Mar 27 11:06:12.519 ft: promotion: injections at depth 1: 0
Fri Mar 27 11:06:12.519 ft: promotion: injections at depth 2: 0
Fri Mar 27 11:06:12.519 ft: promotion: injections at depth 3: 0
Fri Mar 27 11:06:12.519 ft: promotion: injections lower than depth 3: 0
Fri Mar 27 11:06:12.519 ft: promotion: stopped because of a nonempty buffer: 0
Fri Mar 27 11:06:12.519 ft: promotion: stopped at height 1: 0
Fri Mar 27 11:06:12.519 ft: promotion: stopped because the child was locked or not at all in memory: 0
Fri Mar 27 11:06:12.519 ft: promotion: stopped because the child was not fully in memory: 0
Fri Mar 27 11:06:12.519 ft: promotion: stopped anyway, after locking the child: 0
Fri Mar 27 11:06:12.519 ft: basement nodes deserialized with fixed-keysize: 5
Fri Mar 27 11:06:12.519 ft: basement nodes deserialized with variable-keysize: 8
Fri Mar 27 11:06:12.519 ft: promotion: succeeded in using the rightmost leaf shortcut: 0
Fri Mar 27 11:06:12.519 ft: promotion: tried the rightmost leaf shorcut but failed (out-of-bounds): 0
Fri Mar 27 11:06:12.519 ft: promotion: tried the rightmost leaf shorcut but failed (child reactive): 0
Fri Mar 27 11:06:12.519 ft flusher: total nodes potentially flushed by cleaner thread: 0
Fri Mar 27 11:06:12.519 ft flusher: height-one nodes flushed by cleaner thread: 0
Fri Mar 27 11:06:12.519 ft flusher: height-greater-than-one nodes flushed by cleaner thread: 0
Fri Mar 27 11:06:12.519 ft flusher: nodes cleaned which had empty buffers: 0
Fri Mar 27 11:06:12.519 ft flusher: nodes dirtied by cleaner thread: 0
Fri Mar 27 11:06:12.519 ft flusher: max bytes in a buffer flushed by cleaner thread: 0
Fri Mar 27 11:06:12.519 ft flusher: min bytes in a buffer flushed by cleaner thread: 18446744073709551615
Fri Mar 27 11:06:12.519 ft flusher: total bytes in buffers flushed by cleaner thread: 0
Fri Mar 27 11:06:12.519 ft flusher: max workdone in a buffer flushed by cleaner thread: 0
Fri Mar 27 11:06:12.519 ft flusher: min workdone in a buffer flushed by cleaner thread: 18446744073709551615
Fri Mar 27 11:06:12.519 ft flusher: total workdone in buffers flushed by cleaner thread: 0
Fri Mar 27 11:06:12.519 ft flusher: times cleaner thread tries to merge a leaf: 0
Fri Mar 27 11:06:12.519 ft flusher: cleaner thread leaf merges in progress: 0
Fri Mar 27 11:06:12.519 ft flusher: cleaner thread leaf merges successful: 0
Fri Mar 27 11:06:12.519 ft flusher: nodes dirtied by cleaner thread leaf merges: 0
Fri Mar 27 11:06:12.519 ft flusher: total number of flushes done by flusher threads or cleaner threads: 0
Fri Mar 27 11:06:12.519 ft flusher: number of in memory flushes: 0
Fri Mar 27 11:06:12.519 ft flusher: number of flushes that read something off disk: 0
Fri Mar 27 11:06:12.519 ft flusher: number of flushes that triggered another flush in child: 0
Fri Mar 27 11:06:12.519 ft flusher: number of flushes that triggered 1 cascading flush: 0
Fri Mar 27 11:06:12.519 ft flusher: number of flushes that triggered 2 cascading flushes: 0
Fri Mar 27 11:06:12.519 ft flusher: number of flushes that triggered 3 cascading flushes: 0
Fri Mar 27 11:06:12.519 ft flusher: number of flushes that triggered 4 cascading flushes: 0
Fri Mar 27 11:06:12.519 ft flusher: number of flushes that triggered 5 cascading flushes: 0
Fri Mar 27 11:06:12.519 ft flusher: number of flushes that triggered over 5 cascading flushes: 0
Fri Mar 27 11:06:12.519 ft flusher: leaf node splits: 0
Fri Mar 27 11:06:12.519 ft flusher: nonleaf node splits: 0
Fri Mar 27 11:06:12.519 ft flusher: leaf node merges: 0
Fri Mar 27 11:06:12.519 ft flusher: nonleaf node merges: 0
Fri Mar 27 11:06:12.519 ft flusher: leaf node balances: 0
Fri Mar 27 11:06:12.519 hot: operations ever started: 0
Fri Mar 27 11:06:12.519 hot: operations successfully completed: 0
Fri Mar 27 11:06:12.519 hot: operations aborted: 0
Fri Mar 27 11:06:12.519 hot: max number of flushes from root ever required to optimize a tree: 0
Fri Mar 27 11:06:12.519 txn: begin: 695
Fri Mar 27 11:06:12.519 txn: begin read only: 2
Fri Mar 27 11:06:12.519 txn: successful commits: 453
Fri Mar 27 11:06:12.519 txn: aborts: 242
Fri Mar 27 11:06:12.519 logger: next LSN: 2565864190
Fri Mar 27 11:06:12.519 logger: writes: 6
Fri Mar 27 11:06:12.519 logger: writes (bytes): 25888
Fri Mar 27 11:06:12.519 logger: writes (uncompressed bytes): 25888
Fri Mar 27 11:06:12.519 logger: writes (seconds): 0.000031
Fri Mar 27 11:06:12.519 logger: number of long logger write operations: 0
Fri Mar 27 11:06:12.519 indexer: number of indexers successfully created: 0
Fri Mar 27 11:06:12.519 indexer: number of calls to toku_indexer_create_indexer() that failed: 0
Fri Mar 27 11:06:12.519 indexer: number of calls to indexer->build() succeeded: 0
Fri Mar 27 11:06:12.519 indexer: number of calls to indexer->build() failed: 0
Fri Mar 27 11:06:12.519 indexer: number of calls to indexer->close() that succeeded: 0
Fri Mar 27 11:06:12.519 indexer: number of calls to indexer->close() that failed: 0
Fri Mar 27 11:06:12.519 indexer: number of calls to indexer->abort(): 0
Fri Mar 27 11:06:12.519 indexer: number of indexers currently in existence: 0
Fri Mar 27 11:06:12.519 indexer: max number of indexers that ever existed simultaneously: 0
Fri Mar 27 11:06:12.519 loader: number of loaders successfully created: 0
Fri Mar 27 11:06:12.519 loader: number of calls to toku_loader_create_loader() that failed: 0
Fri Mar 27 11:06:12.519 loader: number of calls to loader->put() succeeded: 0
Fri Mar 27 11:06:12.519 loader: number of calls to loader->put() failed: 0
Fri Mar 27 11:06:12.519 loader: number of calls to loader->close() that succeeded: 0
Fri Mar 27 11:06:12.519 loader: number of calls to loader->close() that failed: 0
Fri Mar 27 11:06:12.519 loader: number of calls to loader->abort(): 0
Fri Mar 27 11:06:12.519 loader: number of loaders currently in existence: 0
Fri Mar 27 11:06:12.519 loader: max number of loaders that ever existed simultaneously: 0
Fri Mar 27 11:06:12.519 memory: number of malloc operations: 0
Fri Mar 27 11:06:12.519 memory: number of free operations: 0
Fri Mar 27 11:06:12.519 memory: number of realloc operations: 0
Fri Mar 27 11:06:12.519 memory: number of malloc operations that failed: 0
Fri Mar 27 11:06:12.519 memory: number of realloc operations that failed: 0
Fri Mar 27 11:06:12.519 memory: number of bytes requested: 0
Fri Mar 27 11:06:12.519 memory: number of bytes used (requested + overhead): 0
Fri Mar 27 11:06:12.519 memory: number of bytes freed: 0
Fri Mar 27 11:06:12.519 memory: largest attempted allocation size: 0
Fri Mar 27 11:06:12.519 memory: size of the last failed allocation attempt: 0
Fri Mar 27 11:06:12.519 memory: estimated maximum memory footprint: 0
Fri Mar 27 11:06:12.519 memory: mallocator version: tokumx-2.0.0-2-0-gc7f2e017eb71d93ca51d5073eb1570f6c9ce0ba1
Fri Mar 27 11:06:12.519 memory: mmap threshold: 4194304
Fri Mar 27 11:06:12.519 filesystem: ENOSPC redzone state: 0
Fri Mar 27 11:06:12.519 filesystem: threads currently blocked by full disk: 0
Fri Mar 27 11:06:12.519 filesystem: number of operations rejected by enospc prevention (red zone): 0
Fri Mar 27 11:06:12.519 filesystem: most recent disk full: Thu Jan 1 01:00:00 1970
Fri Mar 27 11:06:12.519 filesystem: number of write operations that returned ENOSPC: 0
Fri Mar 27 11:06:12.519 filesystem: fsync time: 554923
Fri Mar 27 11:06:12.519 filesystem: fsync count: 471
Fri Mar 27 11:06:12.519 filesystem: long fsync time: 0
Fri Mar 27 11:06:12.519 filesystem: long fsync count: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by a full fetch: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by a partial fetch: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by a full eviction: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by a partial eviction: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by a message injection: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by a message application: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by a flush: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by a the cleaner thread: 0
Fri Mar 27 11:06:12.519 context: tree traversals blocked by something uninstrumented: 0
Fri Mar 27 11:06:12.519 context: promotion blocked by a full fetch (should never happen): 0
Fri Mar 27 11:06:12.519 context: promotion blocked by a partial fetch (should never happen): 0
Fri Mar 27 11:06:12.519 context: promotion blocked by a full eviction (should never happen): 0
Fri Mar 27 11:06:12.519 context: promotion blocked by a partial eviction (should never happen): 0
Fri Mar 27 11:06:12.519 context: promotion blocked by a message injection: 0
Fri Mar 27 11:06:12.519 context: promotion blocked by a message application: 0
Fri Mar 27 11:06:12.519 context: promotion blocked by a flush: 0
Fri Mar 27 11:06:12.519 context: promotion blocked by the cleaner thread: 0
Fri Mar 27 11:06:12.519 context: promotion blocked by something uninstrumented: 0
Fri Mar 27 11:06:12.519 context: something uninstrumented blocked by something uninstrumented: 0
Fri Mar 27 11:06:12.519 Memory usage:
Fri Mar 27 11:06:12.525
Fri Mar 27 11:06:12.525 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.525 Crash reason:
Fri Mar 27 11:06:12.525
Fri Mar 27 11:06:12.525 terminate() called
Fri Mar 27 11:06:12.525
Fri Mar 27 11:06:12.525 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.525 Process info:
Fri Mar 27 11:06:12.525
Fri Mar 27 11:06:12.525 OS: Linux Ubuntu 14.04 x86_64
Fri Mar 27 11:06:12.525 NCPU: 8
Fri Mar 27 11:06:12.526 VIRT: 518 MB
Fri Mar 27 11:06:12.526 RES: 518 MB
Fri Mar 27 11:06:12.526 PHYS: 32039 MB
Fri Mar 27 11:06:12.526 RLIMIT_CORE: 0 (soft), unlimited (hard)
Fri Mar 27 11:06:12.526 RLIMIT_CPU: unlimited (soft), unlimited (hard)
Fri Mar 27 11:06:12.526 RLIMIT_DATA: unlimited (soft), unlimited (hard)
Fri Mar 27 11:06:12.526 RLIMIT_FSIZE: unlimited (soft), unlimited (hard)
Fri Mar 27 11:06:12.526 RLIMIT_NOFILE: 1024 (soft), 4096 (hard)
Fri Mar 27 11:06:12.526 RLIMIT_STACK: 8388608 (soft), unlimited (hard)
Fri Mar 27 11:06:12.526 RLIMIT_AS: unlimited (soft), unlimited (hard)
Fri Mar 27 11:06:12.526 _SC_OPEN_MAX: 1024
Fri Mar 27 11:06:12.526 _SC_PAGESIZE: 4096
Fri Mar 27 11:06:12.526 _SC_PHYS_PAGES: 8202028
Fri Mar 27 11:06:12.526 _SC_AVPHYS_PAGES: 4375019
Fri Mar 27 11:06:12.526 _SC_NPROCESSORS_CONF: 8
Fri Mar 27 11:06:12.526 _SC_NPROCESSORS_ONLN: 8
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.526 Parsed server options:
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 dbpath: "atk2/tokumx-atk2.bak/"
Fri Mar 27 11:06:12.526 port: 13456
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.526 Filesystem information:
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 Information for dbpath "/srv/atk2/tokumx-atk2.bak/":
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 type magic: 0x9123683E
Fri Mar 27 11:06:12.526 type: unknown
Fri Mar 27 11:06:12.526 bsize: 4096
Fri Mar 27 11:06:12.526 blocks: 947132928
Fri Mar 27 11:06:12.526 bfree: 606012620
Fri Mar 27 11:06:12.526 bavail: 605308364
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.526 Current operations in progress:
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 { opid: 2, active: true, secs_running: 0, op: "query", ns: "maindb.system.namespaces", query: { listDatabases: 1.0 }, client: "127.0.0.1:57521", desc: "conn1", threadId: "0x7f05ad501700", connectionId: 1, rootTxnid: 631617053, locks: { ^: "R" }, waitingForLock: false, lockStats: { timeLockedMicros: { r: 67, w: 0 }, timeAcquiringMicros: { r: 1, w: 0 } } }
Fri Mar 27 11:06:12.526 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "websvr", threadId: "0x7f058e7ff700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 11:06:12.526 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "clientcursormon", threadId: "0x7f0590bfb700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 11:06:12.526 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "snapshotthread", threadId: "0x7f05913fc700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 11:06:12.526 { opid: 0, active: false, op: "insert", ns: "local.startup_log", insert: {}, client: "0.0.0.0:0", desc: "initandlisten", threadId: "0x7f05ad50a840", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: { r: 87, w: 0 }, timeAcquiringMicros: { r: 1, w: 0 } } }
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.526 OpDebug info:
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 query admin.$cmd query: { listDatabases: 1.0 } ntoreturn:1 keyUpdates:0 locks(micros) r:67 0ms
Fri Mar 27 11:06:12.526 none keyUpdates:0 0ms
Fri Mar 27 11:06:12.526 none keyUpdates:0 0ms
Fri Mar 27 11:06:12.526 none keyUpdates:0 0ms
Fri Mar 27 11:06:12.526 none keyUpdates:0 locks(micros) r:87 0ms
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 ================================================================================
Fri Mar 27 11:06:12.526 Fatal error detected
Fri Mar 27 11:06:12.526 ================================================================================
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 About to gather debugging information, please include all of the following along
Fri Mar 27 11:06:12.526 with logs from other servers in the cluster in a bug report.
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.526 Version info:
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 tokumxVersion: 2.0.0
Fri Mar 27 11:06:12.526 gitVersion: c7f2e017eb71d93ca51d5073eb1570f6c9ce0ba1
Fri Mar 27 11:06:12.526 tokukvVersion: 668f1118593ba0976b6ec68768062f64d418ec83
Fri Mar 27 11:06:12.526 sysInfo: Linux c1ad9bbe58cd 3.11.0-26-generic #45-Ubuntu SMP Tue Jul 15 04:02:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux BOOST_LIB_VERSION=1_49
Fri Mar 27 11:06:12.526 loaderFlags:
Fri Mar 27 11:06:12.526 compilerFlags: -fPIC -fno-strict-aliasing -ggdb -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -pipe -Wnon-virtual-dtor -Woverloaded-virtual -Wno-unused-local-typedefs -fno-builtin-memcmp -O3
Fri Mar 27 11:06:12.526 debug: false
Fri Mar 27 11:06:12.526
Fri Mar 27 11:06:12.526 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.526 Simple stacktrace:
Fri Mar 27 11:06:12.526
0xc4efe3 0x953177 0x955fbb 0x776c97 0x7f05ab7efd40 0x7f05ab7efcc9 0x7f05ab7f30d8 0x776905 0x7f05ac0f9836 0x7f05ac0f88f9 0x7f05ac0f94aa 0x7f05abb8eff3 0x7f05abb8f517 0x86b3a6 0x86b3d5 0x967a31 0x980974 0x980b1d 0xa090bc 0xa2dccf
/opt/tokumx/bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xc4efe3]
/opt/tokumx/bin/mongod() [0x953177]
/opt/tokumx/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x2b) [0x955fbb]
/opt/tokumx/bin/mongod(_ZN5mongo10abruptQuitEi+0xa7) [0x776c97]
/lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f05ab7efd40]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f05ab7efcc9]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f05ab7f30d8]
/opt/tokumx/bin/mongod() [0x776905]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836) [0x7f05ac0f9836]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5d8f9) [0x7f05ac0f88f9]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(__gxx_personality_v0+0x52a) [0x7f05ac0f94aa]
/lib/x86_64-linux-gnu/libgcc_s.so.1(+0xfff3) [0x7f05abb8eff3]
/lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_Resume+0x57) [0x7f05abb8f517]
/opt/tokumx/bin/mongod() [0x86b3a6]
/opt/tokumx/bin/mongod(_ZN5mongo6Client11Transaction5abortEv+0x25) [0x86b3d5]
/opt/tokumx/bin/mongod(_ZN5boost10scoped_ptrIN5mongo6Client11TransactionEED1Ev+0x11) [0x967a31]
/opt/tokumx/bin/mongod(_ZN5mongo13CollectionMap7open_nsERKNS_10StringDataEb+0x394) [0x980974]
/opt/tokumx/bin/mongod(_ZN5mongo13CollectionMap13getCollectionERKNS_10StringDataE+0xad) [0x980b1d]
/opt/tokumx/bin/mongod(_ZN5mongo8Database8diskSizeERmS1_+0x6c) [0xa090bc]
/opt/tokumx/bin/mongod(_ZN5mongo16CmdListDatabases3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x60f) [0xa2dccf]
Fri Mar 27 11:06:12.529
Fri Mar 27 11:06:12.529 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.529 TokuKV engine backtrace:
Fri Mar 27 11:06:12.529
Fri Mar 27 11:06:12.529 Backtrace: (Note: toku_do_assert=0x0x7f05acbe68b0)
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/../lib64/libtokuportability.so(_Z27db_env_do_backtrace_errfuncPFvPKviPKczES0_+0x1f) [0x7f05acbe7c9f]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/../lib64/libtokufractaltree.so(+0x1437d) [0x7f05ace0137d]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod() [0x953248]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo13dumpCrashInfoERKNS_10StringDataE+0x2b) [0x955fbb]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo10abruptQuitEi+0xa7) [0x776c97]
Fri Mar 27 11:06:12.535 /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f05ab7efd40]
Fri Mar 27 11:06:12.535 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f05ab7efcc9]
Fri Mar 27 11:06:12.535 /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f05ab7f30d8]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod() [0x776905]
Fri Mar 27 11:06:12.535 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836) [0x7f05ac0f9836]
Fri Mar 27 11:06:12.535 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5d8f9) [0x7f05ac0f88f9]
Fri Mar 27 11:06:12.535 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(__gxx_personality_v0+0x52a) [0x7f05ac0f94aa]
Fri Mar 27 11:06:12.535 /lib/x86_64-linux-gnu/libgcc_s.so.1(+0xfff3) [0x7f05abb8eff3]
Fri Mar 27 11:06:12.535 /lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_Resume+0x57) [0x7f05abb8f517]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod() [0x86b3a6]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo6Client11Transaction5abortEv+0x25) [0x86b3d5]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5boost10scoped_ptrIN5mongo6Client11TransactionEED1Ev+0x11) [0x967a31]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo13CollectionMap7open_nsERKNS_10StringDataEb+0x394) [0x980974]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo13CollectionMap13getCollectionERKNS_10StringDataE+0xad) [0x980b1d]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo8Database8diskSizeERmS1_+0x6c) [0xa090bc]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo16CmdListDatabases3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x60f) [0xa2dccf]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x39) [0xa1b829]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xcd7) [0xa1efc7]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo12_runCommandsEPKcRKNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x3aa) [0xa1f8ca]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x962c35]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x6e3) [0x9663b3]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x6cb) [0x993a5b]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x8b) [0x78cd3b]
Fri Mar 27 11:06:12.535 /opt/tokumx/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x3dc) [0x89649c]
Fri Mar 27 11:06:12.535 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f05ac7c8182]
Fri Mar 27 11:06:12.535 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f05ab8b400d]
Fri Mar 27 11:06:12.535 BUILD_ID = 107540753
Fri Mar 27 11:06:12.535 time of environment creation: Tue Jan 13 07:26:51 2015
Fri Mar 27 11:06:12.535 time of engine startup: Fri Mar 27 11:06:06 2015
Fri Mar 27 11:06:12.535 time now: Fri Mar 27 11:06:12 2015
Fri Mar 27 11:06:12.535 db opens: 440
Fri Mar 27 11:06:12.535 db closes: 3
Fri Mar 27 11:06:12.535 num open dbs now: 437
Fri Mar 27 11:06:12.535 max open dbs: 440
Fri Mar 27 11:06:12.535 period, in ms, that recovery log is automatically fsynced: 100
Fri Mar 27 11:06:12.535 dictionary inserts: 0
Fri Mar 27 11:06:12.535 dictionary inserts fail: 0
Fri Mar 27 11:06:12.535 dictionary deletes: 1
Fri Mar 27 11:06:12.535 dictionary deletes fail: 0
Fri Mar 27 11:06:12.535 dictionary updates: 0
Fri Mar 27 11:06:12.535 dictionary updates fail: 0
Fri Mar 27 11:06:12.535 dictionary broadcast updates: 0
Fri Mar 27 11:06:12.535 dictionary broadcast updates fail: 0
Fri Mar 27 11:06:12.535 dictionary multi inserts: 1
Fri Mar 27 11:06:12.535 dictionary multi inserts fail: 0
Fri Mar 27 11:06:12.535 dictionary multi deletes: 2
Fri Mar 27 11:06:12.535 dictionary multi deletes fail: 0
Fri Mar 27 11:06:12.535 dictionary updates multi: 0
Fri Mar 27 11:06:12.535 dictionary updates multi fail: 0
Fri Mar 27 11:06:12.535 le: max committed xr: 1
Fri Mar 27 11:06:12.535 le: max provisional xr: 2
Fri Mar 27 11:06:12.535 le: expanded: 0
Fri Mar 27 11:06:12.535 le: max memsize: 883
Fri Mar 27 11:06:12.535 le: size of leafentries before garbage collection (during message application): 0
Fri Mar 27 11:06:12.535 le: size of leafentries after garbage collection (during message application): 0
Fri Mar 27 11:06:12.535 le: size of leafentries before garbage collection (outside message application): 0
Fri Mar 27 11:06:12.535 le: size of leafentries after garbage collection (outside message application): 0
Fri Mar 27 11:06:12.535 checkpoint: period: 60
Fri Mar 27 11:06:12.535 checkpoint: footprint: 0
Fri Mar 27 11:06:12.535 checkpoint: last checkpoint began : Fri Mar 27 11:06:06 2015
Fri Mar 27 11:06:12.535 checkpoint: last complete checkpoint began : Fri Mar 27 11:06:06 2015
Fri Mar 27 11:06:12.535 checkpoint: last complete checkpoint ended: Fri Mar 27 11:06:06 2015
Fri Mar 27 11:06:12.535 checkpoint: time spent during checkpoint (begin and end phases): 0
Fri Mar 27 11:06:12.535 checkpoint: time spent during last checkpoint (begin and end phases): 0
Fri Mar 27 11:06:12.535 checkpoint: last complete checkpoint LSN: 2565863732
Fri Mar 27 11:06:12.535 checkpoint: checkpoints taken : 2
Fri Mar 27 11:06:12.535 checkpoint: checkpoints failed: 0
Fri Mar 27 11:06:12.535 checkpoint: waiters now: 0
Fri Mar 27 11:06:12.535 checkpoint: waiters max: 0
Fri Mar 27 11:06:12.535 checkpoint: non-checkpoint client wait on mo lock: 0
Fri Mar 27 11:06:12.535 checkpoint: non-checkpoint client wait on cs lock: 0
Fri Mar 27 11:06:12.535 checkpoint: checkpoint begin time: 15
Fri Mar 27 11:06:12.535 checkpoint: long checkpoint begin time: 0
Fri Mar 27 11:06:12.535 checkpoint: long checkpoint begin count: 0
Fri Mar 27 11:06:12.535 cachetable: miss: 12
Fri Mar 27 11:06:12.535 cachetable: miss time: 644
Fri Mar 27 11:06:12.535 cachetable: prefetches: 0
Fri Mar 27 11:06:12.535 cachetable: size current: 406407
Fri Mar 27 11:06:12.535 cachetable: size limit: 17334624256
Fri Mar 27 11:06:12.535 cachetable: size writing: 0
Fri Mar 27 11:06:12.535 cachetable: size nonleaf: 0
Fri Mar 27 11:06:12.535 cachetable: size leaf: 403919
Fri Mar 27 11:06:12.535 cachetable: size rollback: 2488
Fri Mar 27 11:06:12.535 cachetable: size cachepressure: 0
Fri Mar 27 11:06:12.535 cachetable: size currently cloned data for checkpoint: 0
Fri Mar 27 11:06:12.535 cachetable: evictions: 0
Fri Mar 27 11:06:12.535 cachetable: cleaner executions: 3
Fri Mar 27 11:06:12.535 cachetable: cleaner period: 2
Fri Mar 27 11:06:12.535 cachetable: cleaner iterations: 5
Fri Mar 27 11:06:12.535 cachetable: number of waits on cache pressure: 0
Fri Mar 27 11:06:12.535 cachetable: time waiting on cache pressure: 0
Fri Mar 27 11:06:12.535 cachetable: number of long waits on cache pressure: 0
Fri Mar 27 11:06:12.535 cachetable: long time waiting on cache pressure: 0
Fri Mar 27 11:06:12.535 locktree: memory size: 576
Fri Mar 27 11:06:12.535 locktree: memory size limit: 1679775334
Fri Mar 27 11:06:12.535 locktree: number of times lock escalation ran: 0
Fri Mar 27 11:06:12.535 locktree: time spent running escalation (seconds): 0.000000
Fri Mar 27 11:06:12.535 locktree: latest post-escalation memory size: 0
Fri Mar 27 11:06:12.535 locktree: number of locktrees open now: 440
Fri Mar 27 11:06:12.535 locktree: number of pending lock requests: 0
Fri Mar 27 11:06:12.535 locktree: number of locktrees eligible for the STO: 5
Fri Mar 27 11:06:12.535 locktree: number of times a locktree ended the STO early: 0
Fri Mar 27 11:06:12.535 locktree: time spent ending the STO early (seconds): 0.000000
Fri Mar 27 11:06:12.535 locktree: number of wait locks: 0
Fri Mar 27 11:06:12.535 locktree: time waiting for locks: 0
Fri Mar 27 11:06:12.535 locktree: number of long wait locks: 0
Fri Mar 27 11:06:12.535 locktree: long time waiting for locks: 0
Fri Mar 27 11:06:12.535 locktree: number of lock timeouts: 0
Fri Mar 27 11:06:12.535 locktree: number of waits on lock escalation: 0
Fri Mar 27 11:06:12.535 locktree: time waiting on lock escalation: 0
Fri Mar 27 11:06:12.535 locktree: number of long waits on lock escalation: 0
Fri Mar 27 11:06:12.535 locktree: long time waiting on lock escalation: 0
Fri Mar 27 11:06:12.535 ft: dictionary updates: 0
Fri Mar 27 11:06:12.535 ft: dictionary broadcast updates: 0
Fri Mar 27 11:06:12.535 ft: descriptor set: 0
Fri Mar 27 11:06:12.535 ft: messages ignored by leaf due to msn: 0
Fri Mar 27 11:06:12.535 ft: total search retries due to TRY_AGAIN: 0
Fri Mar 27 11:06:12.535 ft: searches requiring more tries than the height of the tree: 0
Fri Mar 27 11:06:12.535 ft: searches requiring more tries than the height of the tree plus three: 0
Fri Mar 27 11:06:12.535 ft: leaf nodes flushed to disk (not for checkpoint): 4
Fri Mar 27 11:06:12.535 ft: leaf nodes flushed to disk (not for checkpoint) (bytes): 35840
Fri Mar 27 11:06:12.535 ft: leaf nodes flushed to disk (not for checkpoint) (uncompressed bytes): 187286
Fri Mar 27 11:06:12.535 ft: leaf nodes flushed to disk (not for checkpoint) (seconds): 0.000023
Fri Mar 27 11:06:12.535 ft: nonleaf nodes flushed to disk (not for checkpoint): 0
Fri Mar 27 11:06:12.535 ft: nonleaf nodes flushed to disk (not for checkpoint) (bytes): 0
Fri Mar 27 11:06:12.535 ft: nonleaf nodes flushed to disk (not for checkpoint) (uncompressed bytes): 0
Fri Mar 27 11:06:12.535 ft: nonleaf nodes flushed to disk (not for checkpoint) (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: leaf nodes flushed to disk (for checkpoint): 0
Fri Mar 27 11:06:12.535 ft: leaf nodes flushed to disk (for checkpoint) (bytes): 0
Fri Mar 27 11:06:12.535 ft: leaf nodes flushed to disk (for checkpoint) (uncompressed bytes): 0
Fri Mar 27 11:06:12.535 ft: leaf nodes flushed to disk (for checkpoint) (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: nonleaf nodes flushed to disk (for checkpoint): 0
Fri Mar 27 11:06:12.535 ft: nonleaf nodes flushed to disk (for checkpoint) (bytes): 0
Fri Mar 27 11:06:12.535 ft: nonleaf nodes flushed to disk (for checkpoint) (uncompressed bytes): 0
Fri Mar 27 11:06:12.535 ft: nonleaf nodes flushed to disk (for checkpoint) (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: uncompressed / compressed bytes written (leaf): UNKNOWN STATUS TYPE: 6
Fri Mar 27 11:06:12.535 ft: uncompressed / compressed bytes written (nonleaf): UNKNOWN STATUS TYPE: 6
Fri Mar 27 11:06:12.535 ft: uncompressed / compressed bytes written (overall): UNKNOWN STATUS TYPE: 6
Fri Mar 27 11:06:12.535 ft: nonleaf node partial evictions: 0
Fri Mar 27 11:06:12.535 ft: nonleaf node partial evictions (bytes): 0
Fri Mar 27 11:06:12.535 ft: leaf node partial evictions: 0
Fri Mar 27 11:06:12.535 ft: leaf node partial evictions (bytes): 0
Fri Mar 27 11:06:12.535 ft: leaf node full evictions: 0
Fri Mar 27 11:06:12.535 ft: leaf node full evictions (bytes): 0
Fri Mar 27 11:06:12.535 ft: nonleaf node full evictions: 0
Fri Mar 27 11:06:12.535 ft: nonleaf node full evictions (bytes): 0
Fri Mar 27 11:06:12.535 ft: leaf nodes created: 0
Fri Mar 27 11:06:12.535 ft: nonleaf nodes created: 0
Fri Mar 27 11:06:12.535 ft: leaf nodes destroyed: 0
Fri Mar 27 11:06:12.535 ft: nonleaf nodes destroyed: 0
Fri Mar 27 11:06:12.535 ft: bytes of messages injected at root (all trees): 0
Fri Mar 27 11:06:12.535 ft: bytes of messages flushed from h1 nodes to leaves: 0
Fri Mar 27 11:06:12.535 ft: bytes of messages currently in trees (estimate): 0
Fri Mar 27 11:06:12.535 ft: messages injected at root: 0
Fri Mar 27 11:06:12.535 ft: broadcast messages injected at root: 0
Fri Mar 27 11:06:12.535 ft: basements decompressed as a target of a query: 1
Fri Mar 27 11:06:12.535 ft: basements decompressed for prelocked range: 0
Fri Mar 27 11:06:12.535 ft: basements decompressed for prefetch: 0
Fri Mar 27 11:06:12.535 ft: basements decompressed for write: 0
Fri Mar 27 11:06:12.535 ft: buffers decompressed as a target of a query: 0
Fri Mar 27 11:06:12.535 ft: buffers decompressed for prelocked range: 0
Fri Mar 27 11:06:12.535 ft: buffers decompressed for prefetch: 0
Fri Mar 27 11:06:12.535 ft: buffers decompressed for write: 0
Fri Mar 27 11:06:12.535 ft: pivots fetched for query: 0
Fri Mar 27 11:06:12.535 ft: pivots fetched for query (bytes): 0
Fri Mar 27 11:06:12.535 ft: pivots fetched for query (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: pivots fetched for prefetch: 0
Fri Mar 27 11:06:12.535 ft: pivots fetched for prefetch (bytes): 0
Fri Mar 27 11:06:12.535 ft: pivots fetched for prefetch (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: pivots fetched for write: 4
Fri Mar 27 11:06:12.535 ft: pivots fetched for write (bytes): 35840
Fri Mar 27 11:06:12.535 ft: pivots fetched for write (seconds): 0.000009
Fri Mar 27 11:06:12.535 ft: basements fetched as a target of a query: 0
Fri Mar 27 11:06:12.535 ft: basements fetched as a target of a query (bytes): 0
Fri Mar 27 11:06:12.535 ft: basements fetched as a target of a query (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: basements fetched for prelocked range: 0
Fri Mar 27 11:06:12.535 ft: basements fetched for prelocked range (bytes): 0
Fri Mar 27 11:06:12.535 ft: basements fetched for prelocked range (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: basements fetched for prefetch: 0
Fri Mar 27 11:06:12.535 ft: basements fetched for prefetch (bytes): 0
Fri Mar 27 11:06:12.535 ft: basements fetched for prefetch (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: basements fetched for write: 4
Fri Mar 27 11:06:12.535 ft: basements fetched for write (bytes): 35840
Fri Mar 27 11:06:12.535 ft: basements fetched for write (seconds): 0.000002
Fri Mar 27 11:06:12.535 ft: buffers fetched as a target of a query: 0
Fri Mar 27 11:06:12.535 ft: buffers fetched as a target of a query (bytes): 0
Fri Mar 27 11:06:12.535 ft: buffers fetched as a target of a query (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: buffers fetched for prelocked range: 0
Fri Mar 27 11:06:12.535 ft: buffers fetched for prelocked range (bytes): 0
Fri Mar 27 11:06:12.535 ft: buffers fetched for prelocked range (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: buffers fetched for prefetch: 0
Fri Mar 27 11:06:12.535 ft: buffers fetched for prefetch (bytes): 0
Fri Mar 27 11:06:12.535 ft: buffers fetched for prefetch (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: buffers fetched for write: 0
Fri Mar 27 11:06:12.535 ft: buffers fetched for write (bytes): 0
Fri Mar 27 11:06:12.535 ft: buffers fetched for write (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: leaf compression to memory (seconds): 0.002101
Fri Mar 27 11:06:12.535 ft: leaf serialization to memory (seconds): 0.000085
Fri Mar 27 11:06:12.535 ft: leaf decompression to memory (seconds): 0.000469
Fri Mar 27 11:06:12.535 ft: leaf deserialization to memory (seconds): 0.000131
Fri Mar 27 11:06:12.535 ft: nonleaf compression to memory (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: nonleaf serialization to memory (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: nonleaf decompression to memory (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: nonleaf deserialization to memory (seconds): 0.000000
Fri Mar 27 11:06:12.535 ft: promotion: roots split: 0
Fri Mar 27 11:06:12.535 ft: promotion: leaf roots injected into: 11
Fri Mar 27 11:06:12.535 ft: promotion: h1 roots injected into: 0
Fri Mar 27 11:06:12.535 ft: promotion: injections at depth 0: 0
Fri Mar 27 11:06:12.535 ft: promotion: injections at depth 1: 0
Fri Mar 27 11:06:12.535 ft: promotion: injections at depth 2: 0
Fri Mar 27 11:06:12.535 ft: promotion: injections at depth 3: 0
Fri Mar 27 11:06:12.535 ft: promotion: injections lower than depth 3: 0
Fri Mar 27 11:06:12.535 ft: promotion: stopped because of a nonempty buffer: 0
Fri Mar 27 11:06:12.535 ft: promotion: stopped at height 1: 0
Fri Mar 27 11:06:12.535 ft: promotion: stopped because the child was locked or not at all in memory: 0
Fri Mar 27 11:06:12.535 ft: promotion: stopped because the child was not fully in memory: 0
Fri Mar 27 11:06:12.535 ft: promotion: stopped anyway, after locking the child: 0
Fri Mar 27 11:06:12.535 ft: basement nodes deserialized with fixed-keysize: 5
Fri Mar 27 11:06:12.535 ft: basement nodes deserialized with variable-keysize: 8
Fri Mar 27 11:06:12.535 ft: promotion: succeeded in using the rightmost leaf shortcut: 0
Fri Mar 27 11:06:12.535 ft: promotion: tried the rightmost leaf shorcut but failed (out-of-bounds): 0
Fri Mar 27 11:06:12.535 ft: promotion: tried the rightmost leaf shorcut but failed (child reactive): 0
Fri Mar 27 11:06:12.535 ft flusher: total nodes potentially flushed by cleaner thread: 0
Fri Mar 27 11:06:12.535 ft flusher: height-one nodes flushed by cleaner thread: 0
Fri Mar 27 11:06:12.535 ft flusher: height-greater-than-one nodes flushed by cleaner thread: 0
Fri Mar 27 11:06:12.535 ft flusher: nodes cleaned which had empty buffers: 0
Fri Mar 27 11:06:12.535 ft flusher: nodes dirtied by cleaner thread: 0
Fri Mar 27 11:06:12.535 ft flusher: max bytes in a buffer flushed by cleaner thread: 0
Fri Mar 27 11:06:12.535 ft flusher: min bytes in a buffer flushed by cleaner thread: 18446744073709551615
Fri Mar 27 11:06:12.535 ft flusher: total bytes in buffers flushed by cleaner thread: 0
Fri Mar 27 11:06:12.535 ft flusher: max workdone in a buffer flushed by cleaner thread: 0
Fri Mar 27 11:06:12.535 ft flusher: min workdone in a buffer flushed by cleaner thread: 18446744073709551615
Fri Mar 27 11:06:12.535 ft flusher: total workdone in buffers flushed by cleaner thread: 0
Fri Mar 27 11:06:12.535 ft flusher: times cleaner thread tries to merge a leaf: 0
Fri Mar 27 11:06:12.535 ft flusher: cleaner thread leaf merges in progress: 0
Fri Mar 27 11:06:12.535 ft flusher: cleaner thread leaf merges successful: 0
Fri Mar 27 11:06:12.535 ft flusher: nodes dirtied by cleaner thread leaf merges: 0
Fri Mar 27 11:06:12.535 ft flusher: total number of flushes done by flusher threads or cleaner threads: 0
Fri Mar 27 11:06:12.535 ft flusher: number of in memory flushes: 0
Fri Mar 27 11:06:12.535 ft flusher: number of flushes that read something off disk: 0
Fri Mar 27 11:06:12.535 ft flusher: number of flushes that triggered another flush in child: 0
Fri Mar 27 11:06:12.535 ft flusher: number of flushes that triggered 1 cascading flush: 0
Fri Mar 27 11:06:12.535 ft flusher: number of flushes that triggered 2 cascading flushes: 0
Fri Mar 27 11:06:12.535 ft flusher: number of flushes that triggered 3 cascading flushes: 0
Fri Mar 27 11:06:12.535 ft flusher: number of flushes that triggered 4 cascading flushes: 0
Fri Mar 27 11:06:12.535 ft flusher: number of flushes that triggered 5 cascading flushes: 0
Fri Mar 27 11:06:12.535 ft flusher: number of flushes that triggered over 5 cascading flushes: 0
Fri Mar 27 11:06:12.535 ft flusher: leaf node splits: 0
Fri Mar 27 11:06:12.535 ft flusher: nonleaf node splits: 0
Fri Mar 27 11:06:12.535 ft flusher: leaf node merges: 0
Fri Mar 27 11:06:12.535 ft flusher: nonleaf node merges: 0
Fri Mar 27 11:06:12.535 ft flusher: leaf node balances: 0
Fri Mar 27 11:06:12.535 hot: operations ever started: 0
Fri Mar 27 11:06:12.535 hot: operations successfully completed: 0
Fri Mar 27 11:06:12.535 hot: operations aborted: 0
Fri Mar 27 11:06:12.535 hot: max number of flushes from root ever required to optimize a tree: 0
Fri Mar 27 11:06:12.535 txn: begin: 695
Fri Mar 27 11:06:12.535 txn: begin read only: 2
Fri Mar 27 11:06:12.535 txn: successful commits: 453
Fri Mar 27 11:06:12.535 txn: aborts: 242
Fri Mar 27 11:06:12.535 logger: next LSN: 2565864190
Fri Mar 27 11:06:12.535 logger: writes: 6
Fri Mar 27 11:06:12.535 logger: writes (bytes): 25888
Fri Mar 27 11:06:12.535 logger: writes (uncompressed bytes): 25888
Fri Mar 27 11:06:12.535 logger: writes (seconds): 0.000031
Fri Mar 27 11:06:12.535 logger: number of long logger write operations: 0
Fri Mar 27 11:06:12.535 indexer: number of indexers successfully created: 0
Fri Mar 27 11:06:12.535 indexer: number of calls to toku_indexer_create_indexer() that failed: 0
Fri Mar 27 11:06:12.535 indexer: number of calls to indexer->build() succeeded: 0
Fri Mar 27 11:06:12.535 indexer: number of calls to indexer->build() failed: 0
Fri Mar 27 11:06:12.535 indexer: number of calls to indexer->close() that succeeded: 0
Fri Mar 27 11:06:12.535 indexer: number of calls to indexer->close() that failed: 0
Fri Mar 27 11:06:12.535 indexer: number of calls to indexer->abort(): 0
Fri Mar 27 11:06:12.535 indexer: number of indexers currently in existence: 0
Fri Mar 27 11:06:12.535 indexer: max number of indexers that ever existed simultaneously: 0
Fri Mar 27 11:06:12.535 loader: number of loaders successfully created: 0
Fri Mar 27 11:06:12.535 loader: number of calls to toku_loader_create_loader() that failed: 0
Fri Mar 27 11:06:12.535 loader: number of calls to loader->put() succeeded: 0
Fri Mar 27 11:06:12.535 loader: number of calls to loader->put() failed: 0
Fri Mar 27 11:06:12.535 loader: number of calls to loader->close() that succeeded: 0
Fri Mar 27 11:06:12.535 loader: number of calls to loader->close() that failed: 0
Fri Mar 27 11:06:12.535 loader: number of calls to loader->abort(): 0
Fri Mar 27 11:06:12.535 loader: number of loaders currently in existence: 0
Fri Mar 27 11:06:12.535 loader: max number of loaders that ever existed simultaneously: 0
Fri Mar 27 11:06:12.535 memory: number of malloc operations: 0
Fri Mar 27 11:06:12.535 memory: number of free operations: 0
Fri Mar 27 11:06:12.536 memory: number of realloc operations: 0
Fri Mar 27 11:06:12.536 memory: number of malloc operations that failed: 0
Fri Mar 27 11:06:12.536 memory: number of realloc operations that failed: 0
Fri Mar 27 11:06:12.536 memory: number of bytes requested: 0
Fri Mar 27 11:06:12.536 memory: number of bytes used (requested + overhead): 0
Fri Mar 27 11:06:12.536 memory: number of bytes freed: 0
Fri Mar 27 11:06:12.536 memory: largest attempted allocation size: 0
Fri Mar 27 11:06:12.536 memory: size of the last failed allocation attempt: 0
Fri Mar 27 11:06:12.536 memory: estimated maximum memory footprint: 0
Fri Mar 27 11:06:12.536 memory: mallocator version: tokumx-2.0.0-2-0-gc7f2e017eb71d93ca51d5073eb1570f6c9ce0ba1
Fri Mar 27 11:06:12.536 memory: mmap threshold: 4194304
Fri Mar 27 11:06:12.536 filesystem: ENOSPC redzone state: 0
Fri Mar 27 11:06:12.536 filesystem: threads currently blocked by full disk: 0
Fri Mar 27 11:06:12.536 filesystem: number of operations rejected by enospc prevention (red zone): 0
Fri Mar 27 11:06:12.536 filesystem: most recent disk full: Thu Jan 1 01:00:00 1970
Fri Mar 27 11:06:12.536 filesystem: number of write operations that returned ENOSPC: 0
Fri Mar 27 11:06:12.536 filesystem: fsync time: 554923
Fri Mar 27 11:06:12.536 filesystem: fsync count: 471
Fri Mar 27 11:06:12.536 filesystem: long fsync time: 0
Fri Mar 27 11:06:12.536 filesystem: long fsync count: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by a full fetch: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by a partial fetch: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by a full eviction: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by a partial eviction: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by a message injection: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by a message application: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by a flush: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by a the cleaner thread: 0
Fri Mar 27 11:06:12.536 context: tree traversals blocked by something uninstrumented: 0
Fri Mar 27 11:06:12.536 context: promotion blocked by a full fetch (should never happen): 0
Fri Mar 27 11:06:12.536 context: promotion blocked by a partial fetch (should never happen): 0
Fri Mar 27 11:06:12.536 context: promotion blocked by a full eviction (should never happen): 0
Fri Mar 27 11:06:12.536 context: promotion blocked by a partial eviction (should never happen): 0
Fri Mar 27 11:06:12.536 context: promotion blocked by a message injection: 0
Fri Mar 27 11:06:12.536 context: promotion blocked by a message application: 0
Fri Mar 27 11:06:12.536 context: promotion blocked by a flush: 0
Fri Mar 27 11:06:12.536 context: promotion blocked by the cleaner thread: 0
Fri Mar 27 11:06:12.536 context: promotion blocked by something uninstrumented: 0
Fri Mar 27 11:06:12.536 context: something uninstrumented blocked by something uninstrumented: 0
Fri Mar 27 11:06:12.536 Memory usage:
Fri Mar 27 11:06:12.542
Fri Mar 27 11:06:12.542 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.542 Crash reason:
Fri Mar 27 11:06:12.542
Fri Mar 27 11:06:12.542 Got signal: 6 (Aborted).
Fri Mar 27 11:06:12.542
Fri Mar 27 11:06:12.542 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.542 Process info:
Fri Mar 27 11:06:12.542
Fri Mar 27 11:06:12.542 OS: Linux Ubuntu 14.04 x86_64
Fri Mar 27 11:06:12.542 NCPU: 8
Fri Mar 27 11:06:12.542 VIRT: 518 MB
Fri Mar 27 11:06:12.542 RES: 518 MB
Fri Mar 27 11:06:12.542 PHYS: 32039 MB
Fri Mar 27 11:06:12.542 RLIMIT_CORE: 0 (soft), unlimited (hard)
Fri Mar 27 11:06:12.542 RLIMIT_CPU: unlimited (soft), unlimited (hard)
Fri Mar 27 11:06:12.542 RLIMIT_DATA: unlimited (soft), unlimited (hard)
Fri Mar 27 11:06:12.542 RLIMIT_FSIZE: unlimited (soft), unlimited (hard)
Fri Mar 27 11:06:12.542 RLIMIT_NOFILE: 1024 (soft), 4096 (hard)
Fri Mar 27 11:06:12.542 RLIMIT_STACK: 8388608 (soft), unlimited (hard)
Fri Mar 27 11:06:12.543 RLIMIT_AS: unlimited (soft), unlimited (hard)
Fri Mar 27 11:06:12.543 _SC_OPEN_MAX: 1024
Fri Mar 27 11:06:12.543 _SC_PAGESIZE: 4096
Fri Mar 27 11:06:12.543 _SC_PHYS_PAGES: 8202028
Fri Mar 27 11:06:12.543 _SC_AVPHYS_PAGES: 4375019
Fri Mar 27 11:06:12.543 _SC_NPROCESSORS_CONF: 8
Fri Mar 27 11:06:12.543 _SC_NPROCESSORS_ONLN: 8
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.543 Parsed server options:
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 dbpath: "atk2/tokumx-atk2.bak/"
Fri Mar 27 11:06:12.543 port: 13456
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.543 Filesystem information:
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 Information for dbpath "/srv/atk2/tokumx-atk2.bak/":
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 type magic: 0x9123683E
Fri Mar 27 11:06:12.543 type: unknown
Fri Mar 27 11:06:12.543 bsize: 4096
Fri Mar 27 11:06:12.543 blocks: 947132928
Fri Mar 27 11:06:12.543 bfree: 606012620
Fri Mar 27 11:06:12.543 bavail: 605308364
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.543 Current operations in progress:
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 { opid: 2, active: true, secs_running: 0, op: "query", ns: "maindb.system.namespaces", query: { listDatabases: 1.0 }, client: "127.0.0.1:57521", desc: "conn1", threadId: "0x7f05ad501700", connectionId: 1, rootTxnid: 631617053, locks: { ^: "R" }, waitingForLock: false, lockStats: { timeLockedMicros: { r: 67, w: 0 }, timeAcquiringMicros: { r: 1, w: 0 } } }
Fri Mar 27 11:06:12.543 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "websvr", threadId: "0x7f058e7ff700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 11:06:12.543 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "clientcursormon", threadId: "0x7f0590bfb700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 11:06:12.543 { opid: 0, active: false, op: "none", ns: "", query: {}, desc: "snapshotthread", threadId: "0x7f05913fc700", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: {} } }
Fri Mar 27 11:06:12.543 { opid: 0, active: false, op: "insert", ns: "local.startup_log", insert: {}, client: "0.0.0.0:0", desc: "initandlisten", threadId: "0x7f05ad50a840", rootTxnid: 0, waitingForLock: false, lockStats: { timeLockedMicros: { r: 87, w: 0 }, timeAcquiringMicros: { r: 1, w: 0 } } }
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 --------------------------------------------------------------------------------
Fri Mar 27 11:06:12.543 OpDebug info:
Fri Mar 27 11:06:12.543
Fri Mar 27 11:06:12.543 query admin.$cmd query: { listDatabases: 1.0 } ntoreturn:1 keyUpdates:0 locks(micros) r:67 0ms
Fri Mar 27 11:06:12.543 none keyUpdates:0 0ms
Fri Mar 27 11:06:12.543 none keyUpdates:0 0ms
Fri Mar 27 11:06:12.543 none keyUpdates:0 0ms
Fri Mar 27 11:06:12.543 none keyUpdates:0 locks(micros) r:87 0ms
Fri Mar 27 11:06:12.543
The text was updated successfully, but these errors were encountered:
We have a replica set with 2 replicas + arbiter. This morning, one of the replicas crashed (and won't start again, will file a separate issue), and the other one works mostly fine, but I can't rebuild the replica set because
listDatabasesmakes the thing crash with [1];<rant>even more awesomer, there is absolutely no repair option I can find, because this code can't ever be wrong and filesystems can't ever get corrupted...</rant>
[1]
The text was updated successfully, but these errors were encountered: