2018/03/19 14:58:50 groups.go:86: Current Raft Id: 0 2018/03/19 14:58:50 worker.go:99: Worker listening at address: [::]:7080 2018/03/19 14:58:50 gRPC server started. Listening on port 9080 2018/03/19 14:58:50 HTTP server started. Listening on port 8080 2018/03/19 14:58:50 pool.go:118: == CONNECT ==> Setting dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080 2018/03/19 14:58:50 groups.go:109: Connected to group zero. Assigned group: 2 2018/03/19 14:58:50 pool.go:118: == CONNECT ==> Setting dgraph-server-0.dgraph-server.default.svc.cluster.local:7080 2018/03/19 14:58:50 draft.go:139: Node ID: 2 with GroupID: 2 2018/03/19 14:58:50 node.go:258: Group 2 found 0 entries 2018/03/19 14:58:50 draft.go:755: New Node for group: 2 2018/03/19 14:58:50 raft.go:567: INFO: 2 became follower at term 0 2018/03/19 14:58:50 raft.go:315: INFO: newRaft 2 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0] 2018/03/19 14:58:50 raft.go:567: INFO: 2 became follower at term 1 2018/03/19 14:58:50 node.go:127: Setting conf state to nodes:2 2018/03/19 14:58:50 raft.go:749: INFO: 2 is starting a new election at term 1 2018/03/19 14:58:50 raft.go:580: INFO: 2 became candidate at term 2 2018/03/19 14:58:50 raft.go:664: INFO: 2 received MsgVoteResp from 2 at term 2 2018/03/19 14:58:50 raft.go:621: INFO: 2 became leader at term 2 2018/03/19 14:58:50 node.go:301: INFO: raft.node: 2 elected leader 2 at term 2 2018/03/19 14:58:50 groups.go:336: Serving tablet for: _predicate_ 2018/03/19 14:58:50 mutation.go:159: Done schema update predicate:"_predicate_" value_type:STRING list:true 2018/03/19 14:58:53 pool.go:168: Echo error from dgraph-server-2.dgraph-server.default.svc.cluster.local:7080. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2018/03/19 14:58:53 pool.go:118: == CONNECT ==> Setting dgraph-server-2.dgraph-server.default.svc.cluster.local:7080 2018/03/19 14:58:58 mutation.go:190: Done schema update predicate:"_predicate_" value_type:STRING list:true 2018/03/19 15:00:50 draft.go:696: Writing snapshot at index: 1420, applied mark: 1440 2018/03/19 15:00:57 groups.go:336: Serving tablet for: deleted 2018/03/19 15:00:57 groups.go:336: Serving tablet for: last_update 2018/03/19 15:00:57 groups.go:336: Serving tablet for: name 2018/03/19 15:00:57 groups.go:336: Serving tablet for: receive_newsletter 2018/03/19 15:01:20 draft.go:696: Writing snapshot at index: 8840, applied mark: 8902 2018/03/19 15:01:32 groups.go:476: Unable to sync memberships. Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2018/03/19 15:01:50 draft.go:696: Writing snapshot at index: 14829, applied mark: 15446 2018/03/19 15:02:20 draft.go:696: Writing snapshot at index: 22595, applied mark: 23595 2018/03/19 15:02:50 draft.go:696: Writing snapshot at index: 36804, applied mark: 37142 2018/03/19 15:03:20 draft.go:696: Writing snapshot at index: 53186, applied mark: 54121 2018/03/19 15:03:50 draft.go:696: Writing snapshot at index: 71087, applied mark: 71881 2018/03/19 15:04:20 draft.go:696: Writing snapshot at index: 89031, applied mark: 89881 2018/03/19 15:04:42 groups.go:336: Serving tablet for: _all_ 2018/03/19 15:04:50 draft.go:696: Writing snapshot at index: 103652, applied mark: 103662 2018/03/19 15:05:20 draft.go:696: Writing snapshot at index: 104092, applied mark: 104102 2018/03/19 15:05:50 draft.go:696: Writing snapshot at index: 104296, applied mark: 104525 2018/03/19 15:07:20 draft.go:696: Writing snapshot at index: 118112, applied mark: 118818 2018/03/19 15:07:50 draft.go:696: Writing snapshot at index: 123039, applied mark: 138094 2018/03/19 15:08:20 draft.go:696: Writing snapshot at index: 156791, applied mark: 157085 2018/03/19 15:08:50 draft.go:696: Writing snapshot at index: 174482, applied mark: 175232 2018/03/19 15:08:50 index.go:942: Dropping predicate: [dummy] 2018/03/19 15:08:51 schema.go:73: Deleting schema for predicate: [dummy] 2018/03/19 15:09:20 draft.go:696: Writing snapshot at index: 188541, applied mark: 188594 2018/03/19 15:09:50 draft.go:696: Writing snapshot at index: 205417, applied mark: 206140 2018/03/19 15:10:20 draft.go:696: Writing snapshot at index: 224083, applied mark: 224287 2018/03/19 15:10:50 draft.go:696: Writing snapshot at index: 242037, applied mark: 242856 2018/03/19 15:11:20 draft.go:696: Writing snapshot at index: 259602, applied mark: 260214 2018/03/19 15:11:50 draft.go:696: Writing snapshot at index: 278588, applied mark: 279303 2018/03/19 15:12:20 draft.go:696: Writing snapshot at index: 295619, applied mark: 295629 2018/03/19 15:12:50 draft.go:696: Writing snapshot at index: 313801, applied mark: 314474 2018/03/19 15:13:20 draft.go:696: Writing snapshot at index: 329980, applied mark: 330797 2018/03/19 15:13:50 draft.go:696: Writing snapshot at index: 347685, applied mark: 348668 2018/03/19 15:14:20 draft.go:696: Writing snapshot at index: 365489, applied mark: 366078 2018/03/19 15:14:47 predicate_move.go:281: Move predicate request for pred: [receive_newsletter], src: [2], dst: [3] 2018/03/19 15:14:47 predicate_move.go:164: Sent 21 number of keys for predicate receive_newsletter 2018/03/19 15:14:50 draft.go:696: Writing snapshot at index: 384936, applied mark: 385634 2018/03/19 15:15:20 draft.go:696: Writing snapshot at index: 401954, applied mark: 402815 2018/03/19 15:15:50 draft.go:696: Writing snapshot at index: 419188, applied mark: 420020 2018/03/19 15:16:20 draft.go:696: Writing snapshot at index: 439739, applied mark: 440253 2018/03/19 15:16:50 draft.go:696: Writing snapshot at index: 458349, applied mark: 458949 2018/03/19 15:17:20 draft.go:696: Writing snapshot at index: 478086, applied mark: 478820 2018/03/19 15:17:50 draft.go:696: Writing snapshot at index: 494518, applied mark: 495298 2018/03/19 15:18:20 draft.go:696: Writing snapshot at index: 515052, applied mark: 515652 2018/03/19 15:18:50 draft.go:696: Writing snapshot at index: 534801, applied mark: 535307 2018/03/19 15:19:20 draft.go:696: Writing snapshot at index: 551181, applied mark: 551769 2018/03/19 15:19:50 draft.go:696: Writing snapshot at index: 570212, applied mark: 571055 2018/03/19 15:20:20 draft.go:696: Writing snapshot at index: 580727, applied mark: 587834 2018/03/19 15:20:50 draft.go:696: Writing snapshot at index: 605964, applied mark: 606743 2018/03/19 15:21:20 draft.go:696: Writing snapshot at index: 621159, applied mark: 621224 2018/03/19 15:21:50 draft.go:696: Writing snapshot at index: 633648, applied mark: 634072 2018/03/19 15:22:20 draft.go:696: Writing snapshot at index: 652722, applied mark: 652735 2018/03/19 15:22:47 predicate_move.go:281: Move predicate request for pred: [last_update], src: [2], dst: [3] 2018/03/19 15:22:47 predicate_move.go:164: Sent 27 number of keys for predicate last_update 2018/03/19 15:22:50 draft.go:696: Writing snapshot at index: 662325, applied mark: 662824 2018/03/19 15:23:20 draft.go:696: Writing snapshot at index: 676851, applied mark: 677300 2018/03/19 15:23:50 draft.go:696: Writing snapshot at index: 691679, applied mark: 692440 2018/03/19 15:24:20 draft.go:696: Writing snapshot at index: 698346, applied mark: 709033 2018/03/19 15:24:50 draft.go:696: Writing snapshot at index: 727567, applied mark: 728347 2018/03/19 15:25:20 draft.go:696: Writing snapshot at index: 727600, applied mark: 746530 2018/03/19 15:25:50 draft.go:696: Writing snapshot at index: 761997, applied mark: 762843 2018/03/19 15:26:20 draft.go:696: Writing snapshot at index: 765639, applied mark: 779918 2018/03/19 15:26:50 draft.go:696: Writing snapshot at index: 797945, applied mark: 798684 2018/03/19 15:27:20 draft.go:696: Writing snapshot at index: 812527, applied mark: 813137 2018/03/19 15:27:50 draft.go:696: Writing snapshot at index: 824614, applied mark: 833631 2018/03/19 15:28:20 draft.go:696: Writing snapshot at index: 848877, applied mark: 849691 2018/03/19 15:28:50 draft.go:696: Writing snapshot at index: 867633, applied mark: 868146 2018/03/19 15:29:20 draft.go:696: Writing snapshot at index: 882885, applied mark: 883616 2018/03/19 15:29:50 draft.go:696: Writing snapshot at index: 903013, applied mark: 903770 2018/03/19 15:30:20 draft.go:696: Writing snapshot at index: 920916, applied mark: 921708 2018/03/19 15:30:47 predicate_move.go:281: Move predicate request for pred: [name], src: [2], dst: [3] 2018/03/19 15:30:47 predicate_move.go:164: Sent 33 number of keys for predicate name 2018/03/19 15:30:50 draft.go:696: Writing snapshot at index: 937535, applied mark: 938300 2018/03/19 15:31:20 draft.go:696: Writing snapshot at index: 946779, applied mark: 953768 2018/03/19 15:31:50 draft.go:678: Couldn't take snapshot, txn watermark: [946789], applied watermark: [970449] 2018/03/19 15:32:20 draft.go:696: Writing snapshot at index: 967428, applied mark: 989299 2018/03/19 15:32:50 draft.go:696: Writing snapshot at index: 1006332, applied mark: 1007192 2018/03/19 15:33:20 draft.go:696: Writing snapshot at index: 1022480, applied mark: 1022742 2018/03/19 15:33:50 draft.go:696: Writing snapshot at index: 1036072, applied mark: 1036707 2018/03/19 15:34:20 draft.go:696: Writing snapshot at index: 1053606, applied mark: 1054142 2018/03/19 15:34:50 draft.go:696: Writing snapshot at index: 1073104, applied mark: 1073400 2018/03/19 15:35:20 draft.go:696: Writing snapshot at index: 1089634, applied mark: 1090447 2018/03/19 15:35:50 draft.go:696: Writing snapshot at index: 1103127, applied mark: 1103787 2018/03/19 15:36:20 draft.go:696: Writing snapshot at index: 1116818, applied mark: 1123079 2018/03/19 15:36:50 draft.go:696: Writing snapshot at index: 1143554, applied mark: 1144336 2018/03/19 15:37:20 draft.go:696: Writing snapshot at index: 1160577, applied mark: 1161425 2018/03/19 15:37:50 draft.go:696: Writing snapshot at index: 1175039, applied mark: 1175743 2018/03/19 15:38:20 draft.go:696: Writing snapshot at index: 1196946, applied mark: 1197434 2018/03/19 15:38:47 predicate_move.go:281: Move predicate request for pred: [deleted], src: [2], dst: [3] 2018/03/19 15:38:47 predicate_move.go:164: Sent 33 number of keys for predicate deleted 2018/03/19 15:38:50 draft.go:696: Writing snapshot at index: 1213942, applied mark: 1214368 2018/03/19 15:39:20 draft.go:696: Writing snapshot at index: 1234223, applied mark: 1234787 2018/03/19 15:39:50 draft.go:696: Writing snapshot at index: 1248521, applied mark: 1249262 2018/03/19 15:40:20 draft.go:696: Writing snapshot at index: 1265015, applied mark: 1265770 2018/03/19 15:40:50 draft.go:696: Writing snapshot at index: 1284414, applied mark: 1284467 2018/03/19 15:41:20 draft.go:696: Writing snapshot at index: 1301201, applied mark: 1301934 2018/03/19 15:41:50 draft.go:696: Writing snapshot at index: 1314589, applied mark: 1316091 2018/03/19 15:42:20 draft.go:696: Writing snapshot at index: 1322503, applied mark: 1322513 2018/03/19 15:42:50 draft.go:696: Writing snapshot at index: 1327822, applied mark: 1328728 2018/03/19 15:43:20 draft.go:696: Writing snapshot at index: 1333385, applied mark: 1333395 2018/03/19 15:43:51 draft.go:696: Writing snapshot at index: 1333987, applied mark: 1334225 2018/03/19 15:44:20 draft.go:696: Writing snapshot at index: 1336488, applied mark: 1338095 2018/03/19 15:44:50 draft.go:696: Writing snapshot at index: 1351937, applied mark: 1352699 2018/03/19 15:45:20 draft.go:696: Writing snapshot at index: 1364299, applied mark: 1370820 2018/03/19 15:45:50 draft.go:696: Writing snapshot at index: 1390499, applied mark: 1391313 2018/03/19 15:46:20 draft.go:696: Writing snapshot at index: 1400686, applied mark: 1401365 2018/03/19 15:46:50 draft.go:696: Writing snapshot at index: 1414639, applied mark: 1415280 2018/03/19 15:47:20 draft.go:696: Writing snapshot at index: 1429139, applied mark: 1432175 2018/03/19 15:47:50 draft.go:696: Writing snapshot at index: 1449467, applied mark: 1449916 2018/03/19 15:48:20 draft.go:696: Writing snapshot at index: 1467673, applied mark: 1468056 2018/03/19 15:48:50 draft.go:696: Writing snapshot at index: 1484095, applied mark: 1484868 2018/03/19 15:49:20 draft.go:696: Writing snapshot at index: 1504082, applied mark: 1504704 2018/03/19 15:49:50 draft.go:696: Writing snapshot at index: 1504641, applied mark: 1522618 2018/03/19 15:50:20 draft.go:696: Writing snapshot at index: 1542649, applied mark: 1543304 2018/03/19 15:50:50 draft.go:696: Writing snapshot at index: 1555521, applied mark: 1556254 2018/03/19 15:51:20 draft.go:696: Writing snapshot at index: 1576814, applied mark: 1577460 2018/03/19 15:51:50 draft.go:696: Writing snapshot at index: 1591376, applied mark: 1591787 2018/03/19 15:52:20 draft.go:696: Writing snapshot at index: 1610729, applied mark: 1611526 2018/03/19 15:52:50 draft.go:696: Writing snapshot at index: 1624510, applied mark: 1625020 2018/03/19 15:53:20 draft.go:696: Writing snapshot at index: 1644085, applied mark: 1644789 2018/03/19 15:53:50 draft.go:696: Writing snapshot at index: 1661146, applied mark: 1661637 2018/03/19 15:54:20 draft.go:696: Writing snapshot at index: 1679594, applied mark: 1680276 2018/03/19 15:54:50 draft.go:696: Writing snapshot at index: 1693513, applied mark: 1694315 2018/03/19 15:55:20 draft.go:696: Writing snapshot at index: 1712827, applied mark: 1713577 2018/03/19 15:55:50 draft.go:696: Writing snapshot at index: 1729441, applied mark: 1729992 2018/03/19 15:56:20 draft.go:696: Writing snapshot at index: 1751221, applied mark: 1751627 2018/03/19 15:56:50 draft.go:696: Writing snapshot at index: 1763809, applied mark: 1764367 2018/03/19 15:57:20 draft.go:696: Writing snapshot at index: 1783014, applied mark: 1783658 2018/03/19 15:57:50 draft.go:696: Writing snapshot at index: 1800847, applied mark: 1801344 2018/03/19 15:58:20 draft.go:696: Writing snapshot at index: 1820301, applied mark: 1820779 2018/03/19 15:58:50 draft.go:696: Writing snapshot at index: 1833720, applied mark: 1834664 2018/03/19 15:59:20 draft.go:696: Writing snapshot at index: 1856046, applied mark: 1856712 2018/03/19 15:59:50 draft.go:696: Writing snapshot at index: 1860984, applied mark: 1872233 2018/03/19 16:00:20 draft.go:696: Writing snapshot at index: 1891868, applied mark: 1892708 2018/03/19 16:00:50 draft.go:696: Writing snapshot at index: 1909475, applied mark: 1910131 2018/03/19 16:01:20 draft.go:696: Writing snapshot at index: 1920549, applied mark: 1921295 2018/03/19 16:01:50 draft.go:696: Writing snapshot at index: 1934955, applied mark: 1935799 2018/03/19 16:02:20 draft.go:696: Writing snapshot at index: 1950967, applied mark: 1951456 2018/03/19 16:02:50 draft.go:696: Writing snapshot at index: 1964107, applied mark: 1964741 2018/03/19 16:03:20 draft.go:696: Writing snapshot at index: 1975781, applied mark: 1976561 2018/03/19 16:03:50 draft.go:696: Writing snapshot at index: 1989173, applied mark: 1989604 2018/03/19 16:04:20 draft.go:696: Writing snapshot at index: 2002543, applied mark: 2002719 2018/03/19 16:04:50 draft.go:696: Writing snapshot at index: 2020039, applied mark: 2020690 2018/03/19 16:05:20 draft.go:696: Writing snapshot at index: 2032353, applied mark: 2032846 2018/03/19 16:05:50 draft.go:696: Writing snapshot at index: 2047788, applied mark: 2048553 2018/03/19 16:06:20 draft.go:696: Writing snapshot at index: 2063797, applied mark: 2064152 2018/03/19 16:06:50 draft.go:696: Writing snapshot at index: 2083497, applied mark: 2083918 2018/03/19 16:07:20 draft.go:696: Writing snapshot at index: 2102907, applied mark: 2103153 2018/03/19 16:07:50 draft.go:696: Writing snapshot at index: 2104857, applied mark: 2114457 2018/03/19 16:08:20 draft.go:696: Writing snapshot at index: 2129127, applied mark: 2129848 2018/03/19 16:08:50 draft.go:696: Writing snapshot at index: 2149560, applied mark: 2150639 2018/03/19 16:09:20 draft.go:696: Writing snapshot at index: 2150489, applied mark: 2163071 2018/03/19 16:09:50 draft.go:696: Writing snapshot at index: 2172199, applied mark: 2172855 2018/03/19 16:10:20 draft.go:696: Writing snapshot at index: 2184448, applied mark: 2185197 2018/03/19 16:10:50 draft.go:696: Writing snapshot at index: 2201034, applied mark: 2201882 2018/03/19 16:11:20 draft.go:696: Writing snapshot at index: 2201653, applied mark: 2220718 2018/03/19 16:11:50 draft.go:696: Writing snapshot at index: 2238257, applied mark: 2238901 2018/03/19 16:12:20 draft.go:696: Writing snapshot at index: 2238821, applied mark: 2249433 2018/03/19 16:12:50 draft.go:696: Writing snapshot at index: 2247062, applied mark: 2268224 2018/03/19 16:13:20 draft.go:696: Writing snapshot at index: 2286786, applied mark: 2287374 2018/03/19 16:13:50 draft.go:696: Writing snapshot at index: 2306230, applied mark: 2306818 2018/03/19 16:14:21 draft.go:696: Writing snapshot at index: 2313692, applied mark: 2314508 2018/03/19 16:14:50 draft.go:696: Writing snapshot at index: 2331614, applied mark: 2332397 2018/03/19 16:15:20 draft.go:696: Writing snapshot at index: 2350833, applied mark: 2351629 2018/03/19 16:15:50 draft.go:696: Writing snapshot at index: 2369861, applied mark: 2370630 2018/03/19 16:16:20 draft.go:696: Writing snapshot at index: 2385424, applied mark: 2386189 2018/03/19 16:16:50 draft.go:696: Writing snapshot at index: 2385992, applied mark: 2399819 2018/03/19 16:17:20 draft.go:696: Writing snapshot at index: 2417273, applied mark: 2417791 2018/03/19 16:17:50 draft.go:696: Writing snapshot at index: 2435347, applied mark: 2436141 2018/03/19 16:18:20 draft.go:696: Writing snapshot at index: 2455612, applied mark: 2456545 2018/03/19 16:18:50 draft.go:696: Writing snapshot at index: 2460779, applied mark: 2466355 2018/03/19 16:19:20 draft.go:696: Writing snapshot at index: 2466311, applied mark: 2485124 2018/03/19 16:19:50 draft.go:678: Couldn't take snapshot, txn watermark: [2466321], applied watermark: [2503800] 2018/03/19 16:20:20 draft.go:696: Writing snapshot at index: 2515919, applied mark: 2516716 2018/03/19 16:20:40 pool.go:168: Echo error from dgraph-server-0.dgraph-server.default.svc.cluster.local:7080. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2018/03/19 16:21:20 draft.go:678: Couldn't take snapshot, txn watermark: [2515929], applied watermark: [2518217] 2018/03/19 16:21:50 draft.go:696: Writing snapshot at index: 2516704, applied mark: 2520765 2018/03/19 16:22:20 draft.go:696: Writing snapshot at index: 2523296, applied mark: 2523339 2018/03/19 16:22:50 draft.go:696: Writing snapshot at index: 2525350, applied mark: 2525367 2018/03/19 16:23:20 draft.go:696: Writing snapshot at index: 2527442, applied mark: 2527484 2018/03/19 16:23:50 draft.go:696: Writing snapshot at index: 2529811, applied mark: 2529904 2018/03/19 16:24:20 draft.go:696: Writing snapshot at index: 2536272, applied mark: 2536605 2018/03/19 16:24:50 draft.go:696: Writing snapshot at index: 2548924, applied mark: 2549449 2018/03/19 16:25:20 draft.go:696: Writing snapshot at index: 2560721, applied mark: 2561262 2018/03/19 16:25:50 draft.go:696: Writing snapshot at index: 2581237, applied mark: 2581918 2018/03/19 16:26:20 draft.go:696: Writing snapshot at index: 2597247, applied mark: 2602599 2018/03/19 16:26:50 draft.go:696: Writing snapshot at index: 2621272, applied mark: 2621935 2018/03/19 16:27:20 draft.go:696: Writing snapshot at index: 2635082, applied mark: 2636034 2018/03/19 16:27:50 draft.go:696: Writing snapshot at index: 2654932, applied mark: 2655672 2018/03/19 16:28:20 draft.go:696: Writing snapshot at index: 2675730, applied mark: 2676491 2018/03/19 16:28:50 draft.go:696: Writing snapshot at index: 2694839, applied mark: 2695694 2018/03/19 16:29:20 draft.go:696: Writing snapshot at index: 2706848, applied mark: 2710533 2018/03/19 16:29:50 draft.go:696: Writing snapshot at index: 2722212, applied mark: 2731416 2018/03/19 16:30:20 draft.go:678: Couldn't take snapshot, txn watermark: [2722222], applied watermark: [2749336] 2018/03/19 16:30:50 draft.go:696: Writing snapshot at index: 2766872, applied mark: 2767757 2018/03/19 16:31:21 draft.go:696: Writing snapshot at index: 2781976, applied mark: 2782355 2018/03/19 16:31:50 draft.go:696: Writing snapshot at index: 2798761, applied mark: 2799442 2018/03/19 16:32:20 draft.go:696: Writing snapshot at index: 2816352, applied mark: 2816959 2018/03/19 16:32:50 draft.go:696: Writing snapshot at index: 2834120, applied mark: 2837485 2018/03/19 16:33:20 draft.go:696: Writing snapshot at index: 2856925, applied mark: 2857568 2018/03/19 16:33:50 draft.go:696: Writing snapshot at index: 2871877, applied mark: 2872722 2018/03/19 16:34:20 draft.go:696: Writing snapshot at index: 2889549, applied mark: 2890006 2018/03/19 16:34:50 draft.go:696: Writing snapshot at index: 2905555, applied mark: 2906354 2018/03/19 16:35:20 draft.go:696: Writing snapshot at index: 2922668, applied mark: 2923513 2018/03/19 16:35:50 draft.go:696: Writing snapshot at index: 2934894, applied mark: 2935764 2018/03/19 16:36:20 draft.go:696: Writing snapshot at index: 2954991, applied mark: 2955713 2018/03/19 16:36:50 draft.go:696: Writing snapshot at index: 2975805, applied mark: 2976404 2018/03/19 16:37:20 draft.go:696: Writing snapshot at index: 2996298, applied mark: 2996882 2018/03/19 16:37:50 draft.go:696: Writing snapshot at index: 3010528, applied mark: 3010810 2018/03/19 16:38:20 draft.go:696: Writing snapshot at index: 3030658, applied mark: 3031479 2018/03/19 16:38:50 draft.go:696: Writing snapshot at index: 3051637, applied mark: 3052243 2018/03/19 16:39:20 draft.go:696: Writing snapshot at index: 3052189, applied mark: 3071030 2018/03/19 16:39:50 draft.go:696: Writing snapshot at index: 3086071, applied mark: 3086469 2018/03/19 16:40:20 draft.go:696: Writing snapshot at index: 3100948, applied mark: 3101519 2018/03/19 16:40:50 draft.go:696: Writing snapshot at index: 3113514, applied mark: 3119543 2018/03/19 16:41:20 draft.go:696: Writing snapshot at index: 3125272, applied mark: 3133091 2018/03/19 16:42:09 draft.go:678: Couldn't take snapshot, txn watermark: [3125282], applied watermark: [3133091] 2018/03/19 16:42:25 groups.go:86: Current Raft Id: 2 2018/03/19 16:42:25 gRPC server started. Listening on port 9080 2018/03/19 16:42:25 HTTP server started. Listening on port 8080 2018/03/19 16:42:25 worker.go:99: Worker listening at address: [::]:7080 2018/03/19 16:42:30 pool.go:118: == CONNECT ==> Setting dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080 2018/03/19 16:42:30 groups.go:109: Connected to group zero. Assigned group: 0 2018/03/19 16:42:30 pool.go:118: == CONNECT ==> Setting dgraph-server-0.dgraph-server.default.svc.cluster.local:7080 2018/03/19 16:42:30 pool.go:118: == CONNECT ==> Setting dgraph-server-2.dgraph-server.default.svc.cluster.local:7080 2018/03/19 16:42:30 draft.go:139: Node ID: 2 with GroupID: 2 2018/03/19 16:42:30 node.go:231: Found Snapshot, Metadata: {ConfState:{Nodes:[2] XXX_unrecognized:[]} Index:3125272 Term:2 XXX_unrecognized:[]} 2018/03/19 16:42:30 node.go:246: Found hardstate: {Term:2 Vote:2 Commit:3133102 XXX_unrecognized:[]} 2018/03/19 16:42:30 node.go:258: Group 2 found 7830 entries 2018/03/19 16:42:30 draft.go:744: Restarting node for group: 2 2018/03/19 16:42:30 raft.go:567: INFO: 2 became follower at term 2 2018/03/19 16:42:30 raft.go:315: INFO: newRaft 2 [peers: [2], term: 2, commit: 3133102, applied: 3125272, lastindex: 3133102, lastterm: 2] 2018/03/19 16:42:33 raft.go:749: INFO: 2 is starting a new election at term 2 2018/03/19 16:42:33 raft.go:580: INFO: 2 became candidate at term 3 2018/03/19 16:42:33 raft.go:664: INFO: 2 received MsgVoteResp from 2 at term 3 2018/03/19 16:42:33 raft.go:621: INFO: 2 became leader at term 3 2018/03/19 16:42:33 node.go:301: INFO: raft.node: 2 elected leader 2 at term 3 2018/03/19 16:43:00 draft.go:678: Couldn't take snapshot, txn watermark: [3125282], applied watermark: [3133103] 2018/03/19 16:43:30 draft.go:678: Couldn't take snapshot, txn watermark: [3125282], applied watermark: [3133103] 2018/03/19 16:44:00 draft.go:696: Writing snapshot at index: 3132643, applied mark: 3133105 2018/03/19 16:45:30 draft.go:678: Couldn't take snapshot, txn watermark: [3132653], applied watermark: [3134965] 2018/03/19 16:46:00 draft.go:696: Writing snapshot at index: 3141977, applied mark: 3142273 2018/03/19 16:46:30 draft.go:696: Writing snapshot at index: 3156218, applied mark: 3156520 2018/03/19 16:47:00 draft.go:696: Writing snapshot at index: 3171451, applied mark: 3172187 2018/03/19 16:47:30 draft.go:696: Writing snapshot at index: 3185085, applied mark: 3185855 2018/03/19 16:48:00 draft.go:696: Writing snapshot at index: 3199436, applied mark: 3199993 2018/03/19 16:48:30 draft.go:696: Writing snapshot at index: 3219123, applied mark: 3219606 2018/03/19 16:49:00 draft.go:696: Writing snapshot at index: 3238204, applied mark: 3239099 2018/03/19 16:49:30 draft.go:696: Writing snapshot at index: 3256344, applied mark: 3257108 2018/03/19 16:50:00 draft.go:696: Writing snapshot at index: 3269274, applied mark: 3276322 2018/03/19 16:50:30 draft.go:696: Writing snapshot at index: 3295873, applied mark: 3296638 2018/03/19 16:51:00 draft.go:696: Writing snapshot at index: 3306374, applied mark: 3307111 2018/03/19 16:51:30 draft.go:696: Writing snapshot at index: 3315640, applied mark: 3321512 2018/03/19 16:52:00 draft.go:696: Writing snapshot at index: 3323128, applied mark: 3339080 2018/03/19 16:52:30 draft.go:678: Couldn't take snapshot, txn watermark: [3323138], applied watermark: [3357241] 2018/03/19 16:52:30 index.go:942: Dropping predicate: [name] 2018/03/19 16:52:30 schema.go:73: Deleting schema for predicate: [name] 2018/03/19 16:53:00 draft.go:696: Writing snapshot at index: 3330300, applied mark: 3376362 2018/03/19 16:53:30 draft.go:696: Writing snapshot at index: 3390116, applied mark: 3390907 2018/03/19 16:54:00 draft.go:696: Writing snapshot at index: 3409886, applied mark: 3410492 2018/03/19 16:54:30 draft.go:696: Writing snapshot at index: 3411111, applied mark: 3430086 2018/03/19 16:55:00 draft.go:696: Writing snapshot at index: 3425754, applied mark: 3447371 2018/03/19 16:55:30 draft.go:696: Writing snapshot at index: 3466719, applied mark: 3467437 2018/03/19 16:56:00 draft.go:696: Writing snapshot at index: 3480709, applied mark: 3481484 2018/03/19 16:56:30 draft.go:696: Writing snapshot at index: 3495407, applied mark: 3496233 2018/03/19 16:57:00 draft.go:696: Writing snapshot at index: 3513171, applied mark: 3513884 2018/03/19 16:57:30 draft.go:696: Writing snapshot at index: 3517462, applied mark: 3530678 2018/03/19 16:58:00 draft.go:696: Writing snapshot at index: 3546992, applied mark: 3547805 2018/03/19 16:58:30 draft.go:696: Writing snapshot at index: 3547569, applied mark: 3564836 2018/03/19 16:59:00 draft.go:696: Writing snapshot at index: 3576615, applied mark: 3580745 2018/03/19 16:59:30 draft.go:678: Couldn't take snapshot, txn watermark: [3576625], applied watermark: [3601792] 2018/03/19 17:00:00 draft.go:696: Writing snapshot at index: 3615181, applied mark: 3615778 2018/03/19 17:00:30 draft.go:696: Writing snapshot at index: 3634520, applied mark: 3635030 2018/03/19 17:01:00 draft.go:696: Writing snapshot at index: 3650602, applied mark: 3651325 2018/03/19 17:01:30 draft.go:696: Writing snapshot at index: 3666859, applied mark: 3667510 2018/03/19 17:02:00 draft.go:696: Writing snapshot at index: 3678194, applied mark: 3682536 2018/03/19 17:02:30 draft.go:678: Couldn't take snapshot, txn watermark: [3678204], applied watermark: [3701677] 2018/03/19 17:03:00 draft.go:696: Writing snapshot at index: 3703357, applied mark: 3720565 2018/03/19 17:03:30 draft.go:696: Writing snapshot at index: 3711097, applied mark: 3732741 2018/03/19 17:04:00 draft.go:696: Writing snapshot at index: 3729368, applied mark: 3742329 2018/03/19 17:04:30 draft.go:678: Couldn't take snapshot, txn watermark: [3729378], applied watermark: [3756727] 2018/03/19 17:05:00 draft.go:696: Writing snapshot at index: 3774090, applied mark: 3774908 2018/03/19 17:05:30 draft.go:696: Writing snapshot at index: 3790457, applied mark: 3791112 2018/03/19 17:06:00 draft.go:696: Writing snapshot at index: 3802249, applied mark: 3802799 2018/03/19 17:06:30 draft.go:696: Writing snapshot at index: 3820259, applied mark: 3821006 2018/03/19 17:07:00 draft.go:696: Writing snapshot at index: 3828300, applied mark: 3830878 2018/03/19 17:07:30 draft.go:696: Writing snapshot at index: 3843015, applied mark: 3843047 2018/03/19 17:08:00 draft.go:696: Writing snapshot at index: 3847712, applied mark: 3852811 2018/03/19 17:08:30 draft.go:696: Writing snapshot at index: 3870767, applied mark: 3871150 2018/03/19 17:09:00 draft.go:696: Writing snapshot at index: 3872135, applied mark: 3890442 2018/03/19 17:09:30 draft.go:696: Writing snapshot at index: 3907704, applied mark: 3908380 2018/03/19 17:10:00 draft.go:696: Writing snapshot at index: 3919253, applied mark: 3920025 2018/03/19 17:10:30 draft.go:696: Writing snapshot at index: 3938734, applied mark: 3939368 2018/03/19 17:11:00 draft.go:696: Writing snapshot at index: 3954038, applied mark: 3954861 2018/03/19 17:11:30 draft.go:696: Writing snapshot at index: 3971673, applied mark: 3972498 2018/03/19 17:12:00 draft.go:696: Writing snapshot at index: 3983470, applied mark: 3987350 2018/03/19 17:12:30 draft.go:696: Writing snapshot at index: 4007685, applied mark: 4008096 2018/03/19 17:13:00 draft.go:696: Writing snapshot at index: 4016427, applied mark: 4017162 2018/03/19 17:13:30 draft.go:696: Writing snapshot at index: 4016754, applied mark: 4033772 2018/03/19 17:14:00 draft.go:696: Writing snapshot at index: 4031515, applied mark: 4049605 2018/03/19 17:14:30 draft.go:696: Writing snapshot at index: 4067331, applied mark: 4068121 2018/03/19 17:15:00 draft.go:696: Writing snapshot at index: 4079895, applied mark: 4080800 2018/03/19 17:15:30 draft.go:696: Writing snapshot at index: 4080456, applied mark: 4097299 2018/03/19 17:16:00 draft.go:696: Writing snapshot at index: 4113763, applied mark: 4114530 2018/03/19 17:16:30 draft.go:696: Writing snapshot at index: 4114426, applied mark: 4131865 2018/03/19 17:17:00 draft.go:696: Writing snapshot at index: 4149144, applied mark: 4149872 2018/03/19 17:17:30 draft.go:696: Writing snapshot at index: 4169152, applied mark: 4169834 2018/03/19 17:18:00 draft.go:696: Writing snapshot at index: 4186001, applied mark: 4186634 2018/03/19 17:18:30 draft.go:696: Writing snapshot at index: 4187541, applied mark: 4191682 2018/03/19 17:19:00 draft.go:696: Writing snapshot at index: 4195738, applied mark: 4208967 2018/03/19 17:19:30 draft.go:696: Writing snapshot at index: 4227973, applied mark: 4228411 2018/03/19 17:20:00 draft.go:696: Writing snapshot at index: 4229156, applied mark: 4247217 2018/03/19 17:20:30 draft.go:696: Writing snapshot at index: 4249732, applied mark: 4259551 2018/03/19 17:21:00 draft.go:696: Writing snapshot at index: 4276526, applied mark: 4277303 2018/03/19 17:21:30 draft.go:696: Writing snapshot at index: 4290306, applied mark: 4291034 2018/03/19 17:22:00 draft.go:696: Writing snapshot at index: 4309258, applied mark: 4309799 2018/03/19 17:22:30 draft.go:696: Writing snapshot at index: 4325854, applied mark: 4326421 2018/03/19 17:23:00 draft.go:696: Writing snapshot at index: 4335698, applied mark: 4338517 2018/03/19 17:23:30 draft.go:678: Couldn't take snapshot, txn watermark: [4335708], applied watermark: [4353411] 2018/03/19 17:24:00 draft.go:696: Writing snapshot at index: 4370815, applied mark: 4371233 2018/03/19 17:24:30 draft.go:696: Writing snapshot at index: 4384324, applied mark: 4385126 2018/03/19 17:25:00 draft.go:696: Writing snapshot at index: 4399035, applied mark: 4399472 2018/03/19 17:25:30 draft.go:696: Writing snapshot at index: 4415384, applied mark: 4416200 2018/03/19 17:26:00 draft.go:696: Writing snapshot at index: 4427311, applied mark: 4427981 2018/03/19 17:26:30 draft.go:696: Writing snapshot at index: 4444916, applied mark: 4445320 2018/03/19 17:27:00 draft.go:696: Writing snapshot at index: 4445245, applied mark: 4460392 2018/03/19 17:27:30 draft.go:696: Writing snapshot at index: 4445304, applied mark: 4471900 2018/03/19 17:28:00 draft.go:696: Writing snapshot at index: 4488824, applied mark: 4489610 2018/03/19 17:28:30 draft.go:696: Writing snapshot at index: 4505787, applied mark: 4506152 2018/03/19 17:29:00 draft.go:696: Writing snapshot at index: 4513100, applied mark: 4513665 2018/03/19 17:29:30 draft.go:696: Writing snapshot at index: 4517240, applied mark: 4530655 2018/03/19 17:30:00 draft.go:696: Writing snapshot at index: 4547603, applied mark: 4548101 2018/03/19 17:30:30 draft.go:696: Writing snapshot at index: 4567863, applied mark: 4568358 2018/03/19 17:31:00 draft.go:696: Writing snapshot at index: 4578073, applied mark: 4578762 2018/03/19 17:31:30 draft.go:696: Writing snapshot at index: 4595471, applied mark: 4596206 2018/03/19 17:32:00 draft.go:696: Writing snapshot at index: 4615109, applied mark: 4615832 2018/03/19 17:32:30 draft.go:696: Writing snapshot at index: 4628768, applied mark: 4629321 2018/03/19 17:33:00 draft.go:696: Writing snapshot at index: 4647213, applied mark: 4647778 2018/03/19 17:33:30 draft.go:696: Writing snapshot at index: 4664479, applied mark: 4664917 2018/03/19 17:34:00 draft.go:696: Writing snapshot at index: 4681638, applied mark: 4682564 2018/03/19 17:34:30 draft.go:696: Writing snapshot at index: 4694772, applied mark: 4695638 2018/03/19 17:35:00 draft.go:696: Writing snapshot at index: 4714428, applied mark: 4715130 2018/03/19 17:35:30 draft.go:696: Writing snapshot at index: 4732592, applied mark: 4733398 2018/03/19 17:36:00 draft.go:696: Writing snapshot at index: 4745797, applied mark: 4746479 2018/03/19 17:36:30 draft.go:696: Writing snapshot at index: 4746604, applied mark: 4764849 2018/03/19 17:37:00 draft.go:696: Writing snapshot at index: 4782122, applied mark: 4782713 2018/03/19 17:37:30 draft.go:696: Writing snapshot at index: 4803939, applied mark: 4804458 2018/03/19 17:38:00 draft.go:696: Writing snapshot at index: 4810052, applied mark: 4811034 2018/03/19 17:38:30 draft.go:696: Writing snapshot at index: 4831742, applied mark: 4832450 2018/03/19 17:39:00 draft.go:696: Writing snapshot at index: 4845808, applied mark: 4846323 2018/03/19 17:39:30 draft.go:696: Writing snapshot at index: 4864152, applied mark: 4864898 2018/03/19 17:40:00 draft.go:696: Writing snapshot at index: 4872507, applied mark: 4873394 2018/03/19 17:42:01 pool.go:168: Echo error from dgraph-server-0.dgraph-server.default.svc.cluster.local:7080. Err: rpc error: code = Unavailable desc = transport is closing 2018/03/19 17:42:10 pool.go:168: Echo error from dgraph-server-0.dgraph-server.default.svc.cluster.local:7080. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2018/03/19 17:44:00 draft.go:696: Writing snapshot at index: 4873127, applied mark: 4874988 2018/03/19 17:44:30 draft.go:678: Couldn't take snapshot, txn watermark: [4873137], applied watermark: [4876990] 2018/03/19 17:45:00 draft.go:696: Writing snapshot at index: 4879554, applied mark: 4879564 2018/03/19 17:45:30 draft.go:696: Writing snapshot at index: 4883829, applied mark: 4883892 2018/03/19 17:46:00 draft.go:696: Writing snapshot at index: 4893410, applied mark: 4893520 2018/03/19 17:46:30 draft.go:696: Writing snapshot at index: 4905443, applied mark: 4906117 2018/03/19 17:47:00 draft.go:696: Writing snapshot at index: 4914269, applied mark: 4914785 2018/03/19 17:47:30 draft.go:696: Writing snapshot at index: 4914747, applied mark: 4923944 2018/03/19 17:48:00 draft.go:696: Writing snapshot at index: 4930954, applied mark: 4931628 2018/03/19 17:48:30 draft.go:696: Writing snapshot at index: 4935654, applied mark: 4935771 2018/03/19 17:49:00 draft.go:696: Writing snapshot at index: 4951895, applied mark: 4952646 2018/03/19 17:49:30 draft.go:696: Writing snapshot at index: 4970873, applied mark: 4971275 2018/03/19 17:50:00 draft.go:696: Writing snapshot at index: 4986997, applied mark: 4987663 2018/03/19 17:50:30 draft.go:696: Writing snapshot at index: 5002067, applied mark: 5002755 2018/03/19 17:51:00 draft.go:696: Writing snapshot at index: 5020505, applied mark: 5020539 2018/03/19 17:51:30 draft.go:696: Writing snapshot at index: 5037920, applied mark: 5038438 2018/03/19 17:52:00 draft.go:696: Writing snapshot at index: 5053318, applied mark: 5054143 2018/03/19 17:52:30 draft.go:696: Writing snapshot at index: 5067451, applied mark: 5067676 2018/03/19 17:53:00 draft.go:696: Writing snapshot at index: 5082762, applied mark: 5083304 2018/03/19 17:53:30 draft.go:696: Writing snapshot at index: 5083726, applied mark: 5098257 2018/03/19 17:54:00 draft.go:696: Writing snapshot at index: 5116167, applied mark: 5116707 2018/03/19 17:54:30 draft.go:696: Writing snapshot at index: 5134396, applied mark: 5134779 2018/03/19 17:55:00 draft.go:696: Writing snapshot at index: 5142678, applied mark: 5144129 2018/03/19 17:55:30 draft.go:678: Couldn't take snapshot, txn watermark: [5142688], applied watermark: [5158412] 2018/03/19 17:56:00 draft.go:696: Writing snapshot at index: 5172555, applied mark: 5173245 2018/03/19 17:56:30 draft.go:696: Writing snapshot at index: 5186362, applied mark: 5187004 2018/03/19 17:57:00 draft.go:696: Writing snapshot at index: 5205729, applied mark: 5206048 2018/03/19 17:57:30 draft.go:696: Writing snapshot at index: 5219855, applied mark: 5220470 2018/03/19 17:58:00 draft.go:696: Writing snapshot at index: 5224418, applied mark: 5224455 2018/03/19 17:59:10 pool.go:168: Echo error from dgraph-server-2.dgraph-server.default.svc.cluster.local:7080. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure 2018/03/19 18:03:00 draft.go:696: Writing snapshot at index: 5224438, applied mark: 5231608 2018/03/19 18:03:30 draft.go:696: Writing snapshot at index: 5241269, applied mark: 5241726 2018/03/19 18:04:00 draft.go:696: Writing snapshot at index: 5256849, applied mark: 5257545 2018/03/19 18:04:30 draft.go:696: Writing snapshot at index: 5274225, applied mark: 5274553 2018/03/19 18:05:00 draft.go:696: Writing snapshot at index: 5290143, applied mark: 5290464 2018/03/19 18:05:30 draft.go:696: Writing snapshot at index: 5305328, applied mark: 5305844 2018/03/19 18:06:00 draft.go:696: Writing snapshot at index: 5317917, applied mark: 5318525 2018/03/19 18:06:30 draft.go:696: Writing snapshot at index: 5322626, applied mark: 5334814 2018/03/19 18:07:00 draft.go:696: Writing snapshot at index: 5336477, applied mark: 5350975 2018/03/19 18:07:30 draft.go:678: Couldn't take snapshot, txn watermark: [5336487], applied watermark: [5361361] 2018/03/19 18:08:00 draft.go:696: Writing snapshot at index: 5374038, applied mark: 5374821 2018/03/19 18:08:30 draft.go:696: Writing snapshot at index: 5390401, applied mark: 5391244 2018/03/19 18:09:00 draft.go:696: Writing snapshot at index: 5407405, applied mark: 5408429 2018/03/19 18:09:30 draft.go:696: Writing snapshot at index: 5412856, applied mark: 5423553 2018/03/19 18:10:00 draft.go:696: Writing snapshot at index: 5428396, applied mark: 5438914 2018/03/19 18:10:30 draft.go:678: Couldn't take snapshot, txn watermark: [5428406], applied watermark: [5455120] 2018/03/19 18:11:00 draft.go:696: Writing snapshot at index: 5471288, applied mark: 5472009 2018/03/19 18:11:30 draft.go:696: Writing snapshot at index: 5485297, applied mark: 5485577 2018/03/19 18:12:00 draft.go:696: Writing snapshot at index: 5498887, applied mark: 5499620 2018/03/19 18:12:30 draft.go:696: Writing snapshot at index: 5513111, applied mark: 5513765 2018/03/19 18:13:00 draft.go:696: Writing snapshot at index: 5528837, applied mark: 5529480 2018/03/19 18:13:30 draft.go:696: Writing snapshot at index: 5546336, applied mark: 5547098 2018/03/19 18:14:00 draft.go:696: Writing snapshot at index: 5559574, applied mark: 5559965 2018/03/19 18:14:30 draft.go:696: Writing snapshot at index: 5576636, applied mark: 5577352 2018/03/19 18:15:00 draft.go:696: Writing snapshot at index: 5593093, applied mark: 5593729 2018/03/19 18:15:30 draft.go:696: Writing snapshot at index: 5609608, applied mark: 5610281 2018/03/19 18:16:00 draft.go:696: Writing snapshot at index: 5621911, applied mark: 5622288 2018/03/19 18:16:30 draft.go:696: Writing snapshot at index: 5640244, applied mark: 5640832 2018/03/19 18:17:00 draft.go:696: Writing snapshot at index: 5656956, applied mark: 5657535 2018/03/19 18:17:30 draft.go:696: Writing snapshot at index: 5673590, applied mark: 5673876 2018/03/19 18:18:00 draft.go:696: Writing snapshot at index: 5689904, applied mark: 5690577 2018/03/19 18:18:30 draft.go:696: Writing snapshot at index: 5703646, applied mark: 5704362 2018/03/19 18:19:00 draft.go:696: Writing snapshot at index: 5713512, applied mark: 5720407 2018/03/19 18:19:30 draft.go:678: Couldn't take snapshot, txn watermark: [5713522], applied watermark: [5738764] 2018/03/19 18:20:00 draft.go:696: Writing snapshot at index: 5754179, applied mark: 5754775 2018/03/19 18:20:30 draft.go:696: Writing snapshot at index: 5767918, applied mark: 5768572 2018/03/19 18:21:00 draft.go:696: Writing snapshot at index: 5785129, applied mark: 5785862