forked from matrixorigin/matrixcube
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy path1.log
2729 lines (2728 loc) · 389 KB
/
1.log
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
======== test matrixcube
# go test -v -count=1 -timeout 600s github.com/matrixorigin/matrixcube/storage
go test -v -count=1 -timeout 600s github.com/matrixorigin/matrixcube/raftstore
WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
A future release of golang/protobuf will delete this package,
which has long been excluded from the compatibility promise.
=== RUN TestIssue97
2021/08/20 14:25:04.476486 [info] prophet: prophet logger set
2021/08/20 14:25:04.491698 [info] [raftstore]: start scan gc snap files
2021/08/20 14:25:04.491712 [info] [raftstore]: begin start raftstore
2021/08/20 14:25:04.491714 [info] [raftstore]: begin to start prophet
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-08-20 14:25:04.491766 I | embed: pprof is enabled under /debug/pprof
2021-08-20 14:25:04.491809 I | embed: name = node-0
2021-08-20 14:25:04.491820 I | embed: data dir = /tmp/cube/node-0/prophet
2021-08-20 14:25:04.491822 I | embed: member dir = /tmp/cube/node-0/prophet/member
2021-08-20 14:25:04.491824 I | embed: heartbeat = 100ms
2021-08-20 14:25:04.491825 I | embed: election = 3000ms
2021-08-20 14:25:04.491826 I | embed: snapshot count = 100000
2021-08-20 14:25:04.491833 I | embed: advertise client URLs = http://127.0.0.1:40000
2021-08-20 14:25:04.495499 I | etcdserver: starting member 93ccda9956d988f4 in cluster 6a4ade671189dc2f
raft2021/08/20 14:25:04 INFO: 93ccda9956d988f4 switched to configuration voters=()
raft2021/08/20 14:25:04 INFO: 93ccda9956d988f4 became follower at term 0
raft2021/08/20 14:25:04 INFO: newRaft 93ccda9956d988f4 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2021/08/20 14:25:04 INFO: 93ccda9956d988f4 became follower at term 1
raft2021/08/20 14:25:04 INFO: 93ccda9956d988f4 switched to configuration voters=(10650127570946328820)
2021-08-20 14:25:04.497159 W | auth: simple token is not cryptographically signed
2021-08-20 14:25:04.500562 I | etcdserver: starting server... [version: 3.4.15, cluster version: to_be_decided]
2021-08-20 14:25:04.500673 I | etcdserver: 93ccda9956d988f4 as single-node; fast-forwarding 29 ticks (election ticks 30)
raft2021/08/20 14:25:04 INFO: 93ccda9956d988f4 switched to configuration voters=(10650127570946328820)
2021-08-20 14:25:04.500904 I | etcdserver/membership: added member 93ccda9956d988f4 [http://127.0.0.1:50000] to cluster 6a4ade671189dc2f
2021-08-20 14:25:04.501283 I | embed: listening for peers on 127.0.0.1:50000
raft2021/08/20 14:25:06 INFO: 93ccda9956d988f4 is starting a new election at term 1
raft2021/08/20 14:25:06 INFO: 93ccda9956d988f4 became candidate at term 2
raft2021/08/20 14:25:06 INFO: 93ccda9956d988f4 received MsgVoteResp from 93ccda9956d988f4 at term 2
raft2021/08/20 14:25:06 INFO: 93ccda9956d988f4 became leader at term 2
raft2021/08/20 14:25:06 INFO: raft.node: 93ccda9956d988f4 elected leader 93ccda9956d988f4 at term 2
2021-08-20 14:25:06.496592 I | etcdserver: published {Name:node-0 ClientURLs:[http://127.0.0.1:40000]} to cluster 6a4ade671189dc2f
2021-08-20 14:25:06.496615 I | embed: ready to serve client requests
2021-08-20 14:25:06.496641 I | etcdserver: setting up the initial cluster version to 3.4
2021/08/20 14:25:06.496659 [info] prophet: create etcd v3 client with endpoints [http://127.0.0.1:40000]
2021-08-20 14:25:06.496922 N | embed: serving insecure client requests on 127.0.0.1:40000, this is strongly discouraged!
2021-08-20 14:25:06.497160 N | etcdserver/membership: set the initial cluster version to 3.4
2021-08-20 14:25:06.497211 I | etcdserver/api: enabled capabilities for version 3.4
2021/08/20 14:25:06.498565 [info] prophet: net application started
2021/08/20 14:25:06.498574 [info] prophet: start system time monitor
2021/08/20 14:25:06.498945 [info] prophet: [prophet-leader/node-0]/loop: current leader is []
2021/08/20 14:25:06.499151 [info] prophet: [prophet-leader/node-0]/campaign: load expect leader: []
2021/08/20 14:25:06.499160 [info] prophet: [prophet-leader/node-0]/campaign: start
2021/08/20 14:25:06.499507 [info] prophet: [prophet-leader/node-0]/campaign: grant lease ok
2021/08/20 14:25:06.499891 [info] prophet: [prophet-leader/node-0]/campaign: completed, start keepalive
2021/08/20 14:25:06.499902 [info] prophet: ********node-0 become to leader now********
2021/08/20 14:25:06.501106 [info] prophet: load 0 containers, cost 925.1µs
2021/08/20 14:25:06.501372 [info] prophet: load 0 resources, cost 253.4µs
2021/08/20 14:25:06.501995 [info] prophet: client start init connection to leader 127.0.0.1:30000
2021/08/20 14:25:06.502024 [info] prophet: coordinator starts to collect cluster information
2021/08/20 14:25:06.502027 [info] prophet: coordinator has finished cluster information preparation
2021/08/20 14:25:06.502028 [info] prophet: coordinator starts to run schedulers
2021/08/20 14:25:06.502045 [info] prophet: client start connect to leader 127.0.0.1:30000
2021/08/20 14:25:06.502053 [info] prophet: client read loop started
2021/08/20 14:25:06.502061 [info] prophet: client started
2021/08/20 14:25:06.502064 [info] prophet: client read loop actived, ready to read from leader
2021/08/20 14:25:06.502059 [info] prophet: session 1[127.0.0.1:38468] connected
2021/08/20 14:25:06.502068 [info] [raftstore]: *********Become prophet leader*********
2021/08/20 14:25:06.502070 [info] [raftstore]: begin to bootstrap the cluster
2021/08/20 14:25:06.502113 [info] [raftstore]: raftstore init with store raftAddr:"127.0.0.1:10000" clientAddr:"127.0.0.1:20000" labels:<key:"c" value:"0" > startTime:1629440706 deployPath:"not set"
2021/08/20 14:25:06.502127 [info] [raftstore]: begin to create local store metadata
2021/08/20 14:25:06.502577 [info] prophet: load 0 jobs
2021/08/20 14:25:06.502834 [info] [raftstore]: create local store with id 1
2021/08/20 14:25:06.503223 [info] [raftstore]: the cluster already bootstrap: false
2021/08/20 14:25:06.503237 [info] [raftstore]: begin to bootstrap the cluster with init shards
2021/08/20 14:25:06.503425 [info] prophet: coordinator begins to actively drive push operator
2021/08/20 14:25:06.503441 [info] prophet: coordinator starts patrol resources
2021/08/20 14:25:06.503461 [info] [raftstore]: create init shard id:2 epoch:<confVer:1 version:1 > peers:<id:3 containerID:1 >
2021/08/20 14:25:06.505954 [info] prophet: container 1 limit added
2021/08/20 14:25:06.506023 [info] [raftstore]: prophet started
2021/08/20 14:25:06.506074 [info] [transport]: net application started
2021/08/20 14:25:06.506086 [info] [raftstore]: start listen at 127.0.0.1:10000 for raft
2021/08/20 14:25:06.506088 [info] prophet: container 1 heartbeat stream binded
2021/08/20 14:25:06.506095 [info] [raftstore]: raft worker 0/0 start
2021/08/20 14:25:06.506089 [info] [raftstore]: raft shards workers started
2021/08/20 14:25:06.506150 [info] [raftstore]: shard 2 init with raft local state {HardState:{Term:5 Vote:0 Commit:5 XXX_unrecognized:[]} LastIndex:5 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:06.506176 [info] [raftstore]: shard 2 init raft apply state, state=<{AppliedIndex:5 TruncatedState:{Index:5 Term:5 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:06.506186 [info] [raftstore]: shard 2 init last term, last term=<5>
{"level":"info","ts":"2021-08-20T14:25:06.506+0800","caller":"raft/raft.go:1550","msg":"3 switched to configuration voters=(3)"}
{"level":"info","ts":"2021-08-20T14:25:06.506+0800","caller":"raft/raft.go:711","msg":"3 became follower at term 5"}
{"level":"info","ts":"2021-08-20T14:25:06.506+0800","caller":"raft/raft.go:388","msg":"newRaft 3 [peers: [3], term: 5, commit: 5, applied: 5, lastindex: 5, lastterm: 5]"}
2021/08/20 14:25:06.506244 [info] [raftstore]: shard 2 try to campaign leader, because only self
{"level":"info","ts":"2021-08-20T14:25:06.506+0800","caller":"raft/raft.go:934","msg":"3 is starting a new election at term 5"}
{"level":"info","ts":"2021-08-20T14:25:06.506+0800","caller":"raft/raft.go:724","msg":"3 became candidate at term 6"}
{"level":"info","ts":"2021-08-20T14:25:06.506+0800","caller":"raft/raft.go:835","msg":"3 received MsgVoteResp from 3 at term 6"}
{"level":"info","ts":"2021-08-20T14:25:06.506+0800","caller":"raft/raft.go:776","msg":"3 became leader at term 6"}
2021/08/20 14:25:06.506282 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 add registration job by startShards
2021/08/20 14:25:06.506294 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 add registration job by startShards, completed
2021/08/20 14:25:06.506315 [info] [raftstore]: shard 2 peer 3 raft 3 added, epoch {ConfVer:1 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}], raft worker 0, apply worker apply-0-0
2021/08/20 14:25:06.506326 [info] [raftstore]: starts with 1 shards, including 0 tombstones and 0 applying shards
2021/08/20 14:25:06.506327 [info] [raftstore]: cleanup possible garbage data complete
2021/08/20 14:25:06.506328 [info] [raftstore]: shards started
2021/08/20 14:25:06.506330 [info] [raftstore]: shard timer based tasks started
2021/08/20 14:25:06.506337 [info] [raftstore]: net application started
2021/08/20 14:25:06.506344 [info] [raftstore]: start listen at 127.0.0.1:20000 for client
2021/08/20 14:25:06.506351 [info] [raftstore]: router started
2021/08/20 14:25:06.506388 [info] prophet: client start init connection to leader 127.0.0.1:30000
2021/08/20 14:25:06.506389 [info] [raftstore]: router event loop task started
2021/08/20 14:25:06.506471 [info] prophet: session 2[127.0.0.1:38470] connected
2021/08/20 14:25:06.506475 [info] prophet: client start connect to leader 127.0.0.1:30000
2021/08/20 14:25:06.506476 [info] prophet: watcher init leader connection 127.0.0.1:30000 succeed
2021/08/20 14:25:06.506499 [info] prophet: new watcher 127.0.0.1:38470 added
2021/08/20 14:25:06.506548 [info] [raftstore]: start init event
2021/08/20 14:25:06.516860 [info] [raftstore]: shard 2 ********become leader now********
2021/08/20 14:25:06.518619 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 AdvanceApply with 6
2021/08/20 14:25:06.626911 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 AdvanceApply with 7
2021/08/20 14:25:06.716913 [info] [raftstore]: shard 2 start split check job from [122 0 0 0 0 0 0 0 0] to [122 0 0 0 0 0 0 0 1]
2021/08/20 14:25:06.716985 [info] [raftstore]: shard 2 try to split, size 40 bytes splitKeys [[122 0 0 0 0 0 0 0 0 107 101 121 50]]
2021/08/20 14:25:06.717088 [info] prophet: alloc ids [6] for resource 5 split
2021/08/20 14:25:06.736857 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 AdvanceApply with 8
2021/08/20 14:25:06.736902 [info] [raftstore]: shard 2 update to {ID:2 Start:[] End:[107 101 121 50] Epoch:{ConfVer:1 Version:2 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} State:Running Peers:[{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] DisableSplit:false Group:0 Unique: RuleGroups:[] Data:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} by post applt split
2021/08/20 14:25:06.736925 [info] [raftstore]: shard 5 init with raft local state {HardState:{Term:5 Vote:0 Commit:5 XXX_unrecognized:[]} LastIndex:5 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:06.736942 [info] [raftstore]: shard 5 init raft apply state, state=<{AppliedIndex:5 TruncatedState:{Index:5 Term:5 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:06.736943 [info] [raftstore]: shard 5 init last term, last term=<5>
{"level":"info","ts":"2021-08-20T14:25:06.736+0800","caller":"raft/raft.go:1550","msg":"6 switched to configuration voters=(6)"}
{"level":"info","ts":"2021-08-20T14:25:06.737+0800","caller":"raft/raft.go:711","msg":"6 became follower at term 5"}
{"level":"info","ts":"2021-08-20T14:25:06.737+0800","caller":"raft/raft.go:388","msg":"newRaft 6 [peers: [6], term: 5, commit: 5, applied: 5, lastindex: 5, lastterm: 5]"}
2021/08/20 14:25:06.737018 [info] [raftstore]: shard 5 try to campaign leader, because only self
{"level":"info","ts":"2021-08-20T14:25:06.737+0800","caller":"raft/raft.go:934","msg":"6 is starting a new election at term 5"}
{"level":"info","ts":"2021-08-20T14:25:06.737+0800","caller":"raft/raft.go:724","msg":"6 became candidate at term 6"}
{"level":"info","ts":"2021-08-20T14:25:06.737+0800","caller":"raft/raft.go:835","msg":"6 received MsgVoteResp from 6 at term 6"}
{"level":"info","ts":"2021-08-20T14:25:06.737+0800","caller":"raft/raft.go:776","msg":"6 became leader at term 6"}
2021/08/20 14:25:06.737049 [error] [raftstore]: >>>>>>>>>>>> shard 5 peer 6 add registration job by doApplySplit
2021/08/20 14:25:06.737067 [error] [raftstore]: >>>>>>>>>>>> shard 5 peer 6 add registration job by doApplySplit, completed
2021/08/20 14:25:06.737101 [info] [raftstore]: shard 5 peer 6 raft 6 added, epoch {ConfVer:1 Version:2 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:6 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}], raft worker 0, apply worker apply-0-0
2021/08/20 14:25:06.737125 [info] [raftstore]: shard 2 new shard added, new shards [{ID:5 Start:[107 101 121 50] End:[] Epoch:{ConfVer:1 Version:2 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} State:Running Peers:[{ID:6 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] DisableSplit:false Group:0 Unique: RuleGroups:[] Data:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:06.737151 [info] [raftstore]: shard 5 ********become leader now********
2021/08/20 14:25:06.737223 [info] prophet: resource 2 version changed from 1 to 2
2021/08/20 14:25:06.738591 [error] [raftstore]: >>>>>>>>>>>> shard 5 peer 6 AdvanceApply with 6
2021/08/20 14:25:06.806614 [info] [raftstore]: shard 5 start split check job from [122 0 0 0 0 0 0 0 0 107 101 121 50] to [122 0 0 0 0 0 0 0 1]
2021/08/20 14:25:06.836643 [error] [raftstore]: >>>>>>>>>>>> shard 5 peer 6 AdvanceApply with 7
2021/08/20 14:25:06.916304 [info] [raftstore]: shard 5 start split check job from [122 0 0 0 0 0 0 0 0 107 101 121 50] to [122 0 0 0 0 0 0 0 1]
2021/08/20 14:25:06.916328 [info] [raftstore]: shard 5 try to split, size 40 bytes splitKeys [[122 0 0 0 0 0 0 0 0 107 101 121 51]]
2021/08/20 14:25:06.916695 [info] prophet: alloc ids [8] for resource 7 split
2021/08/20 14:25:06.926270 [error] [raftstore]: >>>>>>>>>>>> shard 5 peer 6 AdvanceApply with 8
2021/08/20 14:25:06.926292 [info] [raftstore]: shard 5 update to {ID:5 Start:[107 101 121 50] End:[107 101 121 51] Epoch:{ConfVer:1 Version:3 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} State:Running Peers:[{ID:6 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] DisableSplit:false Group:0 Unique: RuleGroups:[] Data:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} by post applt split
2021/08/20 14:25:06.926304 [info] [raftstore]: shard 7 init with raft local state {HardState:{Term:5 Vote:0 Commit:5 XXX_unrecognized:[]} LastIndex:5 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:06.926308 [info] [raftstore]: shard 7 init raft apply state, state=<{AppliedIndex:5 TruncatedState:{Index:5 Term:5 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:06.926309 [info] [raftstore]: shard 7 init last term, last term=<5>
{"level":"info","ts":"2021-08-20T14:25:06.926+0800","caller":"raft/raft.go:1550","msg":"8 switched to configuration voters=(8)"}
{"level":"info","ts":"2021-08-20T14:25:06.926+0800","caller":"raft/raft.go:711","msg":"8 became follower at term 5"}
{"level":"info","ts":"2021-08-20T14:25:06.926+0800","caller":"raft/raft.go:388","msg":"newRaft 8 [peers: [8], term: 5, commit: 5, applied: 5, lastindex: 5, lastterm: 5]"}
2021/08/20 14:25:06.926366 [info] [raftstore]: shard 7 try to campaign leader, because only self
{"level":"info","ts":"2021-08-20T14:25:06.926+0800","caller":"raft/raft.go:934","msg":"8 is starting a new election at term 5"}
{"level":"info","ts":"2021-08-20T14:25:06.926+0800","caller":"raft/raft.go:724","msg":"8 became candidate at term 6"}
{"level":"info","ts":"2021-08-20T14:25:06.926+0800","caller":"raft/raft.go:835","msg":"8 received MsgVoteResp from 8 at term 6"}
{"level":"info","ts":"2021-08-20T14:25:06.926+0800","caller":"raft/raft.go:776","msg":"8 became leader at term 6"}
2021/08/20 14:25:06.926397 [error] [raftstore]: >>>>>>>>>>>> shard 7 peer 8 add registration job by doApplySplit
2021/08/20 14:25:06.926415 [error] [raftstore]: >>>>>>>>>>>> shard 7 peer 8 add registration job by doApplySplit, completed
2021/08/20 14:25:06.926421 [info] [raftstore]: shard 7 peer 8 raft 8 added, epoch {ConfVer:1 Version:3 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:8 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}], raft worker 0, apply worker apply-0-0
2021/08/20 14:25:06.926435 [info] [raftstore]: shard 5 new shard added, new shards [{ID:7 Start:[107 101 121 51] End:[] Epoch:{ConfVer:1 Version:3 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} State:Running Peers:[{ID:8 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] DisableSplit:false Group:0 Unique: RuleGroups:[] Data:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:06.926444 [info] [raftstore]: shard 7 ********become leader now********
2021/08/20 14:25:06.926545 [info] prophet: resource 5 version changed from 2 to 3
2021/08/20 14:25:06.927822 [error] [raftstore]: >>>>>>>>>>>> shard 7 peer 8 AdvanceApply with 6
2021/08/20 14:25:06.946792 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 AdvanceApply with 9
2021/08/20 14:25:06.949075 [error] prophet: client read resp from leader 127.0.0.1:30000 failed with read tcp 127.0.0.1:38468->127.0.0.1:30000: use of closed network connection
2021/08/20 14:25:06.949085 [error] prophet: client read loop stopped
2021/08/20 14:25:06.949102 [error] prophet: watcher read events failed with EOF
2021/08/20 14:25:06.949105 [error] prophet: reset watcher conn failed with <nil>, retry later
2021/08/20 14:25:06.949105 [error] prophet: session 2[127.0.0.1:38470] read failed with read tcp 127.0.0.1:30000->127.0.0.1:38470: use of closed network connection
2021/08/20 14:25:06.949292 [info] prophet: [prophet-leader/node-0]/lease: closed
2021/08/20 14:25:06.949331 [info] prophet: patrol resources has been stopped
2021/08/20 14:25:06.949346 [info] prophet: drive push operator has been stopped
2021/08/20 14:25:06.949348 [info] prophet: [prophet-leader/node-0]/keepalive: exit with context done
2021/08/20 14:25:06.949349 [info] prophet: coordinator is stopping
2021/08/20 14:25:06.949353 [info] prophet: coordinator has been stopped
2021/08/20 14:25:06.949356 [info] prophet: [prophet-leader/node-0]/lease: exit with context done
2021/08/20 14:25:06.949365 [error] prophet: [prophet-leader/node-0]/lease: keep lease exit
2021-08-20 14:25:06.949364 I | etcdserver: skipped leadership transfer for single voting member cluster
2021/08/20 14:25:06.956527 [info] [raftstore]: shard 2 handle serve raft stopped
2021/08/20 14:25:06.956542 [info] [raftstore]: shard 5 handle serve raft stopped
2021/08/20 14:25:06.956543 [info] [raftstore]: shard 7 handle serve raft stopped
2021/08/20 14:25:06.956562 [info] [raftstore]: timer based tasks stopped
2021/08/20 14:25:06.956562 [info] [raftstore]: handle resource heartbeat resp task stopped
2021/08/20 14:25:06.956568 [info] [raftstore]: raft worker worker 0/0 exit
2021/08/20 14:25:06.956564 [info] [raftstore]: router event loop task stopped
2021/08/20 14:25:06.956620 [info] [transport]: send snapshot worker stopped
2021/08/20 14:25:06.956628 [info] [transport]: transfer stopped
2021/08/20 14:25:06.956632 [info] [transport]: send snapshot worker stopped
2021/08/20 14:25:06.956632 [info] [transport]: send snapshot worker stopped
2021/08/20 14:25:06.956640 [info] [transport]: send snapshot worker stopped
2021/08/20 14:25:06.956641 [info] [transport]: send snapshot worker stopped
2021/08/20 14:25:06.956644 [info] [transport]: send raft worker stopped
2021/08/20 14:25:06.956644 [info] [transport]: send snapshot worker stopped
2021/08/20 14:25:06.956647 [info] [transport]: send snapshot worker stopped
2021/08/20 14:25:06.956649 [info] [transport]: send snapshot worker stopped
2021/08/20 14:25:06.958951 [info] prophet: prophet logger set
2021/08/20 14:25:06.973147 [info] [raftstore]: begin start raftstore
2021/08/20 14:25:06.973159 [info] [raftstore]: begin to start prophet
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021/08/20 14:25:06.973198 [info] [raftstore]: start scan gc snap files
2021-08-20 14:25:06.973200 I | embed: pprof is enabled under /debug/pprof
2021-08-20 14:25:06.973270 I | embed: name = node-0
2021-08-20 14:25:06.973282 I | embed: data dir = /tmp/cube/node-0/prophet
2021-08-20 14:25:06.973285 I | embed: member dir = /tmp/cube/node-0/prophet/member
2021-08-20 14:25:06.973287 I | embed: heartbeat = 100ms
2021-08-20 14:25:06.973289 I | embed: election = 3000ms
2021-08-20 14:25:06.973291 I | embed: snapshot count = 100000
2021-08-20 14:25:06.973295 I | embed: advertise client URLs = http://127.0.0.1:40000
2021-08-20 14:25:06.973297 I | embed: initial advertise peer URLs = http://127.0.0.1:50000
2021-08-20 14:25:06.973301 I | embed: initial cluster =
2021/08/20 14:25:06.973407 [info] [raftstore]: shard 2 raft tick stopped
2021-08-20 14:25:06.973649 I | etcdserver: restarting member 93ccda9956d988f4 in cluster 6a4ade671189dc2f at commit index 20
{"level":"info","ts":"2021-08-20T14:25:06.973+0800","caller":"raft/raft.go:1550","msg":"93ccda9956d988f4 switched to configuration voters=()"}
{"level":"info","ts":"2021-08-20T14:25:06.973+0800","caller":"raft/raft.go:711","msg":"93ccda9956d988f4 became follower at term 2"}
{"level":"info","ts":"2021-08-20T14:25:06.973+0800","caller":"raft/raft.go:388","msg":"newRaft 93ccda9956d988f4 [peers: [], term: 2, commit: 20, applied: 0, lastindex: 20, lastterm: 2]"}
2021-08-20 14:25:06.974434 W | auth: simple token is not cryptographically signed
2021-08-20 14:25:06.976915 I | etcdserver: starting server... [version: 3.4.15, cluster version: to_be_decided]
{"level":"info","ts":"2021-08-20T14:25:06.977+0800","caller":"raft/raft.go:1550","msg":"93ccda9956d988f4 switched to configuration voters=(10650127570946328820)"}
2021-08-20 14:25:06.977127 I | etcdserver/membership: added member 93ccda9956d988f4 [http://127.0.0.1:50000] to cluster 6a4ade671189dc2f
2021-08-20 14:25:06.977174 N | etcdserver/membership: set the initial cluster version to 3.4
2021-08-20 14:25:06.977214 I | embed: listening for peers on 127.0.0.1:50000
2021/08/20 14:25:07.024343 [info] [raftstore]: shard 5 raft tick stopped
2021/08/20 14:25:07.024357 [info] [raftstore]: shard 7 raft tick stopped
2021/08/20 14:25:07.049646 [info] prophet: [prophet-leader/node-0]/loop: exit
{"level":"info","ts":"2021-08-20T14:25:10.674+0800","caller":"raft/raft.go:934","msg":"93ccda9956d988f4 is starting a new election at term 2"}
{"level":"info","ts":"2021-08-20T14:25:10.674+0800","caller":"raft/raft.go:724","msg":"93ccda9956d988f4 became candidate at term 3"}
{"level":"info","ts":"2021-08-20T14:25:10.674+0800","caller":"raft/raft.go:835","msg":"93ccda9956d988f4 received MsgVoteResp from 93ccda9956d988f4 at term 3"}
{"level":"info","ts":"2021-08-20T14:25:10.674+0800","caller":"raft/raft.go:776","msg":"93ccda9956d988f4 became leader at term 3"}
{"level":"info","ts":"2021-08-20T14:25:10.674+0800","caller":"raft/node.go:325","msg":"raft.node: 93ccda9956d988f4 elected leader 93ccda9956d988f4 at term 3"}
2021-08-20 14:25:10.676008 I | etcdserver: published {Name:node-0 ClientURLs:[http://127.0.0.1:40000]} to cluster 6a4ade671189dc2f
2021/08/20 14:25:10.676023 [info] prophet: create etcd v3 client with endpoints [http://127.0.0.1:40000]
2021-08-20 14:25:10.676024 I | embed: ready to serve client requests
2021-08-20 14:25:10.676697 N | embed: serving insecure client requests on 127.0.0.1:40000, this is strongly discouraged!
2021/08/20 14:25:10.677755 [info] prophet: net application started
2021/08/20 14:25:10.677790 [info] prophet: start system time monitor
2021/08/20 14:25:10.677981 [info] prophet: [prophet-leader/node-0]/loop: current leader is []
2021/08/20 14:25:10.678181 [info] prophet: [prophet-leader/node-0]/campaign: load expect leader: []
2021/08/20 14:25:10.678190 [info] prophet: [prophet-leader/node-0]/campaign: start
2021/08/20 14:25:10.678458 [info] prophet: [prophet-leader/node-0]/campaign: grant lease ok
2021/08/20 14:25:10.678997 [info] prophet: [prophet-leader/node-0]/campaign: completed, start keepalive
2021/08/20 14:25:10.679014 [info] prophet: ********node-0 become to leader now********
2021/08/20 14:25:10.679803 [info] prophet: load 1 containers, cost 693.8µs
2021/08/20 14:25:10.680042 [info] prophet: load 3 resources, cost 225.5µs
2021/08/20 14:25:10.680419 [info] prophet: client start init connection to leader 127.0.0.1:30000
2021/08/20 14:25:10.680428 [info] prophet: client started
2021/08/20 14:25:10.680440 [info] [raftstore]: *********Become prophet leader*********
2021/08/20 14:25:10.680442 [info] [raftstore]: begin to bootstrap the cluster
2021/08/20 14:25:10.680469 [info] [raftstore]: raftstore init with store raftAddr:"127.0.0.1:10000" clientAddr:"127.0.0.1:20000" labels:<key:"c" value:"0" > startTime:1629440710 deployPath:"not set"
2021/08/20 14:25:10.680475 [info] prophet: client start connect to leader 127.0.0.1:30000
2021/08/20 14:25:10.680419 [info] prophet: coordinator starts to collect cluster information
2021/08/20 14:25:10.680423 [info] prophet: client read loop started
2021/08/20 14:25:10.680507 [info] prophet: client read loop actived, ready to read from leader
2021/08/20 14:25:10.680483 [info] prophet: session 1[127.0.0.1:38476] connected
2021/08/20 14:25:10.680585 [info] [raftstore]: load local store 1
2021/08/20 14:25:10.680619 [info] [raftstore]: prophet started
2021/08/20 14:25:10.680680 [info] [transport]: net application started
2021/08/20 14:25:10.680694 [info] [raftstore]: start listen at 127.0.0.1:10000 for raft
2021/08/20 14:25:10.680697 [info] [raftstore]: raft shards workers started
2021/08/20 14:25:10.680731 [info] [raftstore]: shard 2 init with raft local state {HardState:{Term:6 Vote:3 Commit:9 XXX_unrecognized:[]} LastIndex:9 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:10.680754 [info] [raftstore]: shard 2 init raft apply state, state=<{AppliedIndex:9 TruncatedState:{Index:5 Term:5 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:10.680758 [info] [raftstore]: shard 2 init last term, last term=<6>
2021/08/20 14:25:10.680770 [info] [raftstore]: raft worker 0/0 start
{"level":"info","ts":"2021-08-20T14:25:10.680+0800","caller":"raft/raft.go:1550","msg":"3 switched to configuration voters=(3)"}
{"level":"info","ts":"2021-08-20T14:25:10.680+0800","caller":"raft/raft.go:711","msg":"3 became follower at term 6"}
{"level":"info","ts":"2021-08-20T14:25:10.680+0800","caller":"raft/raft.go:388","msg":"newRaft 3 [peers: [3], term: 6, commit: 9, applied: 9, lastindex: 9, lastterm: 6]"}
2021/08/20 14:25:10.680828 [info] [raftstore]: shard 2 try to campaign leader, because only self
2021/08/20 14:25:10.680735 [info] prophet: load 0 jobs
{"level":"info","ts":"2021-08-20T14:25:10.680+0800","caller":"raft/raft.go:934","msg":"3 is starting a new election at term 6"}
{"level":"info","ts":"2021-08-20T14:25:10.680+0800","caller":"raft/raft.go:724","msg":"3 became candidate at term 7"}
{"level":"info","ts":"2021-08-20T14:25:10.680+0800","caller":"raft/raft.go:835","msg":"3 received MsgVoteResp from 3 at term 7"}
{"level":"info","ts":"2021-08-20T14:25:10.680+0800","caller":"raft/raft.go:776","msg":"3 became leader at term 7"}
2021/08/20 14:25:10.680879 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 add registration job by startShards
2021/08/20 14:25:10.680916 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 add registration job by startShards, completed
2021/08/20 14:25:10.680955 [info] [raftstore]: shard 2 peer 3 raft 3 added, epoch {ConfVer:1 Version:2 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}], raft worker 0, apply worker apply-0-0
2021/08/20 14:25:10.680978 [info] [raftstore]: shard 5 init with raft local state {HardState:{Term:6 Vote:6 Commit:9 XXX_unrecognized:[]} LastIndex:9 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:10.680994 [info] [raftstore]: shard 5 init raft apply state, state=<{AppliedIndex:8 TruncatedState:{Index:5 Term:5 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:10.680998 [info] [raftstore]: shard 5 init last term, last term=<6>
{"level":"info","ts":"2021-08-20T14:25:10.681+0800","caller":"raft/raft.go:1550","msg":"6 switched to configuration voters=(6)"}
{"level":"info","ts":"2021-08-20T14:25:10.681+0800","caller":"raft/raft.go:711","msg":"6 became follower at term 6"}
{"level":"info","ts":"2021-08-20T14:25:10.681+0800","caller":"raft/raft.go:388","msg":"newRaft 6 [peers: [6], term: 6, commit: 9, applied: 8, lastindex: 9, lastterm: 6]"}
2021/08/20 14:25:10.681038 [info] [raftstore]: shard 5 try to campaign leader, because only self
{"level":"info","ts":"2021-08-20T14:25:10.681+0800","caller":"raft/raft.go:934","msg":"6 is starting a new election at term 6"}
{"level":"info","ts":"2021-08-20T14:25:10.681+0800","caller":"raft/raft.go:724","msg":"6 became candidate at term 7"}
{"level":"info","ts":"2021-08-20T14:25:10.681+0800","caller":"raft/raft.go:835","msg":"6 received MsgVoteResp from 6 at term 7"}
{"level":"info","ts":"2021-08-20T14:25:10.681+0800","caller":"raft/raft.go:776","msg":"6 became leader at term 7"}
2021/08/20 14:25:10.681123 [error] [raftstore]: >>>>>>>>>>>> shard 5 peer 6 add registration job by startShards
2021/08/20 14:25:10.681135 [error] [raftstore]: >>>>>>>>>>>> shard 5 peer 6 add registration job by startShards, completed
2021/08/20 14:25:10.681147 [info] [raftstore]: shard 5 peer 6 raft 6 added, epoch {ConfVer:1 Version:3 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:6 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}], raft worker 0, apply worker apply-0-0
2021/08/20 14:25:10.681172 [info] [raftstore]: shard 7 init with raft local state {HardState:{Term:6 Vote:8 Commit:6 XXX_unrecognized:[]} LastIndex:6 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:10.681185 [info] [raftstore]: shard 7 init raft apply state, state=<{AppliedIndex:6 TruncatedState:{Index:5 Term:5 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:10.681188 [info] [raftstore]: shard 7 init last term, last term=<6>
{"level":"info","ts":"2021-08-20T14:25:10.681+0800","caller":"raft/raft.go:1550","msg":"8 switched to configuration voters=(8)"}
{"level":"info","ts":"2021-08-20T14:25:10.681+0800","caller":"raft/raft.go:711","msg":"8 became follower at term 6"}
2021/08/20 14:25:10.681226 [info] [raftstore]: shard 2 ********become leader now********
{"level":"info","ts":"2021-08-20T14:25:10.681+0800","caller":"raft/raft.go:388","msg":"newRaft 8 [peers: [8], term: 6, commit: 6, applied: 6, lastindex: 6, lastterm: 6]"}
2021/08/20 14:25:10.681317 [info] [raftstore]: shard 7 try to campaign leader, because only self
{"level":"info","ts":"2021-08-20T14:25:10.681+0800","caller":"raft/raft.go:934","msg":"8 is starting a new election at term 6"}
{"level":"info","ts":"2021-08-20T14:25:10.681+0800","caller":"raft/raft.go:724","msg":"8 became candidate at term 7"}
{"level":"info","ts":"2021-08-20T14:25:10.681+0800","caller":"raft/raft.go:835","msg":"8 received MsgVoteResp from 8 at term 7"}
{"level":"info","ts":"2021-08-20T14:25:10.681+0800","caller":"raft/raft.go:776","msg":"8 became leader at term 7"}
2021/08/20 14:25:10.681358 [error] [raftstore]: >>>>>>>>>>>> shard 7 peer 8 add registration job by startShards
2021/08/20 14:25:10.681376 [error] [raftstore]: >>>>>>>>>>>> shard 7 peer 8 add registration job by startShards, completed
2021/08/20 14:25:10.681384 [info] [raftstore]: shard 7 peer 8 raft 8 added, epoch {ConfVer:1 Version:3 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:8 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}], raft worker 0, apply worker apply-0-0
2021/08/20 14:25:10.681395 [info] [raftstore]: starts with 3 shards, including 0 tombstones and 0 applying shards
2021/08/20 14:25:10.681396 [info] [raftstore]: cleanup possible garbage data complete
2021/08/20 14:25:10.681397 [info] [raftstore]: shards started
2021/08/20 14:25:10.681399 [info] [raftstore]: shard timer based tasks started
2021/08/20 14:25:10.681405 [info] [raftstore]: net application started
2021/08/20 14:25:10.681412 [info] [raftstore]: start listen at 127.0.0.1:20000 for client
2021/08/20 14:25:10.681418 [info] [raftstore]: router started
2021/08/20 14:25:10.681463 [info] prophet: client start init connection to leader 127.0.0.1:30000
2021/08/20 14:25:10.681512 [info] prophet: client start connect to leader 127.0.0.1:30000
2021/08/20 14:25:10.681521 [info] prophet: watcher init leader connection 127.0.0.1:30000 succeed
2021/08/20 14:25:10.681522 [info] [raftstore]: router event loop task started
2021/08/20 14:25:10.681557 [info] prophet: session 2[127.0.0.1:38478] connected
2021/08/20 14:25:10.681590 [info] prophet: new watcher 127.0.0.1:38478 added
2021/08/20 14:25:10.681649 [info] [raftstore]: start init event
2021/08/20 14:25:10.682682 [info] [raftstore]: shard 5 ********become leader now********
2021/08/20 14:25:10.683831 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 AdvanceApply with 10
2021/08/20 14:25:10.683845 [info] [raftstore]: shard 7 ********become leader now********
2021/08/20 14:25:10.691147 [error] [raftstore]: >>>>>>>>>>>> shard 5 peer 6 AdvanceApply with 9
2021/08/20 14:25:10.691161 [error] [raftstore]: >>>>>>>>>>>> shard 5 peer 6 AdvanceApply with 10
2021/08/20 14:25:10.691165 [error] [raftstore]: >>>>>>>>>>>> shard 7 peer 8 AdvanceApply with 7
2021/08/20 14:25:10.791553 [info] [raftstore]: shard 5 start split check job from [122 0 0 0 0 0 0 0 0 107 101 121 50] to [122 0 0 0 0 0 0 0 0 107 101 121 51]
2021/08/20 14:25:10.791665 [error] [raftstore]: shard 5 heartbeat to prophet failed with client is closed
2021/08/20 14:25:10.791684 [error] prophet: client read resp from leader 127.0.0.1:30000 failed with read tcp 127.0.0.1:38476->127.0.0.1:30000: use of closed network connection
2021/08/20 14:25:10.791688 [error] prophet: client read loop stopped
2021/08/20 14:25:10.791686 [error] [raftstore]: shard 7 heartbeat to prophet failed with client is closed
2021/08/20 14:25:10.791708 [error] prophet: flush messages failed with illegal state, closed this session
2021/08/20 14:25:10.791729 [error] prophet: session 1[] read failed with illegal state
2021/08/20 14:25:10.791723 [error] prophet: session 2[127.0.0.1:38478] read failed with read tcp 127.0.0.1:30000->127.0.0.1:38478: use of closed network connection
2021/08/20 14:25:10.791722 [error] prophet: watcher read events failed with EOF
2021/08/20 14:25:10.791735 [error] prophet: reset watcher conn failed with <nil>, retry later
2021/08/20 14:25:10.792156 [info] prophet: [prophet-leader/node-0]/lease: closed
2021/08/20 14:25:10.792213 [info] prophet: [prophet-leader/node-0]/keepalive: exit with context done
2021-08-20 14:25:10.792232 I | etcdserver: skipped leadership transfer for single voting member cluster
2021/08/20 14:25:10.792246 [info] prophet: [prophet-leader/node-0]/lease: exit with context done
2021/08/20 14:25:10.792248 [error] prophet: [prophet-leader/node-0]/lease: keep lease exit
2021/08/20 14:25:10.792252 [info] prophet: coordinator stops running
2021/08/20 14:25:10.792256 [info] prophet: coordinator is stopping
2021/08/20 14:25:10.792258 [info] prophet: coordinator has been stopped
2021/08/20 14:25:10.801527 [info] [raftstore]: shard 2 handle serve raft stopped
2021/08/20 14:25:10.801540 [info] [raftstore]: shard 5 handle serve raft stopped
2021/08/20 14:25:10.801542 [info] [raftstore]: shard 7 handle serve raft stopped
2021/08/20 14:25:10.801561 [info] [raftstore]: raft worker worker 0/0 exit
2021/08/20 14:25:10.801567 [info] [raftstore]: timer based tasks stopped
2021/08/20 14:25:10.801571 [info] [raftstore]: router event loop task stopped
2021/08/20 14:25:10.801603 [info] [transport]: send snapshot worker stopped
2021/08/20 14:25:10.801608 [info] [transport]: transfer stopped
2021/08/20 14:25:10.801617 [info] [transport]: send snapshot worker stopped
2021/08/20 14:25:10.801624 [info] [transport]: send snapshot worker stopped
2021/08/20 14:25:10.801626 [info] [transport]: send snapshot worker stopped
2021/08/20 14:25:10.801628 [info] [transport]: send raft worker stopped
2021/08/20 14:25:10.801624 [info] [transport]: send snapshot worker stopped
2021/08/20 14:25:10.801634 [info] [transport]: send snapshot worker stopped
2021/08/20 14:25:10.801634 [info] [transport]: send snapshot worker stopped
2021/08/20 14:25:10.801640 [info] [transport]: send snapshot worker stopped
--- PASS: TestIssue97 (6.33s)
=== RUN TestSyncData
2021/08/20 14:25:10.804199 [info] prophet: prophet logger set
2021/08/20 14:25:10.804661 [info] [raftstore]: start scan gc snap files
2021/08/20 14:25:10.804891 [info] [raftstore]: start scan gc snap files
2021/08/20 14:25:10.805143 [info] [raftstore]: begin start raftstore
2021/08/20 14:25:10.805154 [info] [raftstore]: begin to start prophet
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021/08/20 14:25:10.805190 [info] [raftstore]: start scan gc snap files
2021-08-20 14:25:10.805203 I | embed: pprof is enabled under /debug/pprof
2021-08-20 14:25:10.805235 I | embed: name = node-0
2021-08-20 14:25:10.805247 I | embed: data dir = /tmp/cube/node-0/prophet
2021-08-20 14:25:10.805249 I | embed: member dir = /tmp/cube/node-0/prophet/member
2021-08-20 14:25:10.805250 I | embed: heartbeat = 100ms
2021-08-20 14:25:10.805252 I | embed: election = 3000ms
2021-08-20 14:25:10.805253 I | embed: snapshot count = 100000
2021-08-20 14:25:10.805256 I | embed: advertise client URLs = http://127.0.0.1:40000
2021-08-20 14:25:10.809893 I | etcdserver: starting member 93ccda9956d988f4 in cluster 6a4ade671189dc2f
{"level":"info","ts":"2021-08-20T14:25:10.809+0800","caller":"raft/raft.go:1550","msg":"93ccda9956d988f4 switched to configuration voters=()"}
{"level":"info","ts":"2021-08-20T14:25:10.809+0800","caller":"raft/raft.go:711","msg":"93ccda9956d988f4 became follower at term 0"}
{"level":"info","ts":"2021-08-20T14:25:10.809+0800","caller":"raft/raft.go:388","msg":"newRaft 93ccda9956d988f4 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"}
{"level":"info","ts":"2021-08-20T14:25:10.809+0800","caller":"raft/raft.go:711","msg":"93ccda9956d988f4 became follower at term 1"}
{"level":"info","ts":"2021-08-20T14:25:10.809+0800","caller":"raft/raft.go:1550","msg":"93ccda9956d988f4 switched to configuration voters=(10650127570946328820)"}
2021-08-20 14:25:10.811649 W | auth: simple token is not cryptographically signed
2021-08-20 14:25:10.814529 I | etcdserver: starting server... [version: 3.4.15, cluster version: to_be_decided]
2021-08-20 14:25:10.814656 I | etcdserver: 93ccda9956d988f4 as single-node; fast-forwarding 29 ticks (election ticks 30)
2021-08-20 14:25:10.814758 I | embed: listening for peers on 127.0.0.1:50000
{"level":"info","ts":"2021-08-20T14:25:10.814+0800","caller":"raft/raft.go:1550","msg":"93ccda9956d988f4 switched to configuration voters=(10650127570946328820)"}
2021-08-20 14:25:10.814876 I | etcdserver/membership: added member 93ccda9956d988f4 [http://127.0.0.1:50000] to cluster 6a4ade671189dc2f
2021/08/20 14:25:10.873551 [info] [raftstore]: shard 2 raft tick stopped
2021/08/20 14:25:10.873580 [info] [raftstore]: shard 5 raft tick stopped
2021/08/20 14:25:10.873582 [info] [raftstore]: shard 7 raft tick stopped
2021/08/20 14:25:10.892880 [info] prophet: [prophet-leader/node-0]/loop: exit
{"level":"info","ts":"2021-08-20T14:25:11.310+0800","caller":"raft/raft.go:934","msg":"93ccda9956d988f4 is starting a new election at term 1"}
{"level":"info","ts":"2021-08-20T14:25:11.310+0800","caller":"raft/raft.go:724","msg":"93ccda9956d988f4 became candidate at term 2"}
{"level":"info","ts":"2021-08-20T14:25:11.311+0800","caller":"raft/raft.go:835","msg":"93ccda9956d988f4 received MsgVoteResp from 93ccda9956d988f4 at term 2"}
{"level":"info","ts":"2021-08-20T14:25:11.311+0800","caller":"raft/raft.go:776","msg":"93ccda9956d988f4 became leader at term 2"}
{"level":"info","ts":"2021-08-20T14:25:11.311+0800","caller":"raft/node.go:325","msg":"raft.node: 93ccda9956d988f4 elected leader 93ccda9956d988f4 at term 2"}
2021-08-20 14:25:11.311418 I | etcdserver: published {Name:node-0 ClientURLs:[http://127.0.0.1:40000]} to cluster 6a4ade671189dc2f
2021/08/20 14:25:11.311437 [info] prophet: create etcd v3 client with endpoints [http://127.0.0.1:40000]
2021-08-20 14:25:11.311437 I | embed: ready to serve client requests
2021-08-20 14:25:11.311537 I | etcdserver: setting up the initial cluster version to 3.4
2021-08-20 14:25:11.311686 N | embed: serving insecure client requests on 127.0.0.1:40000, this is strongly discouraged!
2021-08-20 14:25:11.311775 N | etcdserver/membership: set the initial cluster version to 3.4
2021/08/20 14:25:11.313627 [info] prophet: net application started
2021/08/20 14:25:11.313641 [info] prophet: start system time monitor
2021/08/20 14:25:11.314085 [info] prophet: [prophet-leader/node-0]/loop: current leader is []
2021/08/20 14:25:11.314291 [info] prophet: [prophet-leader/node-0]/campaign: load expect leader: []
2021/08/20 14:25:11.314301 [info] prophet: [prophet-leader/node-0]/campaign: start
2021/08/20 14:25:11.314526 [info] prophet: [prophet-leader/node-0]/campaign: grant lease ok
2021/08/20 14:25:11.315044 [info] prophet: [prophet-leader/node-0]/campaign: completed, start keepalive
2021/08/20 14:25:11.315061 [info] prophet: ********node-0 become to leader now********
2021/08/20 14:25:11.315447 [info] prophet: load 0 containers, cost 337.4µs
2021/08/20 14:25:11.315630 [info] prophet: load 0 resources, cost 173.7µs
2021/08/20 14:25:11.316482 [info] prophet: coordinator starts to collect cluster information
2021/08/20 14:25:11.316493 [info] prophet: client read loop started
2021/08/20 14:25:11.316499 [info] prophet: coordinator has finished cluster information preparation
2021/08/20 14:25:11.316500 [info] prophet: coordinator starts to run schedulers
2021/08/20 14:25:11.316484 [info] prophet: client start init connection to leader 127.0.0.1:30000
2021/08/20 14:25:11.316485 [info] prophet: client started
2021/08/20 14:25:11.316568 [info] [raftstore]: *********Become prophet leader*********
2021/08/20 14:25:11.316570 [info] [raftstore]: begin to bootstrap the cluster
2021/08/20 14:25:11.316585 [info] [raftstore]: raftstore init with store raftAddr:"127.0.0.1:10000" clientAddr:"127.0.0.1:20000" labels:<key:"c" value:"0" > startTime:1629440711 deployPath:"not set"
2021/08/20 14:25:11.316595 [info] [raftstore]: begin to create local store metadata
2021/08/20 14:25:11.316602 [info] prophet: client start connect to leader 127.0.0.1:30000
2021/08/20 14:25:11.316609 [info] prophet: client read loop actived, ready to read from leader
2021/08/20 14:25:11.316611 [info] prophet: session 1[127.0.0.1:38484] connected
2021/08/20 14:25:11.316815 [info] prophet: load 0 jobs
2021/08/20 14:25:11.317114 [info] prophet: coordinator begins to actively drive push operator
2021/08/20 14:25:11.317144 [info] prophet: coordinator starts patrol resources
2021/08/20 14:25:11.317619 [info] [raftstore]: create local store with id 1
2021/08/20 14:25:11.318038 [info] [raftstore]: the cluster already bootstrap: false
2021/08/20 14:25:11.318047 [info] [raftstore]: begin to bootstrap the cluster with init shards
2021/08/20 14:25:11.318195 [info] [raftstore]: create init shard id:2 epoch:<confVer:1 version:1 > peers:<id:3 containerID:1 >
2021/08/20 14:25:11.319401 [info] prophet: container 1 limit added
2021/08/20 14:25:11.319475 [info] [raftstore]: prophet started
2021/08/20 14:25:11.319504 [info] [transport]: net application started
2021/08/20 14:25:11.319514 [info] [raftstore]: start listen at 127.0.0.1:10000 for raft
2021/08/20 14:25:11.319517 [info] [raftstore]: raft shards workers started
2021/08/20 14:25:11.319529 [info] prophet: container 1 heartbeat stream binded
2021/08/20 14:25:11.319534 [info] [raftstore]: shard 2 init with raft local state {HardState:{Term:5 Vote:0 Commit:5 XXX_unrecognized:[]} LastIndex:5 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:11.319541 [info] [raftstore]: shard 2 init raft apply state, state=<{AppliedIndex:5 TruncatedState:{Index:5 Term:5 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:11.319543 [info] [raftstore]: shard 2 init last term, last term=<5>
2021/08/20 14:25:11.319549 [info] [raftstore]: shard 2 change init applied index from 5 to 5
{"level":"info","ts":"2021-08-20T14:25:11.319+0800","caller":"raft/raft.go:1550","msg":"3 switched to configuration voters=(3)"}
2021/08/20 14:25:11.319591 [info] [raftstore]: raft worker 0/0 start
{"level":"info","ts":"2021-08-20T14:25:11.319+0800","caller":"raft/raft.go:711","msg":"3 became follower at term 5"}
{"level":"info","ts":"2021-08-20T14:25:11.319+0800","caller":"raft/raft.go:388","msg":"newRaft 3 [peers: [3], term: 5, commit: 5, applied: 5, lastindex: 5, lastterm: 5]"}
2021/08/20 14:25:11.319605 [info] [raftstore]: shard 2 try to campaign leader, because only self
{"level":"info","ts":"2021-08-20T14:25:11.319+0800","caller":"raft/raft.go:934","msg":"3 is starting a new election at term 5"}
{"level":"info","ts":"2021-08-20T14:25:11.319+0800","caller":"raft/raft.go:724","msg":"3 became candidate at term 6"}
{"level":"info","ts":"2021-08-20T14:25:11.319+0800","caller":"raft/raft.go:835","msg":"3 received MsgVoteResp from 3 at term 6"}
{"level":"info","ts":"2021-08-20T14:25:11.319+0800","caller":"raft/raft.go:776","msg":"3 became leader at term 6"}
2021/08/20 14:25:11.319637 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 add registration job by startShards
2021/08/20 14:25:11.319656 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 add registration job by startShards, completed
2021/08/20 14:25:11.319685 [info] [raftstore]: shard 2 peer 3 raft 3 added, epoch {ConfVer:1 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}], raft worker 0, apply worker apply-0-0
2021/08/20 14:25:11.319700 [info] [raftstore]: starts with 1 shards, including 0 tombstones and 0 applying shards
2021/08/20 14:25:11.319702 [info] [raftstore]: cleanup possible garbage data complete
2021/08/20 14:25:11.319703 [info] [raftstore]: shards started
2021/08/20 14:25:11.319706 [info] [raftstore]: shard timer based tasks started
2021/08/20 14:25:11.319719 [info] [raftstore]: net application started
2021/08/20 14:25:11.319730 [info] [raftstore]: start listen at 127.0.0.1:20000 for client
2021/08/20 14:25:11.319757 [info] [raftstore]: router started
2021/08/20 14:25:11.319785 [info] prophet: client start init connection to leader 127.0.0.1:30000
2021/08/20 14:25:11.319790 [info] [raftstore]: router event loop task started
2021/08/20 14:25:11.319865 [info] prophet: client start connect to leader 127.0.0.1:30000
2021/08/20 14:25:11.319873 [info] prophet: watcher init leader connection 127.0.0.1:30000 succeed
2021/08/20 14:25:11.319884 [info] prophet: session 2[127.0.0.1:38486] connected
2021/08/20 14:25:11.319913 [info] prophet: new watcher 127.0.0.1:38486 added
2021/08/20 14:25:11.319975 [info] [raftstore]: start init event
2021/08/20 14:25:11.330044 [info] [raftstore]: shard 2 ********become leader now********
2021/08/20 14:25:11.330092 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 AdvanceApply with 6
2021/08/20 14:25:11.420946 [info] [raftstore]: begin start raftstore
2021/08/20 14:25:11.420958 [info] [raftstore]: begin to start prophet
{"level":"info","ts":"2021-08-20T14:25:11.421+0800","caller":"raft/raft.go:1550","msg":"93ccda9956d988f4 switched to configuration voters=(10650127570946328820 12791365408210890720)"}
2021-08-20 14:25:11.421840 I | etcdserver/membership: added member b1840f1134a12fe0 [http://127.0.0.1:50001] to cluster 6a4ade671189dc2f
2021-08-20 14:25:11.421861 I | rafthttp: starting peer b1840f1134a12fe0...
2021-08-20 14:25:11.421881 I | rafthttp: started HTTP pipelining with peer b1840f1134a12fe0
2021-08-20 14:25:11.422280 I | rafthttp: started streaming with peer b1840f1134a12fe0 (writer)
2021-08-20 14:25:11.422902 I | rafthttp: started streaming with peer b1840f1134a12fe0 (writer)
2021-08-20 14:25:11.423262 I | rafthttp: started peer b1840f1134a12fe0
2021-08-20 14:25:11.423276 I | rafthttp: started streaming with peer b1840f1134a12fe0 (stream MsgApp v2 reader)
2021-08-20 14:25:11.423280 I | rafthttp: added peer b1840f1134a12fe0
2021-08-20 14:25:11.423305 I | rafthttp: started streaming with peer b1840f1134a12fe0 (stream Message reader)
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-08-20 14:25:11.423626 I | embed: pprof is enabled under /debug/pprof
2021-08-20 14:25:11.423656 I | embed: name = node-1
2021-08-20 14:25:11.423680 I | embed: data dir = /tmp/cube/node-1/prophet
2021-08-20 14:25:11.423683 I | embed: member dir = /tmp/cube/node-1/prophet/member
2021-08-20 14:25:11.423684 I | embed: heartbeat = 100ms
2021-08-20 14:25:11.423686 I | embed: election = 3000ms
2021-08-20 14:25:11.423687 I | embed: snapshot count = 100000
2021-08-20 14:25:11.423689 I | embed: advertise client URLs = http://127.0.0.1:40001
2021-08-20 14:25:11.427864 I | etcdserver: starting member b1840f1134a12fe0 in cluster 6a4ade671189dc2f
{"level":"info","ts":"2021-08-20T14:25:11.427+0800","caller":"raft/raft.go:1550","msg":"b1840f1134a12fe0 switched to configuration voters=()"}
{"level":"info","ts":"2021-08-20T14:25:11.427+0800","caller":"raft/raft.go:711","msg":"b1840f1134a12fe0 became follower at term 0"}
{"level":"info","ts":"2021-08-20T14:25:11.427+0800","caller":"raft/raft.go:388","msg":"newRaft b1840f1134a12fe0 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"}
2021-08-20 14:25:11.429589 W | auth: simple token is not cryptographically signed
2021-08-20 14:25:11.432690 I | rafthttp: started HTTP pipelining with peer 93ccda9956d988f4
2021-08-20 14:25:11.432704 I | rafthttp: starting peer 93ccda9956d988f4...
2021-08-20 14:25:11.432710 I | rafthttp: started HTTP pipelining with peer 93ccda9956d988f4
2021-08-20 14:25:11.432854 I | rafthttp: started streaming with peer 93ccda9956d988f4 (writer)
2021-08-20 14:25:11.432933 I | rafthttp: started streaming with peer 93ccda9956d988f4 (writer)
2021-08-20 14:25:11.432951 I | rafthttp: started peer 93ccda9956d988f4
2021-08-20 14:25:11.432957 I | rafthttp: added peer 93ccda9956d988f4
2021-08-20 14:25:11.432960 I | etcdserver: starting server... [version: 3.4.15, cluster version: to_be_decided]
2021-08-20 14:25:11.433043 I | rafthttp: started streaming with peer 93ccda9956d988f4 (stream Message reader)
2021-08-20 14:25:11.433085 I | rafthttp: started streaming with peer 93ccda9956d988f4 (stream MsgApp v2 reader)
2021-08-20 14:25:11.433219 I | embed: listening for peers on 127.0.0.1:50001
2021-08-20 14:25:11.433320 I | rafthttp: peer 93ccda9956d988f4 became active
2021-08-20 14:25:11.433342 I | rafthttp: peer b1840f1134a12fe0 became active
2021-08-20 14:25:11.433353 I | rafthttp: established a TCP streaming connection with peer 93ccda9956d988f4 (stream MsgApp v2 reader)
2021-08-20 14:25:11.433398 I | rafthttp: established a TCP streaming connection with peer b1840f1134a12fe0 (stream Message writer)
2021-08-20 14:25:11.433415 I | rafthttp: established a TCP streaming connection with peer b1840f1134a12fe0 (stream MsgApp v2 writer)
2021-08-20 14:25:11.433424 I | rafthttp: established a TCP streaming connection with peer 93ccda9956d988f4 (stream Message reader)
{"level":"info","ts":"2021-08-20T14:25:11.510+0800","caller":"raft/raft.go:870","msg":"b1840f1134a12fe0 [term: 0] received a MsgHeartbeat message with higher term from 93ccda9956d988f4 [term: 2]"}
{"level":"info","ts":"2021-08-20T14:25:11.510+0800","caller":"raft/raft.go:711","msg":"b1840f1134a12fe0 became follower at term 2"}
{"level":"info","ts":"2021-08-20T14:25:11.510+0800","caller":"raft/node.go:325","msg":"raft.node: b1840f1134a12fe0 elected leader 93ccda9956d988f4 at term 2"}
{"level":"info","ts":"2021-08-20T14:25:11.512+0800","caller":"raft/raft.go:1550","msg":"b1840f1134a12fe0 switched to configuration voters=(10650127570946328820)"}
2021-08-20 14:25:11.512919 I | etcdserver/membership: added member 93ccda9956d988f4 [http://127.0.0.1:50000] to cluster 6a4ade671189dc2f
2021-08-20 14:25:11.512986 N | etcdserver/membership: set the initial cluster version to 3.4
{"level":"info","ts":"2021-08-20T14:25:11.514+0800","caller":"raft/raft.go:1550","msg":"b1840f1134a12fe0 switched to configuration voters=(10650127570946328820 12791365408210890720)"}
2021-08-20 14:25:11.514835 I | etcdserver/membership: added member b1840f1134a12fe0 [http://127.0.0.1:50001] to cluster 6a4ade671189dc2f
2021-08-20 14:25:11.515167 I | etcdserver: published {Name:node-1 ClientURLs:[http://127.0.0.1:40001]} to cluster 6a4ade671189dc2f
2021-08-20 14:25:11.515186 I | embed: ready to serve client requests
2021/08/20 14:25:11.515195 [info] prophet: create etcd v3 client with endpoints [http://127.0.0.1:40001]
2021-08-20 14:25:11.515483 N | embed: serving insecure client requests on 127.0.0.1:40001, this is strongly discouraged!
2021/08/20 14:25:11.516662 [info] prophet: net application started
2021/08/20 14:25:11.516702 [info] prophet: start system time monitor
2021/08/20 14:25:11.517250 [info] prophet: [prophet-leader/node-1]/loop: current leader is [ôæ¶Ó¶ænode-0127.0.0.1:30000]
2021/08/20 14:25:11.517262 [info] prophet: [prophet-leader/node-1]/loop: start watch leader ôæ¶Ó¶ænode-0127.0.0.1:30000
2021/08/20 14:25:11.517266 [info] prophet: ********node-1 become to follower now********
2021/08/20 14:25:11.517295 [info] prophet: client start init connection to leader 127.0.0.1:30000
2021/08/20 14:25:11.517311 [info] prophet: client read loop started
2021/08/20 14:25:11.517317 [info] prophet: client started
2021/08/20 14:25:11.517320 [info] [raftstore]: *********Become prophet follower*********
2021/08/20 14:25:11.517322 [info] [raftstore]: begin to bootstrap the cluster
2021/08/20 14:25:11.517341 [info] [raftstore]: raftstore init with store raftAddr:"127.0.0.1:10001" clientAddr:"127.0.0.1:20001" labels:<key:"c" value:"1" > startTime:1629440711 deployPath:"not set"
2021/08/20 14:25:11.517355 [info] [raftstore]: begin to create local store metadata
2021/08/20 14:25:11.517370 [info] prophet: client start connect to leader 127.0.0.1:30000
2021/08/20 14:25:11.517379 [info] prophet: session 3[127.0.0.1:38516] connected
2021/08/20 14:25:11.517385 [info] prophet: client read loop actived, ready to read from leader
2021/08/20 14:25:11.517522 [info] [raftstore]: create local store with id 5
2021/08/20 14:25:11.518258 [info] [raftstore]: the cluster already bootstrap: true
2021/08/20 14:25:11.521613 [info] prophet: container 5 limit added
2021/08/20 14:25:11.521692 [info] [raftstore]: prophet started
2021/08/20 14:25:11.521726 [info] prophet: container 5 heartbeat stream binded
2021/08/20 14:25:11.521749 [info] [transport]: net application started
2021/08/20 14:25:11.521761 [info] [raftstore]: start listen at 127.0.0.1:10001 for raft
2021/08/20 14:25:11.521766 [info] [raftstore]: raft shards workers started
2021/08/20 14:25:11.521771 [info] [raftstore]: starts with 0 shards, including 0 tombstones and 0 applying shards
2021/08/20 14:25:11.521772 [info] [raftstore]: cleanup possible garbage data complete
2021/08/20 14:25:11.521773 [info] [raftstore]: shards started
2021/08/20 14:25:11.521775 [info] [raftstore]: shard timer based tasks started
2021/08/20 14:25:11.521781 [info] [raftstore]: net application started
2021/08/20 14:25:11.521782 [info] [raftstore]: start listen at 127.0.0.1:20001 for client
2021/08/20 14:25:11.521790 [info] [raftstore]: router started
2021/08/20 14:25:11.521807 [info] [raftstore]: raft worker 0/0 start
2021/08/20 14:25:11.521820 [info] prophet: client start init connection to leader 127.0.0.1:30000
2021/08/20 14:25:11.521829 [info] [raftstore]: router event loop task started
2021/08/20 14:25:11.521923 [info] prophet: session 4[127.0.0.1:38518] connected
2021/08/20 14:25:11.521923 [info] prophet: client start connect to leader 127.0.0.1:30000
2021/08/20 14:25:11.521944 [info] prophet: watcher init leader connection 127.0.0.1:30000 succeed
2021/08/20 14:25:11.521987 [info] prophet: new watcher 127.0.0.1:38518 added
2021/08/20 14:25:11.522097 [info] [raftstore]: start init event
2021-08-20 14:25:11.523928 I | rafthttp: established a TCP streaming connection with peer 93ccda9956d988f4 (stream Message writer)
2021-08-20 14:25:11.523973 I | rafthttp: established a TCP streaming connection with peer b1840f1134a12fe0 (stream Message reader)
2021-08-20 14:25:11.523987 I | rafthttp: established a TCP streaming connection with peer b1840f1134a12fe0 (stream MsgApp v2 reader)
2021-08-20 14:25:11.524164 I | rafthttp: established a TCP streaming connection with peer 93ccda9956d988f4 (stream MsgApp v2 writer)
2021/08/20 14:25:11.531834 [info] prophet: [prophet-leader/node-1]/watch: leader updated to ôæ¶Ó¶ænode-0127.0.0.1:30000
2021/08/20 14:25:11.618122 [info] prophet: resource 2 add operator add-rule-peer {add peer: container [5]} (kind:resource,replica, resource:2(1,1), createAt:2021-08-20 14:25:11.6180939 +0800 CST m=+7.151620501, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, steps:[add learner peer 6 on container 5, promote learner peer 6 on container 5 to voter]),
2021/08/20 14:25:11.618147 [info] prophet: resource 2 send schedule command, step add learner peer 6 on container 5, source create
2021/08/20 14:25:11.618244 [info] [raftstore]: shard-2 AddLearnerNode peer {ID:6 ContainerID:5 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:11.620391 [info] [raftstore]: shard-2 propose conf change peer [{ChangeType:AddLearnerNode Peer:{ID:6 ContainerID:5 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:11.620496 [info] prophet: resource 2 send schedule command, step add learner peer 6 on container 5, source heartbeat
2021/08/20 14:25:11.620512 [info] [raftstore]: shard 2 do apply change peer changeType:AddLearnerNode peer:<id:6 containerID:5 role:Learner > at epoch {ConfVer:1 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:11.620521 [info] [raftstore]: shard-2 add learner peer {ID:6 ContainerID:5 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} successfully, peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:11.620568 [info] [raftstore]: shard-2 AddLearnerNode peer {ID:6 ContainerID:5 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:11.630139 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 AdvanceApply with 7
{"level":"info","ts":"2021-08-20T14:25:11.630+0800","caller":"raft/raft.go:1550","msg":"3 switched to configuration voters=(3) learners=(6)"}
2021/08/20 14:25:11.630209 [info] [raftstore]: shard 2 notify pd with {Transition:ConfChangeTransitionAuto Changes:[{Type:ConfChangeAddLearnerNode NodeID:6 XXX_unrecognized:[]}] Context:[10 32 10 16 33 141 251 230 30 102 76 222 183 130 63 53 180 11 83 239 16 2 26 4 8 3 16 1 42 4 8 1 16 1 26 14 8 1 18 10 8 3 18 6 8 6 16 5 24 1] XXX_unrecognized:[]} with changes [{ChangeType:AddLearnerNode Peer:{ID:6 ContainerID:5 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] at epoch {ConfVer:2 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:11.630230 [info] [raftstore]: shard 2 applied changes [{ChangeType:AddLearnerNode Peer:{ID:6 ContainerID:5 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] at epoch {ConfVer:2 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, new peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:11.630255 [info] [raftstore]: shard-2 propose conf change peer [{ChangeType:AddLearnerNode Peer:{ID:6 ContainerID:5 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:11.630380 [info] [raftstore]: shard 2 do apply change peer changeType:AddLearnerNode peer:<id:6 containerID:5 role:Learner > at epoch {ConfVer:2 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:11.630406 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 AdvanceApply with 8
2021/08/20 14:25:11.630438 [info] prophet: resource 2 ConfVer changed from 1 to 2
2021/08/20 14:25:11.630739 [info] [transport]: connected to store 5
2021/08/20 14:25:11.630762 [info] [transport]: session 1[127.0.0.1:58404] connected
2021/08/20 14:25:11.630801 [info] [raftstore]: shard 2 target peer doesn't exist, peer=<{ID:6 ContainerID:5 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}> message=<MsgApp>
2021/08/20 14:25:11.630809 [info] [raftstore]: shard 2 replicate peer {ID:6 ContainerID:5 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:11.630825 [info] [raftstore]: shard 2 init with raft local state {HardState:{Term:0 Vote:0 Commit:0 XXX_unrecognized:[]} LastIndex:0 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:11.630849 [info] [raftstore]: shard 2 init raft apply state, state=<{AppliedIndex:0 TruncatedState:{Index:0 Term:0 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:11.630850 [info] [raftstore]: shard 2 init last term, last term=<0>
2021/08/20 14:25:11.630854 [info] [raftstore]: shard 2 change init applied index from 0 to 0
{"level":"info","ts":"2021-08-20T14:25:11.630+0800","caller":"raft/raft.go:1550","msg":"6 switched to configuration voters=()"}
{"level":"info","ts":"2021-08-20T14:25:11.630+0800","caller":"raft/raft.go:711","msg":"6 became follower at term 0"}
{"level":"info","ts":"2021-08-20T14:25:11.630+0800","caller":"raft/raft.go:388","msg":"newRaft 6 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"}
2021/08/20 14:25:11.630929 [info] [raftstore]: shard 2 peer 6 raft 6 added, epoch {ConfVer:2 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:6 ContainerID:5 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}], raft worker 0, apply worker apply-0-0
{"level":"info","ts":"2021-08-20T14:25:11.630+0800","caller":"raft/raft.go:870","msg":"6 [term: 0] received a MsgHeartbeat message with higher term from 3 [term: 6]"}
{"level":"info","ts":"2021-08-20T14:25:11.630+0800","caller":"raft/raft.go:711","msg":"6 became follower at term 6"}
2021/08/20 14:25:11.631000 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:11.631059 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 skip with no delegates, 0 commits, result <nil>
2021/08/20 14:25:11.631677 [info] prophet: resource 2 send schedule command, step promote learner peer 6 on container 5 to voter, source heartbeat
2021/08/20 14:25:11.631756 [info] [raftstore]: shard-2 AddNode peer {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:11.632082 [info] [transport]: connected to store 1
2021/08/20 14:25:11.632101 [info] [transport]: session 1[127.0.0.1:36090] connected
2021/08/20 14:25:11.632140 [info] [raftstore]: shard-2 propose conf change peer [{ChangeType:AddNode Peer:{ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:11.632193 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:11.632197 [info] [raftstore]: shard 2 do apply change peer peer:<id:6 containerID:5 > at epoch {ConfVer:2 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:11.632245 [info] [raftstore]: shard-2 add peer {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} successfully, peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:11.632249 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 skip with no delegates, 0 commits, result <nil>
{"level":"error","ts":"2021-08-20T14:25:11.632+0800","caller":"raft/log.go:354","msg":"3 >>>>>>>>>>>>>>>> [1:10:6)","stacktrace":"go.etcd.io/etcd/pkg/logutil.(*zapRaftLogger).Errorf\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/pkg/logutil/zap_raft.go:69\ngo.etcd.io/etcd/raft.(*raftLog).mustCheckOutOfBounds\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:354\ngo.etcd.io/etcd/raft.(*raftLog).slice\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:308\ngo.etcd.io/etcd/raft.(*raftLog).entries\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:258\ngo.etcd.io/etcd/raft.(*raft).maybeSendAppend\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:459\ngo.etcd.io/etcd/raft.(*raft).sendAppend\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:442\ngo.etcd.io/etcd/raft.stepLeader\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:1150\ngo.etcd.io/etcd/raft.(*raft).Step\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:995\ngo.etcd.io/etcd/raft.(*RawNode).Step\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/rawnode.go:114\ngithub.com/matrixorigin/matrixcube/raftstore.(*peerReplica).handleStep\n\t/root/projects/matrixcube/raftstore/peer_event_loop.go:257\ngithub.com/matrixorigin/matrixcube/raftstore.(*peerReplica).handleEvent\n\t/root/projects/matrixcube/raftstore/peer_event_loop.go:183\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask.func1.1\n\t/root/projects/matrixcube/raftstore/store.go:420\nsync.(*Map).Range\n\t/opt/lib/go/src/sync/map.go:345\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask.func1\n\t/root/projects/matrixcube/raftstore/store.go:418\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask\n\t/root/projects/matrixcube/raftstore/store.go:443\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).startRaftWorkers.func1\n\t/root/projects/matrixcube/raftstore/store.go:406\ngithub.com/fagongzi/util/task.(*Runner).RunCancelableTask.func1\n\t/root/projects/matrixcube/vendor/github.com/fagongzi/util/task/task.go:350"}
2021/08/20 14:25:11.632370 [info] [raftstore]: shard 2 start snapshot, epoch=<{ConfVer:2 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:11.632472 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 AdvanceApply with 9
{"level":"info","ts":"2021-08-20T14:25:11.632+0800","caller":"raft/raft.go:1550","msg":"3 switched to configuration voters=(3 6)"}
{"level":"error","ts":"2021-08-20T14:25:11.632+0800","caller":"raft/log.go:354","msg":"3 >>>>>>>>>>>>>>>> [1:10:6)","stacktrace":"go.etcd.io/etcd/pkg/logutil.(*zapRaftLogger).Errorf\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/pkg/logutil/zap_raft.go:69\ngo.etcd.io/etcd/raft.(*raftLog).mustCheckOutOfBounds\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:354\ngo.etcd.io/etcd/raft.(*raftLog).slice\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:308\ngo.etcd.io/etcd/raft.(*raftLog).entries\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:258\ngo.etcd.io/etcd/raft.(*raft).maybeSendAppend\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:459\ngo.etcd.io/etcd/raft.(*raft).switchToConfig.func1\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:1586\ngo.etcd.io/etcd/raft/tracker.(*ProgressTracker).Visit\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/tracker/tracker.go:209\ngo.etcd.io/etcd/raft.(*raft).switchToConfig\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:1585\ngo.etcd.io/etcd/raft.(*raft).applyConfChange\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:1537\ngo.etcd.io/etcd/raft.(*RawNode).ApplyConfChange\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/rawnode.go:103\ngithub.com/matrixorigin/matrixcube/raftstore.(*peerReplica).doApplyConfChange\n\t/root/projects/matrixcube/raftstore/peer_event_post_apply.go:106\ngithub.com/matrixorigin/matrixcube/raftstore.(*peerReplica).doPostApplyResult\n\t/root/projects/matrixcube/raftstore/peer_event_post_apply.go:92\ngithub.com/matrixorigin/matrixcube/raftstore.(*peerReplica).doPollApply\n\t/root/projects/matrixcube/raftstore/peer_event_post_apply.go:51\ngithub.com/matrixorigin/matrixcube/raftstore.(*peerReplica).handleApplyResult\n\t/root/projects/matrixcube/raftstore/peer_event_post_apply.go:39\ngithub.com/matrixorigin/matrixcube/raftstore.(*peerReplica).handleEvent\n\t/root/projects/matrixcube/raftstore/peer_event_loop.go:186\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask.func1.1\n\t/root/projects/matrixcube/raftstore/store.go:420\nsync.(*Map).Range\n\t/opt/lib/go/src/sync/map.go:345\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask.func1\n\t/root/projects/matrixcube/raftstore/store.go:418\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask\n\t/root/projects/matrixcube/raftstore/store.go:443\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).startRaftWorkers.func1\n\t/root/projects/matrixcube/raftstore/store.go:406\ngithub.com/fagongzi/util/task.(*Runner).RunCancelableTask.func1\n\t/root/projects/matrixcube/vendor/github.com/fagongzi/util/task/task.go:350"}
2021/08/20 14:25:11.632563 [info] [raftstore]: shard 2 notify pd with {Transition:ConfChangeTransitionAuto Changes:[{Type:ConfChangeAddNode NodeID:6 XXX_unrecognized:[]}] Context:[10 32 10 16 129 206 171 239 156 52 70 206 134 50 248 138 14 79 77 202 16 2 26 4 8 3 16 1 42 4 8 2 16 1 26 10 8 1 18 6 18 4 8 6 16 5] XXX_unrecognized:[]} with changes [{ChangeType:AddNode Peer:{ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] at epoch {ConfVer:3 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:11.632578 [info] [raftstore]: shard 2 applied changes [{ChangeType:AddNode Peer:{ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] at epoch {ConfVer:3 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, new peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:11.632638 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:11.632675 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 skip with no delegates, 0 commits, result <nil>
2021/08/20 14:25:11.632644 [info] prophet: resource 2 ConfVer changed from 2 to 3
{"level":"error","ts":"2021-08-20T14:25:11.632+0800","caller":"raft/log.go:354","msg":"3 >>>>>>>>>>>>>>>> [1:10:6)","stacktrace":"go.etcd.io/etcd/pkg/logutil.(*zapRaftLogger).Errorf\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/pkg/logutil/zap_raft.go:69\ngo.etcd.io/etcd/raft.(*raftLog).mustCheckOutOfBounds\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:354\ngo.etcd.io/etcd/raft.(*raftLog).slice\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:308\ngo.etcd.io/etcd/raft.(*raftLog).entries\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:258\ngo.etcd.io/etcd/raft.(*raft).maybeSendAppend\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:459\ngo.etcd.io/etcd/raft.(*raft).sendAppend\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:442\ngo.etcd.io/etcd/raft.stepLeader\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:1208\ngo.etcd.io/etcd/raft.(*raft).Step\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:995\ngo.etcd.io/etcd/raft.(*RawNode).Step\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/rawnode.go:114\ngithub.com/matrixorigin/matrixcube/raftstore.(*peerReplica).handleStep\n\t/root/projects/matrixcube/raftstore/peer_event_loop.go:257\ngithub.com/matrixorigin/matrixcube/raftstore.(*peerReplica).handleEvent\n\t/root/projects/matrixcube/raftstore/peer_event_loop.go:183\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask.func1.1\n\t/root/projects/matrixcube/raftstore/store.go:420\nsync.(*Map).Range\n\t/opt/lib/go/src/sync/map.go:345\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask.func1\n\t/root/projects/matrixcube/raftstore/store.go:418\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask\n\t/root/projects/matrixcube/raftstore/store.go:443\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).startRaftWorkers.func1\n\t/root/projects/matrixcube/raftstore/store.go:406\ngithub.com/fagongzi/util/task.(*Runner).RunCancelableTask.func1\n\t/root/projects/matrixcube/vendor/github.com/fagongzi/util/task/task.go:350"}
2021/08/20 14:25:11.633034 [info] [raftstore]: shard 2 snapshot created, epoch=<confVer:3 version:1 > term=<6> index=<9>
2021/08/20 14:25:11.633798 [info] prophet: resource 2 operator(add-rule-peer {add peer: container [5]} (kind:resource,replica, resource:2(1,1), createAt:2021-08-20 14:25:11.6180939 +0800 CST m=+7.151620501, startAt:2021-08-20 14:25:11.6181373 +0800 CST m=+7.151663901, currentStep:2, steps:[add learner peer 6 on container 5, promote learner peer 6 on container 5 to voter]) finished) finish, takes 15.6469ms,
2021/08/20 14:25:11.725651 [info] [raftstore]: begin start raftstore
2021/08/20 14:25:11.725663 [info] [raftstore]: begin to start prophet
2021-08-20 14:25:11.726450 W | etcdserver: not healthy for reconfigure, rejecting member add {ID:c9f427f3296b95b1 RaftAttributes:{PeerURLs:[http://127.0.0.1:50002] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
{"level":"warn","ts":"2021-08-20T14:25:11.726+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-eaac19d1-edee-457e-a78e-889917edda0a/127.0.0.1:40000","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: unhealthy cluster"}
2021/08/20 14:25:11.726550 [error] prophet: add member to embed etcd failed with etcdserver: unhealthy cluster, retry later
2021/08/20 14:25:11.774402 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:11.774442 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 skip with no delegates, 0 commits, result <nil>
{"level":"error","ts":"2021-08-20T14:25:11.774+0800","caller":"raft/log.go:354","msg":"3 >>>>>>>>>>>>>>>> [1:10:6)","stacktrace":"go.etcd.io/etcd/pkg/logutil.(*zapRaftLogger).Errorf\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/pkg/logutil/zap_raft.go:69\ngo.etcd.io/etcd/raft.(*raftLog).mustCheckOutOfBounds\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:354\ngo.etcd.io/etcd/raft.(*raftLog).slice\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:308\ngo.etcd.io/etcd/raft.(*raftLog).entries\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:258\ngo.etcd.io/etcd/raft.(*raft).maybeSendAppend\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:459\ngo.etcd.io/etcd/raft.(*raft).sendAppend\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:442\ngo.etcd.io/etcd/raft.stepLeader\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:1208\ngo.etcd.io/etcd/raft.(*raft).Step\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:995\ngo.etcd.io/etcd/raft.(*RawNode).Step\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/rawnode.go:114\ngithub.com/matrixorigin/matrixcube/raftstore.(*peerReplica).handleStep\n\t/root/projects/matrixcube/raftstore/peer_event_loop.go:257\ngithub.com/matrixorigin/matrixcube/raftstore.(*peerReplica).handleEvent\n\t/root/projects/matrixcube/raftstore/peer_event_loop.go:183\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask.func1.1\n\t/root/projects/matrixcube/raftstore/store.go:420\nsync.(*Map).Range\n\t/opt/lib/go/src/sync/map.go:345\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask.func1\n\t/root/projects/matrixcube/raftstore/store.go:418\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask\n\t/root/projects/matrixcube/raftstore/store.go:443\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).startRaftWorkers.func1\n\t/root/projects/matrixcube/raftstore/store.go:406\ngithub.com/fagongzi/util/task.(*Runner).RunCancelableTask.func1\n\t/root/projects/matrixcube/vendor/github.com/fagongzi/util/task/task.go:350"}
2021/08/20 14:25:11.774593 [info] [transport]: shard 2 start send pending snap, epoch=<confVer:3 version:1 > term=<6> index=<9>
2021/08/20 14:25:11.774712 [info] [transport]: session 2[127.0.0.1:58410] connected
2021/08/20 14:25:11.774716 [info] [raftstore]: shard 2 try to send snap, header=<shard:<id:2 epoch:<confVer:3 version:1 > peers:<id:3 containerID:1 > peers:<id:6 containerID:5 > > from:<id:3 containerID:1 > to:<id:6 containerID:5 role:Learner > term:6 index:9 >,size=<124>
2021/08/20 14:25:11.774768 [info] [raftstore]: shard 2 send snap complete
2021/08/20 14:25:11.774787 [info] [transport]: shard 2 pending snap sent succ, size=<124>, epoch=<confVer:3 version:1 > term=<6> index=<9>
2021/08/20 14:25:11.774717 [info] [transport]: connected to store 5
{"level":"info","ts":"2021-08-20T14:25:11.774+0800","caller":"raft/log.go:301","msg":"log [committed=0, applied=0, unstable.offset=1, len(unstable.Entries)=0] starts to restore snapshot [index: 9, term: 6]"}
{"level":"info","ts":"2021-08-20T14:25:11.774+0800","caller":"raft/raft.go:1550","msg":"6 switched to configuration voters=(3 6)"}
{"level":"info","ts":"2021-08-20T14:25:11.774+0800","caller":"raft/raft.go:1503","msg":"6 [commit: 9, lastindex: 9, lastterm: 6] restored snapshot [index: 9, term: 6]"}
{"level":"info","ts":"2021-08-20T14:25:11.774+0800","caller":"raft/raft.go:1420","msg":"6 [commit: 9] restored snapshot [index: 9, term: 6]"}
2021/08/20 14:25:11.774924 [info] [raftstore]: shard 2 begin to apply snapshot
2021/08/20 14:25:11.774939 [info] [raftstore]: shard 2 clear 0 meta keys and 0 raft keys
2021/08/20 14:25:11.774948 [info] [raftstore]: shard 2 apply snapshot state completed, apply state {AppliedIndex:9 TruncatedState:{Index:9 Term:6 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:11.774964 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 start applying snapshot
2021/08/20 14:25:11.774984 [info] [raftstore]: shard 2 begin apply snapshot data
2021/08/20 14:25:11.775074 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:11.775127 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 add registration job by handleRaftReadyApply
2021/08/20 14:25:11.775148 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 applying snapshot to set read 9
2021/08/20 14:25:11.775172 [info] [raftstore]: shard 2 snapshot is applied, shard=<{ID:2 Start:[] End:[] Epoch:{ConfVer:3 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} State:Running Peers:[{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] DisableSplit:false Group:0 Unique: RuleGroups:[] Data:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:11.775199 [info] [raftstore]: shard 2 shard changed after apply snapshot, from=<{ID:2 Start:[] End:[] Epoch:{ConfVer:2 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} State:Running Peers:[{ID:6 ContainerID:5 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] DisableSplit:false Group:0 Unique: RuleGroups:[] Data:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}> to=<{ID:2 Start:[] End:[] Epoch:{ConfVer:3 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} State:Running Peers:[{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] DisableSplit:false Group:0 Unique: RuleGroups:[] Data:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:11.775216 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 AdvanceApply with 9
2021/08/20 14:25:11.775303 [info] [raftstore]: shard 2 delete exists snap gz file /tmp/cube/node-1/snapshots/2_6_9.gz, header is shard:<id:2 epoch:<confVer:2 version:1 > peers:<id:6 containerID:5 role:Learner > peers:<id:3 containerID:1 > > from:<> to:<> term:6 index:9
2021/08/20 14:25:11.775363 [info] [raftstore]: shard 2 apply snapshot data complete, {Term:6 Vote:0 Commit:9 XXX_unrecognized:[]}
2021/08/20 14:25:11.775380 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 add registration job by handleRaftReadyApply, completed
2021/08/20 14:25:11.823859 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:11.973889 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:12.174021 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021-08-20 14:25:12.227551 W | etcdserver: not healthy for reconfigure, rejecting member add {ID:8ae68e5303dd0380 RaftAttributes:{PeerURLs:[http://127.0.0.1:50002] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
{"level":"warn","ts":"2021-08-20T14:25:12.227+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-eaac19d1-edee-457e-a78e-889917edda0a/127.0.0.1:40000","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: unhealthy cluster"}
2021/08/20 14:25:12.227673 [error] prophet: add member to embed etcd failed with etcdserver: unhealthy cluster, retry later
2021/08/20 14:25:12.373764 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:12.573921 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021-08-20 14:25:12.727881 W | etcdserver: not healthy for reconfigure, rejecting member add {ID:8ae68e5303dd0380 RaftAttributes:{PeerURLs:[http://127.0.0.1:50002] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
{"level":"warn","ts":"2021-08-20T14:25:12.728+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-eaac19d1-edee-457e-a78e-889917edda0a/127.0.0.1:40000","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: unhealthy cluster"}
2021/08/20 14:25:12.728136 [error] prophet: add member to embed etcd failed with etcdserver: unhealthy cluster, retry later
2021/08/20 14:25:12.773859 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:12.974356 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:13.174011 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021-08-20 14:25:13.229073 W | etcdserver: not healthy for reconfigure, rejecting member add {ID:755c3d9656559ea9 RaftAttributes:{PeerURLs:[http://127.0.0.1:50002] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
{"level":"warn","ts":"2021-08-20T14:25:13.229+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-eaac19d1-edee-457e-a78e-889917edda0a/127.0.0.1:40000","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: unhealthy cluster"}
2021/08/20 14:25:13.229207 [error] prophet: add member to embed etcd failed with etcdserver: unhealthy cluster, retry later
2021/08/20 14:25:13.373871 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:13.573830 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021-08-20 14:25:13.729729 W | etcdserver: not healthy for reconfigure, rejecting member add {ID:755c3d9656559ea9 RaftAttributes:{PeerURLs:[http://127.0.0.1:50002] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
{"level":"warn","ts":"2021-08-20T14:25:13.729+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-eaac19d1-edee-457e-a78e-889917edda0a/127.0.0.1:40000","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: unhealthy cluster"}
2021/08/20 14:25:13.729849 [error] prophet: add member to embed etcd failed with etcdserver: unhealthy cluster, retry later
2021/08/20 14:25:13.774383 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:13.973641 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:14.174317 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021-08-20 14:25:14.231061 W | etcdserver: not healthy for reconfigure, rejecting member add {ID:ad228268458467e4 RaftAttributes:{PeerURLs:[http://127.0.0.1:50002] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
{"level":"warn","ts":"2021-08-20T14:25:14.231+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-eaac19d1-edee-457e-a78e-889917edda0a/127.0.0.1:40000","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: unhealthy cluster"}
2021/08/20 14:25:14.231177 [error] prophet: add member to embed etcd failed with etcdserver: unhealthy cluster, retry later
2021/08/20 14:25:14.374372 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:14.574003 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021-08-20 14:25:14.732249 W | etcdserver: not healthy for reconfigure, rejecting member add {ID:ad228268458467e4 RaftAttributes:{PeerURLs:[http://127.0.0.1:50002] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
{"level":"warn","ts":"2021-08-20T14:25:14.732+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-eaac19d1-edee-457e-a78e-889917edda0a/127.0.0.1:40000","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: unhealthy cluster"}
2021/08/20 14:25:14.732390 [error] prophet: add member to embed etcd failed with etcdserver: unhealthy cluster, retry later
2021/08/20 14:25:14.774271 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:14.973416 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:15.174370 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021-08-20 14:25:15.233355 W | etcdserver: not healthy for reconfigure, rejecting member add {ID:9b724de3bf934c9d RaftAttributes:{PeerURLs:[http://127.0.0.1:50002] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
{"level":"warn","ts":"2021-08-20T14:25:15.233+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-eaac19d1-edee-457e-a78e-889917edda0a/127.0.0.1:40000","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: unhealthy cluster"}
2021/08/20 14:25:15.233470 [error] prophet: add member to embed etcd failed with etcdserver: unhealthy cluster, retry later
2021/08/20 14:25:15.373483 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:15.573512 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021-08-20 14:25:15.734727 W | etcdserver: not healthy for reconfigure, rejecting member add {ID:9b724de3bf934c9d RaftAttributes:{PeerURLs:[http://127.0.0.1:50002] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
{"level":"warn","ts":"2021-08-20T14:25:15.734+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-eaac19d1-edee-457e-a78e-889917edda0a/127.0.0.1:40000","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: unhealthy cluster"}
2021/08/20 14:25:15.734875 [error] prophet: add member to embed etcd failed with etcdserver: unhealthy cluster, retry later
2021/08/20 14:25:15.774259 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:15.973422 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:16.173749 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021-08-20 14:25:16.235976 W | etcdserver: not healthy for reconfigure, rejecting member add {ID:6b1ace49a9f387c8 RaftAttributes:{PeerURLs:[http://127.0.0.1:50002] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
{"level":"warn","ts":"2021-08-20T14:25:16.236+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-eaac19d1-edee-457e-a78e-889917edda0a/127.0.0.1:40000","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: unhealthy cluster"}
2021/08/20 14:25:16.236125 [error] prophet: add member to embed etcd failed with etcdserver: unhealthy cluster, retry later
2021/08/20 14:25:16.374129 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:16.573640 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
{"level":"info","ts":"2021-08-20T14:25:16.738+0800","caller":"raft/raft.go:1550","msg":"93ccda9956d988f4 switched to configuration voters=(7717707727232010184 10650127570946328820 12791365408210890720)"}
{"level":"info","ts":"2021-08-20T14:25:16.738+0800","caller":"raft/raft.go:1550","msg":"b1840f1134a12fe0 switched to configuration voters=(7717707727232010184 10650127570946328820 12791365408210890720)"}
2021-08-20 14:25:16.738534 I | etcdserver/membership: added member 6b1ace49a9f387c8 [http://127.0.0.1:50002] to cluster 6a4ade671189dc2f
2021-08-20 14:25:16.738538 I | etcdserver/membership: added member 6b1ace49a9f387c8 [http://127.0.0.1:50002] to cluster 6a4ade671189dc2f
2021-08-20 14:25:16.738541 I | rafthttp: starting peer 6b1ace49a9f387c8...
2021-08-20 14:25:16.738544 I | rafthttp: starting peer 6b1ace49a9f387c8...
2021-08-20 14:25:16.738554 I | rafthttp: started HTTP pipelining with peer 6b1ace49a9f387c8
2021-08-20 14:25:16.738679 I | rafthttp: started HTTP pipelining with peer 6b1ace49a9f387c8
2021-08-20 14:25:16.738776 I | rafthttp: started streaming with peer 6b1ace49a9f387c8 (writer)
2021-08-20 14:25:16.738849 I | rafthttp: started streaming with peer 6b1ace49a9f387c8 (writer)
2021-08-20 14:25:16.738930 I | rafthttp: started streaming with peer 6b1ace49a9f387c8 (writer)
2021-08-20 14:25:16.739262 I | rafthttp: started streaming with peer 6b1ace49a9f387c8 (stream MsgApp v2 reader)
2021-08-20 14:25:16.739287 I | rafthttp: started streaming with peer 6b1ace49a9f387c8 (writer)
2021-08-20 14:25:16.739442 I | rafthttp: started peer 6b1ace49a9f387c8
2021-08-20 14:25:16.739488 I | rafthttp: started streaming with peer 6b1ace49a9f387c8 (stream Message reader)
2021-08-20 14:25:16.739525 I | rafthttp: added peer 6b1ace49a9f387c8
2021-08-20 14:25:16.740525 I | rafthttp: started peer 6b1ace49a9f387c8
2021-08-20 14:25:16.740543 I | rafthttp: added peer 6b1ace49a9f387c8
2021-08-20 14:25:16.740718 I | rafthttp: started streaming with peer 6b1ace49a9f387c8 (stream Message reader)
2021-08-20 14:25:16.741355 I | rafthttp: started streaming with peer 6b1ace49a9f387c8 (stream MsgApp v2 reader)
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-08-20 14:25:16.742152 I | embed: pprof is enabled under /debug/pprof
2021-08-20 14:25:16.742193 I | embed: name = node-2
2021-08-20 14:25:16.742225 I | embed: data dir = /tmp/cube/node-2/prophet
2021-08-20 14:25:16.742227 I | embed: member dir = /tmp/cube/node-2/prophet/member
2021-08-20 14:25:16.742229 I | embed: heartbeat = 100ms
2021-08-20 14:25:16.742230 I | embed: election = 3000ms
2021-08-20 14:25:16.742232 I | embed: snapshot count = 100000
2021-08-20 14:25:16.742235 I | embed: advertise client URLs = http://127.0.0.1:40002
2021-08-20 14:25:16.746866 I | etcdserver: starting member 6b1ace49a9f387c8 in cluster 6a4ade671189dc2f
{"level":"info","ts":"2021-08-20T14:25:16.746+0800","caller":"raft/raft.go:1550","msg":"6b1ace49a9f387c8 switched to configuration voters=()"}
{"level":"info","ts":"2021-08-20T14:25:16.746+0800","caller":"raft/raft.go:711","msg":"6b1ace49a9f387c8 became follower at term 0"}
{"level":"info","ts":"2021-08-20T14:25:16.746+0800","caller":"raft/raft.go:388","msg":"newRaft 6b1ace49a9f387c8 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"}
2021-08-20 14:25:16.748565 W | auth: simple token is not cryptographically signed
2021-08-20 14:25:16.751447 I | rafthttp: started HTTP pipelining with peer 93ccda9956d988f4
2021-08-20 14:25:16.751464 I | rafthttp: started HTTP pipelining with peer b1840f1134a12fe0
2021-08-20 14:25:16.751469 I | rafthttp: starting peer 93ccda9956d988f4...
2021-08-20 14:25:16.751473 I | rafthttp: started HTTP pipelining with peer 93ccda9956d988f4
2021-08-20 14:25:16.751663 I | rafthttp: started streaming with peer 93ccda9956d988f4 (writer)
2021-08-20 14:25:16.751722 I | rafthttp: started peer 93ccda9956d988f4
2021-08-20 14:25:16.751734 I | rafthttp: started streaming with peer 93ccda9956d988f4 (writer)
2021-08-20 14:25:16.751737 I | rafthttp: added peer 93ccda9956d988f4
2021-08-20 14:25:16.751739 I | rafthttp: started streaming with peer 93ccda9956d988f4 (stream Message reader)
2021-08-20 14:25:16.751744 I | rafthttp: starting peer b1840f1134a12fe0...
2021-08-20 14:25:16.751752 I | rafthttp: started HTTP pipelining with peer b1840f1134a12fe0
2021-08-20 14:25:16.751811 I | rafthttp: started streaming with peer 93ccda9956d988f4 (stream MsgApp v2 reader)
2021-08-20 14:25:16.751919 I | rafthttp: started streaming with peer b1840f1134a12fe0 (writer)
2021-08-20 14:25:16.751948 I | rafthttp: peer 6b1ace49a9f387c8 became active
2021-08-20 14:25:16.751959 I | rafthttp: established a TCP streaming connection with peer 6b1ace49a9f387c8 (stream Message writer)
2021-08-20 14:25:16.751983 I | rafthttp: established a TCP streaming connection with peer 6b1ace49a9f387c8 (stream MsgApp v2 writer)
2021-08-20 14:25:16.751999 I | rafthttp: peer 93ccda9956d988f4 became active
2021-08-20 14:25:16.752005 I | rafthttp: established a TCP streaming connection with peer 93ccda9956d988f4 (stream Message reader)
2021-08-20 14:25:16.752023 I | rafthttp: started streaming with peer b1840f1134a12fe0 (writer)
2021-08-20 14:25:16.752025 I | rafthttp: established a TCP streaming connection with peer 93ccda9956d988f4 (stream MsgApp v2 reader)
2021-08-20 14:25:16.752042 I | rafthttp: started peer b1840f1134a12fe0
2021-08-20 14:25:16.752060 I | rafthttp: added peer b1840f1134a12fe0
2021-08-20 14:25:16.752065 I | etcdserver: starting server... [version: 3.4.15, cluster version: to_be_decided]
2021-08-20 14:25:16.752099 I | rafthttp: started streaming with peer b1840f1134a12fe0 (stream Message reader)
2021-08-20 14:25:16.752142 I | rafthttp: started streaming with peer b1840f1134a12fe0 (stream MsgApp v2 reader)
2021-08-20 14:25:16.752319 I | rafthttp: peer 6b1ace49a9f387c8 became active
2021-08-20 14:25:16.752338 I | rafthttp: established a TCP streaming connection with peer 6b1ace49a9f387c8 (stream Message writer)
2021-08-20 14:25:16.752343 I | rafthttp: established a TCP streaming connection with peer 6b1ace49a9f387c8 (stream MsgApp v2 writer)
2021-08-20 14:25:16.752349 I | embed: listening for peers on 127.0.0.1:50002
2021-08-20 14:25:16.752362 I | rafthttp: peer b1840f1134a12fe0 became active
2021-08-20 14:25:16.752368 I | rafthttp: established a TCP streaming connection with peer b1840f1134a12fe0 (stream MsgApp v2 reader)
2021-08-20 14:25:16.752393 I | rafthttp: established a TCP streaming connection with peer b1840f1134a12fe0 (stream Message reader)
2021/08/20 14:25:16.773996 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021-08-20 14:25:16.803197 I | etcdserver: 6b1ace49a9f387c8 initialized peer connection; fast-forwarding 28 ticks (election ticks 30) with 2 active peer(s)
{"level":"info","ts":"2021-08-20T14:25:16.810+0800","caller":"raft/raft.go:870","msg":"6b1ace49a9f387c8 [term: 0] received a MsgHeartbeat message with higher term from 93ccda9956d988f4 [term: 2]"}
{"level":"info","ts":"2021-08-20T14:25:16.810+0800","caller":"raft/raft.go:711","msg":"6b1ace49a9f387c8 became follower at term 2"}
{"level":"info","ts":"2021-08-20T14:25:16.810+0800","caller":"raft/node.go:325","msg":"raft.node: 6b1ace49a9f387c8 elected leader 93ccda9956d988f4 at term 2"}
{"level":"info","ts":"2021-08-20T14:25:16.812+0800","caller":"raft/raft.go:1550","msg":"6b1ace49a9f387c8 switched to configuration voters=(10650127570946328820)"}
2021-08-20 14:25:16.812150 I | etcdserver/membership: added member 93ccda9956d988f4 [http://127.0.0.1:50000] to cluster 6a4ade671189dc2f
2021-08-20 14:25:16.812194 N | etcdserver/membership: set the initial cluster version to 3.4
{"level":"info","ts":"2021-08-20T14:25:16.812+0800","caller":"raft/raft.go:1550","msg":"6b1ace49a9f387c8 switched to configuration voters=(10650127570946328820 12791365408210890720)"}
2021-08-20 14:25:16.812558 I | etcdserver/membership: added member b1840f1134a12fe0 [http://127.0.0.1:50001] to cluster 6a4ade671189dc2f
{"level":"info","ts":"2021-08-20T14:25:16.812+0800","caller":"raft/raft.go:1550","msg":"6b1ace49a9f387c8 switched to configuration voters=(7717707727232010184 10650127570946328820 12791365408210890720)"}
2021-08-20 14:25:16.812756 I | etcdserver/membership: added member 6b1ace49a9f387c8 [http://127.0.0.1:50002] to cluster 6a4ade671189dc2f
2021-08-20 14:25:16.813326 I | etcdserver: published {Name:node-2 ClientURLs:[http://127.0.0.1:40002]} to cluster 6a4ade671189dc2f
2021-08-20 14:25:16.813341 I | embed: ready to serve client requests
2021/08/20 14:25:16.813342 [info] prophet: create etcd v3 client with endpoints [http://127.0.0.1:40002]
2021-08-20 14:25:16.813579 N | embed: serving insecure client requests on 127.0.0.1:40002, this is strongly discouraged!
2021/08/20 14:25:16.814267 [info] prophet: net application started
2021/08/20 14:25:16.814274 [info] prophet: start system time monitor
2021/08/20 14:25:16.814742 [info] prophet: [prophet-leader/node-2]/loop: current leader is [ôæ¶Ó¶ænode-0127.0.0.1:30000]
2021/08/20 14:25:16.814755 [info] prophet: [prophet-leader/node-2]/loop: start watch leader ôæ¶Ó¶ænode-0127.0.0.1:30000
2021/08/20 14:25:16.814757 [info] prophet: ********node-2 become to follower now********
2021/08/20 14:25:16.814786 [info] prophet: client start init connection to leader 127.0.0.1:30000
2021/08/20 14:25:16.814787 [info] prophet: client read loop started
2021/08/20 14:25:16.814791 [info] prophet: client started
2021/08/20 14:25:16.814844 [info] prophet: session 5[127.0.0.1:38582] connected
2021/08/20 14:25:16.814845 [info] [raftstore]: *********Become prophet follower*********
2021/08/20 14:25:16.814882 [info] [raftstore]: begin to bootstrap the cluster
2021/08/20 14:25:16.814901 [info] [raftstore]: raftstore init with store raftAddr:"127.0.0.1:10002" clientAddr:"127.0.0.1:20002" labels:<key:"c" value:"2" > startTime:1629440716 deployPath:"not set"
2021/08/20 14:25:16.814904 [info] [raftstore]: begin to create local store metadata
2021/08/20 14:25:16.814854 [info] prophet: client start connect to leader 127.0.0.1:30000
2021/08/20 14:25:16.814949 [info] prophet: client read loop actived, ready to read from leader
2021/08/20 14:25:16.815020 [info] [raftstore]: create local store with id 7
2021/08/20 14:25:16.815451 [info] [raftstore]: the cluster already bootstrap: true
2021/08/20 14:25:16.817755 [info] prophet: container 7 limit added
2021/08/20 14:25:16.817834 [info] [raftstore]: prophet started
2021/08/20 14:25:16.817889 [info] [transport]: net application started
2021/08/20 14:25:16.817899 [info] [raftstore]: start listen at 127.0.0.1:10002 for raft
2021/08/20 14:25:16.817901 [info] [raftstore]: raft shards workers started
2021/08/20 14:25:16.817834 [info] prophet: container 7 heartbeat stream binded
2021/08/20 14:25:16.817904 [info] [raftstore]: starts with 0 shards, including 0 tombstones and 0 applying shards
2021/08/20 14:25:16.817909 [info] [raftstore]: cleanup possible garbage data complete
2021/08/20 14:25:16.817909 [info] [raftstore]: raft worker 0/0 start
2021/08/20 14:25:16.817910 [info] [raftstore]: shards started
2021/08/20 14:25:16.817912 [info] [raftstore]: shard timer based tasks started
2021/08/20 14:25:16.817919 [info] [raftstore]: net application started
2021/08/20 14:25:16.817920 [info] [raftstore]: start listen at 127.0.0.1:20002 for client
2021/08/20 14:25:16.817930 [info] [raftstore]: router started
2021/08/20 14:25:16.817944 [info] prophet: client start init connection to leader 127.0.0.1:30000
2021/08/20 14:25:16.817948 [info] [raftstore]: router event loop task started
2021/08/20 14:25:16.818005 [info] prophet: client start connect to leader 127.0.0.1:30000
2021/08/20 14:25:16.818007 [info] prophet: watcher init leader connection 127.0.0.1:30000 succeed
2021/08/20 14:25:16.818013 [info] prophet: session 6[127.0.0.1:38584] connected
2021/08/20 14:25:16.818034 [info] prophet: new watcher 127.0.0.1:38584 added
2021/08/20 14:25:16.818108 [info] [raftstore]: start init event
2021-08-20 14:25:16.840291 I | rafthttp: established a TCP streaming connection with peer b1840f1134a12fe0 (stream MsgApp v2 writer)
2021-08-20 14:25:16.840311 I | rafthttp: established a TCP streaming connection with peer 6b1ace49a9f387c8 (stream MsgApp v2 reader)
2021-08-20 14:25:16.840326 I | rafthttp: established a TCP streaming connection with peer b1840f1134a12fe0 (stream Message writer)
2021-08-20 14:25:16.840337 I | rafthttp: established a TCP streaming connection with peer 6b1ace49a9f387c8 (stream Message reader)
2021/08/20 14:25:16.842534 [info] prophet: resource 2 add operator add-rule-peer {add peer: container [7]} (kind:resource,replica, resource:2(1,3), createAt:2021-08-20 14:25:16.8425177 +0800 CST m=+12.376044301, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, steps:[add learner peer 8 on container 7, promote learner peer 8 on container 7 to voter]),
2021/08/20 14:25:16.842549 [info] prophet: resource 2 send schedule command, step add learner peer 8 on container 7, source create
2021/08/20 14:25:16.842607 [info] [raftstore]: shard-2 AddLearnerNode peer {ID:8 ContainerID:7 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021-08-20 14:25:16.842630 I | rafthttp: established a TCP streaming connection with peer 93ccda9956d988f4 (stream Message writer)
2021-08-20 14:25:16.842649 I | rafthttp: established a TCP streaming connection with peer 6b1ace49a9f387c8 (stream MsgApp v2 reader)
2021-08-20 14:25:16.842651 I | rafthttp: established a TCP streaming connection with peer 6b1ace49a9f387c8 (stream Message reader)
2021-08-20 14:25:16.842660 I | rafthttp: established a TCP streaming connection with peer 93ccda9956d988f4 (stream MsgApp v2 writer)
2021/08/20 14:25:16.850350 [info] [raftstore]: shard-2 propose conf change peer [{ChangeType:AddLearnerNode Peer:{ID:8 ContainerID:7 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:16.850452 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:16.850608 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:16.850634 [info] [raftstore]: shard 2 do apply change peer changeType:AddLearnerNode peer:<id:8 containerID:7 role:Learner > at epoch {ConfVer:3 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:16.850646 [info] [raftstore]: shard-2 add learner peer {ID:8 ContainerID:7 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} successfully, peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:8 ContainerID:7 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:16.850675 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 AdvanceApply with 10
{"level":"info","ts":"2021-08-20T14:25:16.850+0800","caller":"raft/raft.go:1550","msg":"3 switched to configuration voters=(3 6) learners=(8)"}
2021/08/20 14:25:16.850749 [info] [raftstore]: shard 2 notify pd with {Transition:ConfChangeTransitionAuto Changes:[{Type:ConfChangeAddLearnerNode NodeID:8 XXX_unrecognized:[]}] Context:[10 32 10 16 199 204 83 164 116 131 73 101 184 121 210 1 243 92 244 168 16 2 26 4 8 3 16 1 42 4 8 3 16 1 26 14 8 1 18 10 8 3 18 6 8 8 16 7 24 1] XXX_unrecognized:[]} with changes [{ChangeType:AddLearnerNode Peer:{ID:8 ContainerID:7 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] at epoch {ConfVer:4 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:16.850772 [info] [raftstore]: shard 2 applied changes [{ChangeType:AddLearnerNode Peer:{ID:8 ContainerID:7 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] at epoch {ConfVer:4 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, new peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:8 ContainerID:7 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:16.850786 [info] prophet: [prophet-leader/node-2]/watch: leader updated to ôæ¶Ó¶ænode-0127.0.0.1:30000
2021/08/20 14:25:16.850681 [info] [raftstore]: shard 2 do apply change peer changeType:AddLearnerNode peer:<id:8 containerID:7 role:Learner > at epoch {ConfVer:3 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:16.850835 [info] [raftstore]: shard-2 add learner peer {ID:8 ContainerID:7 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} successfully, peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:8 ContainerID:7 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:16.850866 [info] prophet: resource 2 ConfVer changed from 3 to 4
2021/08/20 14:25:16.851930 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 AdvanceApply with 10
{"level":"info","ts":"2021-08-20T14:25:16.851+0800","caller":"raft/raft.go:1550","msg":"6 switched to configuration voters=(3 6) learners=(8)"}
2021/08/20 14:25:16.851992 [info] [raftstore]: shard 2 applied changes [{ChangeType:AddLearnerNode Peer:{ID:8 ContainerID:7 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] at epoch {ConfVer:4 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, new peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:8 ContainerID:7 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:16.852582 [info] prophet: resource 2 send schedule command, step promote learner peer 8 on container 7 to voter, source heartbeat
2021/08/20 14:25:16.852665 [info] [transport]: connected to store 7
2021/08/20 14:25:16.852690 [info] [raftstore]: shard-2 AddNode peer {ID:8 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:16.852711 [info] [transport]: session 1[127.0.0.1:40862] connected
2021/08/20 14:25:16.852729 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:16.852750 [info] [raftstore]: shard 2 target peer doesn't exist, peer=<{ID:8 ContainerID:7 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}> message=<MsgApp>
2021/08/20 14:25:16.852757 [info] [raftstore]: shard 2 replicate peer {ID:8 ContainerID:7 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:16.852772 [info] [raftstore]: shard 2 init with raft local state {HardState:{Term:0 Vote:0 Commit:0 XXX_unrecognized:[]} LastIndex:0 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:16.852786 [info] [raftstore]: shard 2 init raft apply state, state=<{AppliedIndex:0 TruncatedState:{Index:0 Term:0 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:16.852787 [info] [raftstore]: shard 2 init last term, last term=<0>
2021/08/20 14:25:16.852790 [info] [raftstore]: shard 2 change init applied index from 0 to 0
2021/08/20 14:25:16.852801 [info] [raftstore]: shard-2 propose conf change peer [{ChangeType:AddNode Peer:{ID:8 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
{"level":"info","ts":"2021-08-20T14:25:16.852+0800","caller":"raft/raft.go:1550","msg":"8 switched to configuration voters=()"}
{"level":"info","ts":"2021-08-20T14:25:16.852+0800","caller":"raft/raft.go:711","msg":"8 became follower at term 0"}
{"level":"info","ts":"2021-08-20T14:25:16.852+0800","caller":"raft/raft.go:388","msg":"newRaft 8 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"}
2021/08/20 14:25:16.852897 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:16.852937 [info] [raftstore]: shard 2 peer 8 raft 8 added, epoch {ConfVer:4 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:8 ContainerID:7 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}], raft worker 0, apply worker apply-0-0
{"level":"info","ts":"2021-08-20T14:25:16.852+0800","caller":"raft/raft.go:870","msg":"8 [term: 0] received a MsgHeartbeat message with higher term from 3 [term: 6]"}
{"level":"info","ts":"2021-08-20T14:25:16.853+0800","caller":"raft/raft.go:711","msg":"8 became follower at term 6"}
2021/08/20 14:25:16.853027 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:16.853031 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 send 1 msgs
2021/08/20 14:25:16.853038 [info] [raftstore]: shard 2 do apply change peer peer:<id:8 containerID:7 > at epoch {ConfVer:4 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:8 ContainerID:7 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:16.853046 [info] [raftstore]: shard-2 add peer {ID:8 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} successfully, peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:8 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:16.853054 [info] [raftstore]: shard 2 do apply change peer peer:<id:8 containerID:7 > at epoch {ConfVer:4 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:8 ContainerID:7 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:16.853055 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 AdvanceApply with 11
2021/08/20 14:25:16.853062 [info] [raftstore]: shard-2 add peer {ID:8 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} successfully, peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:8 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:16.853071 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 AdvanceApply with 11
{"level":"info","ts":"2021-08-20T14:25:16.853+0800","caller":"raft/raft.go:1550","msg":"6 switched to configuration voters=(3 6 8)"}
2021/08/20 14:25:16.853076 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 skip with no delegates, 0 commits, result <nil>
2021/08/20 14:25:16.853111 [info] [raftstore]: shard 2 applied changes [{ChangeType:AddNode Peer:{ID:8 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] at epoch {ConfVer:5 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, new peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:8 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
{"level":"info","ts":"2021-08-20T14:25:16.853+0800","caller":"raft/raft.go:1550","msg":"3 switched to configuration voters=(3 6 8)"}
2021/08/20 14:25:16.853126 [info] [raftstore]: shard 2 notify pd with {Transition:ConfChangeTransitionAuto Changes:[{Type:ConfChangeAddNode NodeID:8 XXX_unrecognized:[]}] Context:[10 32 10 16 39 152 60 195 179 112 78 217 147 7 41 2 55 168 12 203 16 2 26 4 8 3 16 1 42 4 8 4 16 1 26 10 8 1 18 6 18 4 8 8 16 7] XXX_unrecognized:[]} with changes [{ChangeType:AddNode Peer:{ID:8 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] at epoch {ConfVer:5 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:16.853145 [info] [raftstore]: shard 2 applied changes [{ChangeType:AddNode Peer:{ID:8 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] at epoch {ConfVer:5 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, new peers [{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:8 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:16.853178 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:16.853222 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 send 1 msgs
2021/08/20 14:25:16.853229 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 skip with no delegates, 0 commits, result <nil>
2021/08/20 14:25:16.853232 [info] prophet: resource 2 ConfVer changed from 4 to 5
2021/08/20 14:25:16.853852 [info] [transport]: connected to store 1
2021/08/20 14:25:16.853878 [info] [transport]: session 2[127.0.0.1:36160] connected
2021/08/20 14:25:16.853941 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 send 2 msgs
2021/08/20 14:25:16.853966 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 skip with no delegates, 0 commits, result <nil>
{"level":"error","ts":"2021-08-20T14:25:16.854+0800","caller":"raft/log.go:354","msg":"3 >>>>>>>>>>>>>>>> [1:12:6)","stacktrace":"go.etcd.io/etcd/pkg/logutil.(*zapRaftLogger).Errorf\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/pkg/logutil/zap_raft.go:69\ngo.etcd.io/etcd/raft.(*raftLog).mustCheckOutOfBounds\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:354\ngo.etcd.io/etcd/raft.(*raftLog).slice\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:308\ngo.etcd.io/etcd/raft.(*raftLog).entries\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:258\ngo.etcd.io/etcd/raft.(*raft).maybeSendAppend\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:459\ngo.etcd.io/etcd/raft.(*raft).sendAppend\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:442\ngo.etcd.io/etcd/raft.stepLeader\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:1150\ngo.etcd.io/etcd/raft.(*raft).Step\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:995\ngo.etcd.io/etcd/raft.(*RawNode).Step\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/rawnode.go:114\ngithub.com/matrixorigin/matrixcube/raftstore.(*peerReplica).handleStep\n\t/root/projects/matrixcube/raftstore/peer_event_loop.go:257\ngithub.com/matrixorigin/matrixcube/raftstore.(*peerReplica).handleEvent\n\t/root/projects/matrixcube/raftstore/peer_event_loop.go:183\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask.func1.1\n\t/root/projects/matrixcube/raftstore/store.go:420\nsync.(*Map).Range\n\t/opt/lib/go/src/sync/map.go:345\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask.func1\n\t/root/projects/matrixcube/raftstore/store.go:418\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask\n\t/root/projects/matrixcube/raftstore/store.go:443\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).startRaftWorkers.func1\n\t/root/projects/matrixcube/raftstore/store.go:406\ngithub.com/fagongzi/util/task.(*Runner).RunCancelableTask.func1\n\t/root/projects/matrixcube/vendor/github.com/fagongzi/util/task/task.go:350"}
2021/08/20 14:25:16.854063 [info] [raftstore]: shard 2 start snapshot, epoch=<{ConfVer:5 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:16.854472 [info] prophet: resource 2 operator(add-rule-peer {add peer: container [7]} (kind:resource,replica, resource:2(1,3), createAt:2021-08-20 14:25:16.8425177 +0800 CST m=+12.376044301, startAt:2021-08-20 14:25:16.8425445 +0800 CST m=+12.376071101, currentStep:2, steps:[add learner peer 8 on container 7, promote learner peer 8 on container 7 to voter]) finished) finish, takes 11.9157ms,
2021/08/20 14:25:16.855160 [info] [raftstore]: shard 2 snapshot created, epoch=<confVer:5 version:1 > term=<6> index=<11>
{"level":"error","ts":"2021-08-20T14:25:16.930+0800","caller":"raft/log.go:354","msg":"3 >>>>>>>>>>>>>>>> [1:13:6)","stacktrace":"go.etcd.io/etcd/pkg/logutil.(*zapRaftLogger).Errorf\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/pkg/logutil/zap_raft.go:69\ngo.etcd.io/etcd/raft.(*raftLog).mustCheckOutOfBounds\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:354\ngo.etcd.io/etcd/raft.(*raftLog).slice\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:308\ngo.etcd.io/etcd/raft.(*raftLog).entries\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/log.go:258\ngo.etcd.io/etcd/raft.(*raft).maybeSendAppend\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:459\ngo.etcd.io/etcd/raft.(*raft).sendAppend\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:442\ngo.etcd.io/etcd/raft.(*raft).bcastAppend.func1\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:538\ngo.etcd.io/etcd/raft/tracker.(*ProgressTracker).Visit\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/tracker/tracker.go:209\ngo.etcd.io/etcd/raft.(*raft).bcastAppend\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:534\ngo.etcd.io/etcd/raft.stepLeader\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:1090\ngo.etcd.io/etcd/raft.(*raft).Step\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/raft.go:995\ngo.etcd.io/etcd/raft.(*RawNode).Propose\n\t/root/projects/matrixcube/vendor/go.etcd.io/etcd/raft/rawnode.go:83\ngithub.com/matrixorigin/matrixcube/raftstore.(*peerReplica).proposeNormal\n\t/root/projects/matrixcube/raftstore/peer_event_proposal.go:198\ngithub.com/matrixorigin/matrixcube/raftstore.(*peerReplica).propose\n\t/root/projects/matrixcube/raftstore/peer_event_proposal.go:135\ngithub.com/matrixorigin/matrixcube/raftstore.(*peerReplica).handleRequest\n\t/root/projects/matrixcube/raftstore/peer_event_proposal.go:98\ngithub.com/matrixorigin/matrixcube/raftstore.(*peerReplica).handleEvent\n\t/root/projects/matrixcube/raftstore/peer_event_loop.go:187\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask.func1.1\n\t/root/projects/matrixcube/raftstore/store.go:420\nsync.(*Map).Range\n\t/opt/lib/go/src/sync/map.go:345\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask.func1\n\t/root/projects/matrixcube/raftstore/store.go:418\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).runPRTask\n\t/root/projects/matrixcube/raftstore/store.go:447\ngithub.com/matrixorigin/matrixcube/raftstore.(*store).startRaftWorkers.func1\n\t/root/projects/matrixcube/raftstore/store.go:406\ngithub.com/fagongzi/util/task.(*Runner).RunCancelableTask.func1\n\t/root/projects/matrixcube/vendor/github.com/fagongzi/util/task/task.go:350"}
2021/08/20 14:25:16.930623 [info] [transport]: shard 2 start send pending snap, epoch=<confVer:5 version:1 > term=<6> index=<11>
2021/08/20 14:25:16.930638 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:16.930744 [info] [raftstore]: shard 2 try to send snap, header=<shard:<id:2 epoch:<confVer:5 version:1 > peers:<id:3 containerID:1 > peers:<id:6 containerID:5 > peers:<id:8 containerID:7 > > from:<id:3 containerID:1 > to:<id:8 containerID:7 role:Learner > term:6 index:11 >,size=<124>
2021/08/20 14:25:16.930789 [info] [raftstore]: shard 2 send snap complete
2021/08/20 14:25:16.930805 [info] [transport]: shard 2 pending snap sent succ, size=<124>, epoch=<confVer:5 version:1 > term=<6> index=<11>
2021/08/20 14:25:16.930900 [info] [transport]: connected to store 7
2021/08/20 14:25:16.930915 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 AdvanceApply with 12
2021/08/20 14:25:16.930912 [info] [transport]: session 2[127.0.0.1:40866] connected
2021/08/20 14:25:16.930999 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
{"level":"info","ts":"2021-08-20T14:25:16.931+0800","caller":"raft/log.go:301","msg":"log [committed=0, applied=0, unstable.offset=1, len(unstable.Entries)=0] starts to restore snapshot [index: 11, term: 6]"}
2021/08/20 14:25:16.931066 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 AdvanceApply with 12
2021/08/20 14:25:16.931072 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
{"level":"info","ts":"2021-08-20T14:25:16.931+0800","caller":"raft/raft.go:1550","msg":"8 switched to configuration voters=(3 6 8)"}
{"level":"info","ts":"2021-08-20T14:25:16.931+0800","caller":"raft/raft.go:1503","msg":"8 [commit: 11, lastindex: 11, lastterm: 6] restored snapshot [index: 11, term: 6]"}
{"level":"info","ts":"2021-08-20T14:25:16.931+0800","caller":"raft/raft.go:1420","msg":"8 [commit: 11] restored snapshot [index: 11, term: 6]"}
2021/08/20 14:25:16.931119 [info] [raftstore]: shard 2 begin to apply snapshot
2021/08/20 14:25:16.931141 [info] [raftstore]: shard 2 clear 0 meta keys and 0 raft keys
2021/08/20 14:25:16.931149 [info] [raftstore]: shard 2 apply snapshot state completed, apply state {AppliedIndex:11 TruncatedState:{Index:11 Term:6 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:16.931156 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 start applying snapshot
2021/08/20 14:25:16.931160 [info] [raftstore]: shard 2 begin apply snapshot data
2021/08/20 14:25:16.931188 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 AdvanceApply with 13
2021/08/20 14:25:16.931191 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:16.931220 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 AdvanceApply with 13
2021/08/20 14:25:16.931235 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 send 1 msgs
2021/08/20 14:25:16.931276 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 add registration job by handleRaftReadyApply
2021/08/20 14:25:16.931293 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 applying snapshot to set read 11
2021/08/20 14:25:16.931316 [info] [raftstore]: shard 2 snapshot is applied, shard=<{ID:2 Start:[] End:[] Epoch:{ConfVer:5 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} State:Running Peers:[{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:8 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] DisableSplit:false Group:0 Unique: RuleGroups:[] Data:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:16.931330 [info] [raftstore]: shard 2 shard changed after apply snapshot, from=<{ID:2 Start:[] End:[] Epoch:{ConfVer:4 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} State:Running Peers:[{ID:8 ContainerID:7 Role:Learner XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] DisableSplit:false Group:0 Unique: RuleGroups:[] Data:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}> to=<{ID:2 Start:[] End:[] Epoch:{ConfVer:5 Version:1 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} State:Running Peers:[{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:8 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] DisableSplit:false Group:0 Unique: RuleGroups:[] Data:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:16.931349 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 AdvanceApply with 11
2021/08/20 14:25:16.931712 [info] [raftstore]: shard 2 delete exists snap gz file /tmp/cube/node-2/snapshots/2_6_11.gz, header is shard:<id:2 epoch:<confVer:4 version:1 > peers:<id:8 containerID:7 role:Learner > peers:<id:3 containerID:1 > > from:<> to:<> term:6 index:11
2021/08/20 14:25:16.931749 [info] [raftstore]: shard 2 apply snapshot data complete, {Term:6 Vote:0 Commit:11 XXX_unrecognized:[]}
2021/08/20 14:25:16.931753 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 add registration job by handleRaftReadyApply, completed
2021/08/20 14:25:16.973603 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:16.973614 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 send 3 msgs
2021/08/20 14:25:16.973675 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 AdvanceApply with 12
2021/08/20 14:25:16.973755 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 send 3 msgs
2021/08/20 14:25:16.973816 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 AdvanceApply with 13
2021/08/20 14:25:17.030677 [info] [raftstore]: shard 2 start split check job from [122 0 0 0 0 0 0 0 0] to [122 0 0 0 0 0 0 0 1]
2021/08/20 14:25:17.030713 [info] [raftstore]: shard 2 try to split, size 40 bytes splitKeys [[122 0 0 0 0 0 0 0 0 107 101 121 50]]
2021/08/20 14:25:17.030778 [info] prophet: alloc ids [10 11 12] for resource 9 split
2021/08/20 14:25:17.040442 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 send 1 msgs
2021/08/20 14:25:17.040453 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
2021/08/20 14:25:17.040556 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 3 AdvanceApply with 14
2021/08/20 14:25:17.040592 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 send 1 msgs
2021/08/20 14:25:17.040605 [info] [raftstore]: shard 2 update to {ID:2 Start:[] End:[107 101 121 50] Epoch:{ConfVer:5 Version:2 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} State:Running Peers:[{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:8 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] DisableSplit:false Group:0 Unique: RuleGroups:[] Data:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} by post applt split
2021/08/20 14:25:17.040629 [info] [raftstore]: shard 9 init with raft local state {HardState:{Term:5 Vote:0 Commit:5 XXX_unrecognized:[]} LastIndex:5 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:17.040636 [info] [raftstore]: shard 9 init raft apply state, state=<{AppliedIndex:5 TruncatedState:{Index:5 Term:5 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:17.040638 [info] [raftstore]: shard 9 init last term, last term=<5>
2021/08/20 14:25:17.040641 [info] [raftstore]: shard 9 change init applied index from 5 to 5
2021/08/20 14:25:17.040651 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 8 AdvanceApply with 14
{"level":"info","ts":"2021-08-20T14:25:17.040+0800","caller":"raft/raft.go:1550","msg":"a switched to configuration voters=(10 11 12)"}
2021/08/20 14:25:17.040566 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 send 1 msgs
{"level":"info","ts":"2021-08-20T14:25:17.040+0800","caller":"raft/raft.go:711","msg":"a became follower at term 5"}
2021/08/20 14:25:17.040690 [info] [raftstore]: shard 2 update to {ID:2 Start:[] End:[107 101 121 50] Epoch:{ConfVer:5 Version:2 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} State:Running Peers:[{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:8 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] DisableSplit:false Group:0 Unique: RuleGroups:[] Data:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} by post applt split
{"level":"info","ts":"2021-08-20T14:25:17.040+0800","caller":"raft/raft.go:388","msg":"newRaft a [peers: [a,b,c], term: 5, commit: 5, applied: 5, lastindex: 5, lastterm: 5]"}
2021/08/20 14:25:17.040704 [info] [raftstore]: shard 9 init with raft local state {HardState:{Term:5 Vote:0 Commit:5 XXX_unrecognized:[]} LastIndex:5 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:17.040706 [error] [raftstore]: >>>>>>>>>>>> shard 9 peer 10 add registration job by doApplySplit
2021/08/20 14:25:17.040710 [info] [raftstore]: shard 9 init raft apply state, state=<{AppliedIndex:5 TruncatedState:{Index:5 Term:5 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:17.040718 [info] [raftstore]: shard 9 peer 10 raft a added, epoch {ConfVer:5 Version:2 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:10 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:11 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:12 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}], raft worker 0, apply worker apply-0-0
2021/08/20 14:25:17.040733 [error] [raftstore]: >>>>>>>>>>>> shard 2 peer 6 AdvanceApply with 14
2021/08/20 14:25:17.040744 [info] [raftstore]: shard 2 new shard added, new shards [{ID:9 Start:[107 101 121 50] End:[] Epoch:{ConfVer:5 Version:2 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} State:Running Peers:[{ID:10 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:11 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:12 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] DisableSplit:false Group:0 Unique: RuleGroups:[] Data:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
2021/08/20 14:25:17.040746 [info] [raftstore]: shard 9 init last term, last term=<5>
2021/08/20 14:25:17.040795 [info] [raftstore]: shard 9 change init applied index from 5 to 5
{"level":"info","ts":"2021-08-20T14:25:17.040+0800","caller":"raft/raft.go:934","msg":"a is starting a new election at term 5"}
{"level":"info","ts":"2021-08-20T14:25:17.040+0800","caller":"raft/raft.go:724","msg":"a became candidate at term 6"}
{"level":"info","ts":"2021-08-20T14:25:17.040+0800","caller":"raft/raft.go:835","msg":"a received MsgVoteResp from a at term 6"}
{"level":"info","ts":"2021-08-20T14:25:17.040+0800","caller":"raft/raft.go:822","msg":"a [logterm: 5, index: 5] sent MsgVote request to b at term 6"}
{"level":"info","ts":"2021-08-20T14:25:17.040+0800","caller":"raft/raft.go:1550","msg":"c switched to configuration voters=(10 11 12)"}
{"level":"info","ts":"2021-08-20T14:25:17.040+0800","caller":"raft/raft.go:822","msg":"a [logterm: 5, index: 5] sent MsgVote request to c at term 6"}
{"level":"info","ts":"2021-08-20T14:25:17.040+0800","caller":"raft/raft.go:711","msg":"c became follower at term 5"}
{"level":"info","ts":"2021-08-20T14:25:17.040+0800","caller":"raft/raft.go:388","msg":"newRaft c [peers: [a,b,c], term: 5, commit: 5, applied: 5, lastindex: 5, lastterm: 5]"}
2021/08/20 14:25:17.040819 [info] prophet: resource 2 version changed from 1 to 2
2021/08/20 14:25:17.040854 [error] [raftstore]: >>>>>>>>>>>> shard 9 peer 10 send 2 msgs
2021/08/20 14:25:17.040749 [info] [raftstore]: shard 2 update to {ID:2 Start:[] End:[107 101 121 50] Epoch:{ConfVer:5 Version:2 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} State:Running Peers:[{ID:3 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:6 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:8 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] DisableSplit:false Group:0 Unique: RuleGroups:[] Data:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} by post applt split
2021/08/20 14:25:17.041088 [info] [raftstore]: shard 9 init with raft local state {HardState:{Term:5 Vote:0 Commit:5 XXX_unrecognized:[]} LastIndex:5 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/08/20 14:25:17.041095 [info] [raftstore]: shard 9 init raft apply state, state=<{AppliedIndex:5 TruncatedState:{Index:5 Term:5 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}>
2021/08/20 14:25:17.041097 [info] [raftstore]: shard 9 init last term, last term=<5>
2021/08/20 14:25:17.041102 [info] [raftstore]: shard 9 change init applied index from 5 to 5
{"level":"info","ts":"2021-08-20T14:25:17.041+0800","caller":"raft/raft.go:1550","msg":"b switched to configuration voters=(10 11 12)"}
{"level":"info","ts":"2021-08-20T14:25:17.041+0800","caller":"raft/raft.go:711","msg":"b became follower at term 5"}
{"level":"info","ts":"2021-08-20T14:25:17.041+0800","caller":"raft/raft.go:388","msg":"newRaft b [peers: [a,b,c], term: 5, commit: 5, applied: 5, lastindex: 5, lastterm: 5]"}
2021/08/20 14:25:17.041172 [error] [raftstore]: >>>>>>>>>>>> shard 9 peer 11 add registration job by doApplySplit
2021/08/20 14:25:17.040904 [error] [raftstore]: >>>>>>>>>>>> shard 9 peer 12 add registration job by doApplySplit
2021/08/20 14:25:17.041192 [error] [raftstore]: >>>>>>>>>>>> shard 9 peer 11 add registration job by doApplySplit, completed
2021/08/20 14:25:17.041211 [error] [raftstore]: >>>>>>>>>>>> shard 9 peer 12 add registration job by doApplySplit, completed
2021/08/20 14:25:17.041220 [info] [raftstore]: shard 9 peer 11 raft b added, epoch {ConfVer:5 Version:2 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:10 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:11 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:12 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}], raft worker 0, apply worker apply-0-0
2021/08/20 14:25:17.041241 [info] [raftstore]: shard 2 new shard added, new shards [{ID:9 Start:[107 101 121 50] End:[] Epoch:{ConfVer:5 Version:2 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} State:Running Peers:[{ID:10 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:11 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:12 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] DisableSplit:false Group:0 Unique: RuleGroups:[] Data:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
{"level":"info","ts":"2021-08-20T14:25:17.041+0800","caller":"raft/raft.go:870","msg":"b [term: 5] received a MsgVote message with higher term from a [term: 6]"}
{"level":"info","ts":"2021-08-20T14:25:17.041+0800","caller":"raft/raft.go:711","msg":"b became follower at term 6"}
{"level":"info","ts":"2021-08-20T14:25:17.041+0800","caller":"raft/raft.go:971","msg":"b [logterm: 5, index: 5, vote: 0] cast MsgVote for a [logterm: 5, index: 5] at term 6"}
2021/08/20 14:25:17.041266 [info] [raftstore]: shard 9 peer 12 raft c added, epoch {ConfVer:5 Version:2 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}, peers [{ID:10 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:11 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:12 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}], raft worker 0, apply worker apply-0-0
2021/08/20 14:25:17.041313 [info] [raftstore]: shard 2 new shard added, new shards [{ID:9 Start:[107 101 121 50] End:[] Epoch:{ConfVer:5 Version:2 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} State:Running Peers:[{ID:10 ContainerID:1 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:11 ContainerID:5 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {ID:12 ContainerID:7 Role:Voter XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] DisableSplit:false Group:0 Unique: RuleGroups:[] Data:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
{"level":"info","ts":"2021-08-20T14:25:17.041+0800","caller":"raft/raft.go:870","msg":"c [term: 5] received a MsgVote message with higher term from a [term: 6]"}
{"level":"info","ts":"2021-08-20T14:25:17.041+0800","caller":"raft/raft.go:711","msg":"c became follower at term 6"}
{"level":"info","ts":"2021-08-20T14:25:17.041+0800","caller":"raft/raft.go:971","msg":"c [logterm: 5, index: 5, vote: 0] cast MsgVote for a [logterm: 5, index: 5] at term 6"}
2021/08/20 14:25:17.041346 [error] [raftstore]: >>>>>>>>>>>> shard 9 peer 12 send 1 msgs
2021/08/20 14:25:17.040711 [error] [raftstore]: >>>>>>>>>>>> shard 9 peer 10 add registration job by doApplySplit, completed
2021/08/20 14:25:17.041290 [error] [raftstore]: >>>>>>>>>>>> shard 9 peer 11 send 1 msgs
{"level":"info","ts":"2021-08-20T14:25:17.041+0800","caller":"raft/raft.go:835","msg":"a received MsgVoteResp from c at term 6"}
{"level":"info","ts":"2021-08-20T14:25:17.041+0800","caller":"raft/raft.go:1319","msg":"a has received 2 MsgVoteResp votes and 0 vote rejections"}
{"level":"info","ts":"2021-08-20T14:25:17.041+0800","caller":"raft/raft.go:776","msg":"a became leader at term 6"}
2021/08/20 14:25:17.041465 [info] [raftstore]: shard 9 ********become leader now********
2021/08/20 14:25:17.041472 [error] [raftstore]: >>>>>>>>>>>> shard 9 peer 10 send 2 msgs
2021/08/20 14:25:17.041536 [error] [raftstore]: >>>>>>>>>>>> shard 9 peer 11 send 1 msgs