Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TestWatchReconnInit: panic: runtime error: index out of range #6364

Closed
heyitsanthony opened this issue Sep 6, 2016 · 0 comments
Closed

TestWatchReconnInit: panic: runtime error: index out of range #6364

heyitsanthony opened this issue Sep 6, 2016 · 0 comments
Labels
area/testing
Milestone
v3.1.0

Comments

@heyitsanthony
Copy link
Contributor

via semaphore (https://semaphoreci.com/coreos/etcd/branches/pull-request-6358/builds/1)

=== RUN   TestWatchReconnInit
2016-09-06 12:52:02.254965 I | integration: launching node1210077675517516077 (unix://localhost:node1210077675517516077.sock.bridge)
2016-09-06 12:52:02.256852 I | integration: launching node8598017571830101651 (unix://localhost:node8598017571830101651.sock.bridge)
2016-09-06 12:52:02.257289 I | integration: launching node7197444051238512015 (unix://localhost:node7197444051238512015.sock.bridge)
2016-09-06 12:52:02.276407 I | etcdserver: name = node1210077675517516077
2016-09-06 12:52:02.276469 I | etcdserver: data dir = /tmp/etcd602410689
2016-09-06 12:52:02.276515 I | etcdserver: member dir = /tmp/etcd602410689/member
2016-09-06 12:52:02.276549 I | etcdserver: heartbeat = 10ms
2016-09-06 12:52:02.276582 I | etcdserver: election = 100ms
2016-09-06 12:52:02.276614 I | etcdserver: snapshot count = 0
2016-09-06 12:52:02.276658 I | etcdserver: advertise client URLs = unix://127.0.0.1:21442.12467.sock
2016-09-06 12:52:02.276713 I | etcdserver: initial advertise peer URLs = unix://127.0.0.1:21441.12467.sock
2016-09-06 12:52:02.276859 I | etcdserver: initial cluster = node1210077675517516077=unix://127.0.0.1:21441.12467.sock,node7197444051238512015=unix://127.0.0.1:21445.12467.sock,node8598017571830101651=unix://127.0.0.1:21443.12467.sock
2016-09-06 12:52:02.287867 I | etcdserver: starting member 418af4c6b612647f in cluster b63c407bd9a290a8
2016-09-06 12:52:02.287961 I | raft: 418af4c6b612647f became follower at term 0
2016-09-06 12:52:02.288024 I | raft: newRaft 418af4c6b612647f [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2016-09-06 12:52:02.288070 I | raft: 418af4c6b612647f became follower at term 1
2016-09-06 12:52:02.290686 I | rafthttp: starting peer 230887a8cd6de247...
2016-09-06 12:52:02.291056 I | rafthttp: started HTTP pipelining with peer 230887a8cd6de247
2016-09-06 12:52:02.295308 I | rafthttp: started peer 230887a8cd6de247
2016-09-06 12:52:02.295566 I | rafthttp: added peer 230887a8cd6de247
2016-09-06 12:52:02.295650 I | rafthttp: starting peer 923551603403627a...
2016-09-06 12:52:02.296022 I | rafthttp: started HTTP pipelining with peer 923551603403627a
2016-09-06 12:52:02.298941 I | etcdserver: name = node8598017571830101651
2016-09-06 12:52:02.299032 I | etcdserver: data dir = /tmp/etcd665784620
2016-09-06 12:52:02.299086 I | etcdserver: member dir = /tmp/etcd665784620/member
2016-09-06 12:52:02.299123 I | etcdserver: heartbeat = 10ms
2016-09-06 12:52:02.299154 I | etcdserver: election = 100ms
2016-09-06 12:52:02.299185 I | etcdserver: snapshot count = 0
2016-09-06 12:52:02.299233 I | etcdserver: advertise client URLs = unix://127.0.0.1:21444.12467.sock
2016-09-06 12:52:02.299279 I | etcdserver: initial advertise peer URLs = unix://127.0.0.1:21443.12467.sock
2016-09-06 12:52:02.299453 I | etcdserver: initial cluster = node1210077675517516077=unix://127.0.0.1:21441.12467.sock,node7197444051238512015=unix://127.0.0.1:21445.12467.sock,node8598017571830101651=unix://127.0.0.1:21443.12467.sock
2016-09-06 12:52:02.300314 I | rafthttp: started streaming with peer 230887a8cd6de247 (writer)
2016-09-06 12:52:02.300420 I | rafthttp: started streaming with peer 230887a8cd6de247 (writer)
2016-09-06 12:52:02.300555 I | rafthttp: started streaming with peer 230887a8cd6de247 (stream MsgApp v2 reader)
2016-09-06 12:52:02.300902 I | rafthttp: started streaming with peer 230887a8cd6de247 (stream Message reader)
2016-09-06 12:52:02.301587 I | rafthttp: started streaming with peer 923551603403627a (writer)
2016-09-06 12:52:02.301735 I | rafthttp: started streaming with peer 923551603403627a (writer)
2016-09-06 12:52:02.301844 I | rafthttp: started streaming with peer 923551603403627a (stream MsgApp v2 reader)
2016-09-06 12:52:02.303272 I | rafthttp: started peer 923551603403627a
2016-09-06 12:52:02.303507 I | rafthttp: added peer 923551603403627a
2016-09-06 12:52:02.303562 I | etcdserver: set snapshot count to default 10000
2016-09-06 12:52:02.303610 I | etcdserver: starting server... [version: 3.0.0+git, cluster version: to_be_decided]
2016-09-06 12:52:02.304868 I | rafthttp: started streaming with peer 923551603403627a (stream Message reader)
2016-09-06 12:52:02.310569 I | etcdserver: starting member 230887a8cd6de247 in cluster b63c407bd9a290a8
2016-09-06 12:52:02.310772 I | raft: 230887a8cd6de247 became follower at term 0
2016-09-06 12:52:02.310896 I | raft: newRaft 230887a8cd6de247 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2016-09-06 12:52:02.310992 I | raft: 230887a8cd6de247 became follower at term 1
2016-09-06 12:52:02.312717 I | membership: added member 230887a8cd6de247 [unix://127.0.0.1:21443.12467.sock] to cluster b63c407bd9a290a8
2016-09-06 12:52:02.313625 I | rafthttp: starting peer 418af4c6b612647f...
2016-09-06 12:52:02.314002 I | membership: added member 418af4c6b612647f [unix://127.0.0.1:21441.12467.sock] to cluster b63c407bd9a290a8
2016-09-06 12:52:02.314140 I | rafthttp: started HTTP pipelining with peer 418af4c6b612647f
2016-09-06 12:52:02.316061 I | membership: added member 923551603403627a [unix://127.0.0.1:21445.12467.sock] to cluster b63c407bd9a290a8
2016-09-06 12:52:02.316495 I | etcdserver: name = node7197444051238512015
2016-09-06 12:52:02.316604 I | etcdserver: data dir = /tmp/etcd466525339
2016-09-06 12:52:02.316677 I | etcdserver: member dir = /tmp/etcd466525339/member
2016-09-06 12:52:02.316780 I | etcdserver: heartbeat = 10ms
2016-09-06 12:52:02.316859 I | etcdserver: election = 100ms
2016-09-06 12:52:02.316945 I | etcdserver: snapshot count = 0
2016-09-06 12:52:02.317051 I | etcdserver: advertise client URLs = unix://127.0.0.1:21446.12467.sock
2016-09-06 12:52:02.317140 I | etcdserver: initial advertise peer URLs = unix://127.0.0.1:21445.12467.sock
2016-09-06 12:52:02.317351 I | etcdserver: initial cluster = node1210077675517516077=unix://127.0.0.1:21441.12467.sock,node7197444051238512015=unix://127.0.0.1:21445.12467.sock,node8598017571830101651=unix://127.0.0.1:21443.12467.sock
2016-09-06 12:52:02.318657 I | rafthttp: started peer 418af4c6b612647f
2016-09-06 12:52:02.318876 I | rafthttp: added peer 418af4c6b612647f
2016-09-06 12:52:02.318948 I | rafthttp: starting peer 923551603403627a...
2016-09-06 12:52:02.319281 I | rafthttp: started HTTP pipelining with peer 923551603403627a
2016-09-06 12:52:02.320429 I | integration: launched node1210077675517516077 (unix://localhost:node1210077675517516077.sock.bridge)
2016-09-06 12:52:02.320712 I | rafthttp: started streaming with peer 418af4c6b612647f (writer)
2016-09-06 12:52:02.325213 I | rafthttp: started streaming with peer 418af4c6b612647f (stream MsgApp v2 reader)
2016-09-06 12:52:02.325631 I | rafthttp: started streaming with peer 418af4c6b612647f (writer)
2016-09-06 12:52:02.325741 I | rafthttp: started streaming with peer 418af4c6b612647f (stream Message reader)
2016-09-06 12:52:02.326829 I | raft: 418af4c6b612647f is starting a new election at term 1
2016-09-06 12:52:02.326920 I | raft: 418af4c6b612647f became candidate at term 2
2016-09-06 12:52:02.326967 I | raft: 418af4c6b612647f received vote from 418af4c6b612647f at term 2
2016-09-06 12:52:02.327023 I | raft: 418af4c6b612647f [logterm: 1, index: 3] sent vote request to 923551603403627a at term 2
2016-09-06 12:52:02.327082 I | raft: 418af4c6b612647f [logterm: 1, index: 3] sent vote request to 230887a8cd6de247 at term 2
2016-09-06 12:52:02.338262 I | rafthttp: started peer 923551603403627a
2016-09-06 12:52:02.338528 I | rafthttp: added peer 923551603403627a
2016-09-06 12:52:02.338591 I | etcdserver: set snapshot count to default 10000
2016-09-06 12:52:02.338640 I | etcdserver: starting server... [version: 3.0.0+git, cluster version: to_be_decided]
2016-09-06 12:52:02.350513 I | rafthttp: started streaming with peer 923551603403627a (writer)
2016-09-06 12:52:02.350738 I | rafthttp: started streaming with peer 923551603403627a (writer)
2016-09-06 12:52:02.350887 I | rafthttp: started streaming with peer 923551603403627a (stream Message reader)
2016-09-06 12:52:02.351213 I | rafthttp: started streaming with peer 923551603403627a (stream MsgApp v2 reader)
2016-09-06 12:52:02.352906 I | rafthttp: peer 230887a8cd6de247 became active
2016-09-06 12:52:02.352980 I | rafthttp: established a TCP streaming connection with peer 230887a8cd6de247 (stream MsgApp v2 writer)
2016-09-06 12:52:02.353830 I | membership: added member 230887a8cd6de247 [unix://127.0.0.1:21443.12467.sock] to cluster b63c407bd9a290a8
2016-09-06 12:52:02.354960 I | membership: added member 418af4c6b612647f [unix://127.0.0.1:21441.12467.sock] to cluster b63c407bd9a290a8
2016-09-06 12:52:02.355361 I | etcdserver: starting member 923551603403627a in cluster b63c407bd9a290a8
2016-09-06 12:52:02.355482 I | raft: 923551603403627a became follower at term 0
2016-09-06 12:52:02.355572 I | raft: newRaft 923551603403627a [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2016-09-06 12:52:02.355645 I | raft: 923551603403627a became follower at term 1
2016-09-06 12:52:02.357691 I | rafthttp: starting peer 230887a8cd6de247...
2016-09-06 12:52:02.358171 I | rafthttp: started HTTP pipelining with peer 230887a8cd6de247
2016-09-06 12:52:02.365782 I | rafthttp: started streaming with peer 230887a8cd6de247 (writer)
2016-09-06 12:52:02.366435 I | membership: added member 923551603403627a [unix://127.0.0.1:21445.12467.sock] to cluster b63c407bd9a290a8
2016-09-06 12:52:02.366873 I | rafthttp: started streaming with peer 230887a8cd6de247 (writer)
2016-09-06 12:52:02.408399 I | rafthttp: started peer 230887a8cd6de247
2016-09-06 12:52:02.408770 I | rafthttp: added peer 230887a8cd6de247
2016-09-06 12:52:02.408847 I | rafthttp: starting peer 418af4c6b612647f...
2016-09-06 12:52:02.409232 I | rafthttp: started HTTP pipelining with peer 418af4c6b612647f
2016-09-06 12:52:02.413201 I | rafthttp: peer 418af4c6b612647f became active
2016-09-06 12:52:02.413284 I | rafthttp: established a TCP streaming connection with peer 418af4c6b612647f (stream MsgApp v2 reader)
2016-09-06 12:52:02.414802 I | integration: launched node8598017571830101651 (unix://localhost:node8598017571830101651.sock.bridge)
2016-09-06 12:52:02.415641 I | rafthttp: established a TCP streaming connection with peer 230887a8cd6de247 (stream Message writer)
2016-09-06 12:52:02.416629 I | rafthttp: established a TCP streaming connection with peer 418af4c6b612647f (stream Message writer)
2016-09-06 12:52:02.418527 I | rafthttp: established a TCP streaming connection with peer 418af4c6b612647f (stream MsgApp v2 writer)
2016-09-06 12:52:02.418630 I | rafthttp: established a TCP streaming connection with peer 418af4c6b612647f (stream Message reader)
2016-09-06 12:52:02.419384 I | rafthttp: established a TCP streaming connection with peer 230887a8cd6de247 (stream Message reader)
2016-09-06 12:52:02.420661 I | rafthttp: started peer 418af4c6b612647f
2016-09-06 12:52:02.420894 I | rafthttp: added peer 418af4c6b612647f
2016-09-06 12:52:02.420975 I | etcdserver: set snapshot count to default 10000
2016-09-06 12:52:02.421045 I | etcdserver: starting server... [version: 3.0.0+git, cluster version: to_be_decided]
2016-09-06 12:52:02.422487 I | rafthttp: established a TCP streaming connection with peer 230887a8cd6de247 (stream MsgApp v2 reader)
2016-09-06 12:52:02.422801 I | raft: 230887a8cd6de247 is starting a new election at term 1
2016-09-06 12:52:02.422956 I | raft: 230887a8cd6de247 became candidate at term 2
2016-09-06 12:52:02.423027 I | raft: 230887a8cd6de247 received vote from 230887a8cd6de247 at term 2
2016-09-06 12:52:02.423112 I | raft: 230887a8cd6de247 [logterm: 1, index: 3] sent vote request to 418af4c6b612647f at term 2
2016-09-06 12:52:02.423196 I | raft: 230887a8cd6de247 [logterm: 1, index: 3] sent vote request to 923551603403627a at term 2
2016-09-06 12:52:02.423329 I | raft: 230887a8cd6de247 [logterm: 1, index: 3, vote: 230887a8cd6de247] rejected vote from 418af4c6b612647f [logterm: 1, index: 3] at term 2
2016-09-06 12:52:02.423614 I | rafthttp: started streaming with peer 230887a8cd6de247 (stream MsgApp v2 reader)
2016-09-06 12:52:02.423712 I | rafthttp: started streaming with peer 418af4c6b612647f (stream MsgApp v2 reader)
2016-09-06 12:52:02.424026 I | rafthttp: started streaming with peer 418af4c6b612647f (stream Message reader)
2016-09-06 12:52:02.424116 I | rafthttp: started streaming with peer 230887a8cd6de247 (stream Message reader)
2016-09-06 12:52:02.424571 I | rafthttp: started streaming with peer 418af4c6b612647f (writer)
2016-09-06 12:52:02.424693 I | rafthttp: started streaming with peer 418af4c6b612647f (writer)
2016-09-06 12:52:02.427202 I | rafthttp: peer 923551603403627a became active
2016-09-06 12:52:02.427358 I | rafthttp: established a TCP streaming connection with peer 923551603403627a (stream MsgApp v2 writer)
2016-09-06 12:52:02.427826 I | rafthttp: established a TCP streaming connection with peer 923551603403627a (stream Message writer)
2016-09-06 12:52:02.428032 I | rafthttp: peer 230887a8cd6de247 became active
2016-09-06 12:52:02.428105 I | rafthttp: established a TCP streaming connection with peer 230887a8cd6de247 (stream Message reader)
2016-09-06 12:52:02.428803 I | raft: 418af4c6b612647f [logterm: 1, index: 3, vote: 418af4c6b612647f] rejected vote from 230887a8cd6de247 [logterm: 1, index: 3] at term 2
2016-09-06 12:52:02.428927 I | raft: 418af4c6b612647f received vote rejection from 230887a8cd6de247 at term 2
2016-09-06 12:52:02.429011 I | raft: 418af4c6b612647f [quorum:2] has received 1 votes and 1 vote rejections
2016-09-06 12:52:02.429207 I | rafthttp: established a TCP streaming connection with peer 230887a8cd6de247 (stream MsgApp v2 reader)
2016-09-06 12:52:02.429602 I | rafthttp: established a TCP streaming connection with peer 923551603403627a (stream Message reader)
2016-09-06 12:52:02.430219 I | raft: 230887a8cd6de247 received vote rejection from 418af4c6b612647f at term 2
2016-09-06 12:52:02.430324 I | raft: 230887a8cd6de247 [quorum:2] has received 1 votes and 1 vote rejections
2016-09-06 12:52:02.430574 I | rafthttp: peer 923551603403627a became active
2016-09-06 12:52:02.430676 I | rafthttp: established a TCP streaming connection with peer 923551603403627a (stream MsgApp v2 reader)
2016-09-06 12:52:02.431822 I | rafthttp: established a TCP streaming connection with peer 230887a8cd6de247 (stream MsgApp v2 writer)
2016-09-06 12:52:02.432115 I | rafthttp: established a TCP streaming connection with peer 923551603403627a (stream MsgApp v2 reader)
2016-09-06 12:52:02.432522 I | raft: 923551603403627a [term: 1] received a MsgVote message with higher term from 418af4c6b612647f [term: 2]
2016-09-06 12:52:02.432643 I | raft: 923551603403627a became follower at term 2
2016-09-06 12:52:02.432736 I | raft: 923551603403627a [logterm: 1, index: 3, vote: 0] voted for 418af4c6b612647f [logterm: 1, index: 3] at term 2
2016-09-06 12:52:02.432852 I | rafthttp: established a TCP streaming connection with peer 230887a8cd6de247 (stream Message writer)
2016-09-06 12:52:02.433158 I | rafthttp: established a TCP streaming connection with peer 923551603403627a (stream Message reader)
2016-09-06 12:52:02.435369 I | rafthttp: established a TCP streaming connection with peer 923551603403627a (stream MsgApp v2 writer)
2016-09-06 12:52:02.435444 I | rafthttp: peer 418af4c6b612647f became active
2016-09-06 12:52:02.435564 I | rafthttp: established a TCP streaming connection with peer 418af4c6b612647f (stream MsgApp v2 writer)
2016-09-06 12:52:02.435643 I | rafthttp: established a TCP streaming connection with peer 418af4c6b612647f (stream Message writer)
2016-09-06 12:52:02.436160 I | rafthttp: established a TCP streaming connection with peer 923551603403627a (stream Message writer)
2016-09-06 12:52:02.436352 I | rafthttp: established a TCP streaming connection with peer 418af4c6b612647f (stream MsgApp v2 reader)
2016-09-06 12:52:02.436439 I | membership: added member 230887a8cd6de247 [unix://127.0.0.1:21443.12467.sock] to cluster b63c407bd9a290a8
2016-09-06 12:52:02.437262 I | membership: added member 418af4c6b612647f [unix://127.0.0.1:21441.12467.sock] to cluster b63c407bd9a290a8
2016-09-06 12:52:02.437859 I | rafthttp: established a TCP streaming connection with peer 418af4c6b612647f (stream Message reader)
2016-09-06 12:52:02.438057 I | raft: 418af4c6b612647f received vote from 923551603403627a at term 2
2016-09-06 12:52:02.438112 I | raft: 418af4c6b612647f [quorum:2] has received 2 votes and 1 vote rejections
2016-09-06 12:52:02.438191 I | raft: 418af4c6b612647f became leader at term 2
2016-09-06 12:52:02.438251 I | raft: raft.node: 418af4c6b612647f elected leader 418af4c6b612647f at term 2
2016-09-06 12:52:02.438476 I | integration: launched node7197444051238512015 (unix://localhost:node7197444051238512015.sock.bridge)
2016-09-06 12:52:02.438949 I | raft: 923551603403627a [logterm: 1, index: 3, vote: 418af4c6b612647f] rejected vote from 230887a8cd6de247 [logterm: 1, index: 3] at term 2
2016-09-06 12:52:02.439078 I | raft: 230887a8cd6de247 became follower at term 2
2016-09-06 12:52:02.439148 I | raft: raft.node: 230887a8cd6de247 elected leader 418af4c6b612647f at term 2
2016-09-06 12:52:02.439219 I | raft: raft.node: 923551603403627a elected leader 418af4c6b612647f at term 2
2016-09-06 12:52:02.439485 I | membership: added member 923551603403627a [unix://127.0.0.1:21445.12467.sock] to cluster b63c407bd9a290a8
2016-09-06 12:52:02.443240 I | etcdserver: published {Name:node1210077675517516077 ClientURLs:[unix://127.0.0.1:21442.12467.sock]} to cluster b63c407bd9a290a8
2016-09-06 12:52:02.444096 I | etcdserver: setting up the initial cluster version to 3.0
2016-09-06 12:52:02.445773 I | etcdserver: published {Name:node8598017571830101651 ClientURLs:[unix://127.0.0.1:21444.12467.sock]} to cluster b63c407bd9a290a8
2016-09-06 12:52:02.447697 I | etcdserver: published {Name:node7197444051238512015 ClientURLs:[unix://127.0.0.1:21446.12467.sock]} to cluster b63c407bd9a290a8
2016-09-06 12:52:02.447942 N | membership: set the initial cluster version to 3.0
2016-09-06 12:52:02.448428 N | membership: set the initial cluster version to 3.0
2016-09-06 12:52:02.448830 N | membership: set the initial cluster version to 3.0
2016-09-06 12:52:02.468896 I | v3rpc/grpc: transport: http2Client.notifyError got notified that the client transport was broken EOF.
panic: runtime error: index out of range

goroutine 34603 [running]:
panic(0x11fb340, 0xc8200140d0)
    /usr/local/golang/1.6.3/go/src/runtime/panic.go:481 +0x3ff
github.com/coreos/etcd/clientv3.(*watchGrpcStream).serveStream(0xc821b96f00, 0xc82374e3f0)
    /home/runner/workspace/src/github.com/coreos/etcd/gopath/src/github.com/coreos/etcd/clientv3/watch.go:610 +0xffa
created by github.com/coreos/etcd/clientv3.(*watchGrpcStream).addStream
    /home/runner/workspace/src/github.com/coreos/etcd/gopath/src/github.com/coreos/etcd/clientv3/watch.go:378 +0x4e8
exit status 2
FAIL    github.com/coreos/etcd/clientv3/integration 109.222s
@xiang90 xiang90 added this to the v3.1.0 milestone Sep 6, 2016
heyitsanthony pushed a commit to heyitsanthony/etcd that referenced this issue Sep 7, 2016
clientv3: drain buffered WatchResponses before resuming
f5e4aa4 
Otherwise, the watcherStream can receive WatchResponses in the
middle of a resume, corrupting the stream.

Fixes etcd-io#6364
heyitsanthony pushed a commit to heyitsanthony/etcd that referenced this issue Sep 7, 2016
clientv3: drain buffered WatchResponses before resuming
ad318ee 
Otherwise, the watcherStream can receive WatchResponses in the
middle of a resume, corrupting the stream.

Fixes etcd-io#6364
@heyitsanthony heyitsanthony mentioned this issue Sep 7, 2016
Merged
heyitsanthony pushed a commit that referenced this issue Sep 15, 2016
clientv3: drain buffered WatchResponses before resuming
4abc381 
Otherwise, the watcherStream can receive WatchResponses in the
middle of a resume, corrupting the stream.

Fixes #6364
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/testing
Milestone
v3.1.0
Development

No branches or pull requests
2 participants
@xiang90 @heyitsanthony