Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Race and panic in ProcessRaftMessage #148

Closed
mrjana opened this issue Mar 15, 2016 · 2 comments
Closed

Race and panic in ProcessRaftMessage #148

mrjana opened this issue Mar 15, 2016 · 2 comments

Comments

@mrjana
Copy link
Contributor

mrjana commented Mar 15, 2016 

==================
WARNING: DATA RACE
Read by goroutine 1146:
  github.com/docker/swarm-v2/manager/state.(*Node).ProcessRaftMessage()
      /vagrant/gowork/src/github.com/docker/swarm-v2/manager/state/raft.go:415 +0x55
  github.com/docker/swarm-v2/api._Manager_ProcessRaftMessage_Handler()
      /vagrant/gowork/src/github.com/docker/swarm-v2/api/manager.pb.go:314 +0x150
  github.com/docker/swarm-v2/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
      /vagrant/gowork/src/github.com/docker/swarm-v2/vendor/google.golang.org/grpc/server.go:505 +0x105b
  github.com/docker/swarm-v2/vendor/google.golang.org/grpc.(*Server).handleStream()
      /vagrant/gowork/src/github.com/docker/swarm-v2/vendor/google.golang.org/grpc/server.go:654 +0x1418
  github.com/docker/swarm-v2/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /vagrant/gowork/src/github.com/docker/swarm-v2/vendor/google.golang.org/grpc/server.go:323 +0xad

Previous write by goroutine 64:
  github.com/docker/swarm-v2/manager/state.(*Node).Start.func1()
      /vagrant/gowork/src/github.com/docker/swarm-v2/manager/state/raft.go:324 +0x8ca

Goroutine 1146 (running) created at:
  github.com/docker/swarm-v2/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /vagrant/gowork/src/github.com/docker/swarm-v2/vendor/google.golang.org/grpc/server.go:324 +0xa7
  github.com/docker/swarm-v2/vendor/google.golang.org/grpc/transport.(*http2Server).operateHeaders()
      /vagrant/gowork/src/github.com/docker/swarm-v2/vendor/google.golang.org/grpc/transport/http2_server.go:212 +0x17fa
  github.com/docker/swarm-v2/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams()
      /vagrant/gowork/src/github.com/docker/swarm-v2/vendor/google.golang.org/grpc/transport/http2_server.go:276 +0x1166
  github.com/docker/swarm-v2/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /vagrant/gowork/src/github.com/docker/swarm-v2/vendor/google.golang.org/grpc/server.go:325 +0x1dc
  github.com/docker/swarm-v2/vendor/google.golang.org/grpc.(*Server).serveNewHTTP2Transport()
      /vagrant/gowork/src/github.com/docker/swarm-v2/vendor/google.golang.org/grpc/server.go:312 +0x54d
  github.com/docker/swarm-v2/vendor/google.golang.org/grpc.(*Server).handleRawConn()
      /vagrant/gowork/src/github.com/docker/swarm-v2/vendor/google.golang.org/grpc/server.go:289 +0x5cb

Goroutine 64 (finished) created at:
  github.com/docker/swarm-v2/manager/state.(*Node).Start()
      /vagrant/gowork/src/github.com/docker/swarm-v2/manager/state/raft.go:329 +0x1bc
  github.com/docker/swarm-v2/manager/state.newInitNode()
      /vagrant/gowork/src/github.com/docker/swarm-v2/manager/state/raft_test.go:53 +0xb5f
  github.com/docker/swarm-v2/manager/state.newRaftCluster()
      /vagrant/gowork/src/github.com/docker/swarm-v2/manager/state/raft_test.go:123 +0x82
  github.com/docker/swarm-v2/manager/state.TestRaftBootstrap()
      /vagrant/gowork/src/github.com/docker/swarm-v2/manager/state/raft_test.go:164 +0x51
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:473 +0xdc
==================
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x68 pc=0x4ac505]

goroutine 1303 [running]:
panic(0xda76a0, 0xc82000e160)
    /usr/local/go/src/runtime/panic.go:464 +0x3ff
github.com/docker/swarm-v2/manager/state.(*Node).ProcessRaftMessage(0xc820138c40, 0x7f7383684370, 0xc823014630, 0xc820128f20, 0x0, 0x0, 0x0)
    /vagrant/gowork/src/github.com/docker/swarm-v2/manager/state/raft.go:415 +0x135
github.com/docker/swarm-v2/api._Manager_ProcessRaftMessage_Handler(0xe82980, 0xc820138c40, 0x7f7383684370, 0xc823014630, 0xc822d31c00, 0x0, 0x0, 0x0, 0x0)
    /vagrant/gowork/src/github.com/docker/swarm-v2/api/manager.pb.go:314 +0x151
github.com/docker/swarm-v2/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc82012d300, 0x7f73836f18f8, 0xc8224a47e0, 0xc822ec4d20, 0xc82037de00, 0x12e9490, 0xc823014600, 0x0, 0x0)
    /vagrant/gowork/src/github.com/docker/swarm-v2/vendor/google.golang.org/grpc/server.go:505 +0x105c
github.com/docker/swarm-v2/vendor/google.golang.org/grpc.(*Server).handleStream(0xc82012d300, 0x7f73836f18f8, 0xc8224a47e0, 0xc822ec4d20, 0xc823014600)
    /vagrant/gowork/src/github.com/docker/swarm-v2/vendor/google.golang.org/grpc/server.go:654 +0x1419
github.com/docker/swarm-v2/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc8224e6690, 0xc82012d300, 0x7f73836f18f8, 0xc8224a47e0, 0xc822ec4d20)
    /vagrant/gowork/src/github.com/docker/swarm-v2/vendor/google.golang.org/grpc/server.go:323 +0xae
created by github.com/docker/swarm-v2/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
    /vagrant/gowork/src/github.com/docker/swarm-v2/vendor/google.golang.org/grpc/server.go:324 +0xa8
FAIL    github.com/docker/swarm-v2/manager/state    3.277s
@mrjana
Copy link
Contributor Author

mrjana commented Mar 15, 2016

Another panic in RaftLeaderLeave

--- FAIL: TestRaftLeaderLeave (9.04s)
        Error Trace:    raft_test.go:427
    Error:      Received unexpected error "raft: failed to process the request: node lost leader status"
    Messages:   failed to propose value

        Error Trace:    raft_test.go:221
            memory.go:169
            raft_test.go:224
            raft_test.go:430
    Error:      "[]" should have 1 item(s), but has 0

panic: runtime error: index out of range [recovered]
    panic: runtime error: index out of range

goroutine 41 [running]:
panic(0xda76a0, 0xc82000e110)
    /usr/local/go/src/runtime/panic.go:464 +0x3ff
testing.tRunner.func1(0xc8201fa750)
    /usr/local/go/src/testing/testing.go:467 +0x22a
panic(0xda76a0, 0xc82000e110)
    /usr/local/go/src/runtime/panic.go:426 +0x521
github.com/docker/swarm-v2/manager/state.checkValue.func1(0x7f70f43f39c8, 0xc823a1a200, 0x0, 0x0)
    /vagrant/gowork/src/github.com/docker/swarm-v2/manager/state/raft_test.go:222 +0x30d
github.com/docker/swarm-v2/manager/state.(*MemoryStore).View(0xc824074e10, 0xc820ee7db0, 0x0, 0x0)
    /vagrant/gowork/src/github.com/docker/swarm-v2/manager/state/memory.go:169 +0x14d
github.com/docker/swarm-v2/manager/state.checkValue(0xc8201fa750, 0xc823d21b20, 0x0)
    /vagrant/gowork/src/github.com/docker/swarm-v2/manager/state/raft_test.go:224 +0xcb
github.com/docker/swarm-v2/manager/state.TestRaftLeaderLeave(0xc8201fa750)
    /vagrant/gowork/src/github.com/docker/swarm-v2/manager/state/raft_test.go:430 +0xe0e
testing.tRunner(0xc8201fa750, 0x12ec4c8)
    /usr/local/go/src/testing/testing.go:473 +0xdd
created by testing.RunTests
    /usr/local/go/src/testing/testing.go:582 +0xae3
FAIL    github.com/docker/swarm-v2/manager/state    14.319s

@aaronlehmann aaronlehmann mentioned this issue Mar 15, 2016
Merged
@abronan abronan closed this as completed Mar 16, 2016
@abronan abronan reopened this Mar 16, 2016
@abronan
Copy link
Contributor

abronan commented Mar 16, 2016

Apparently not over (seems to only happen in the CI though can't reproduce locally):

--- FAIL: TestRaftLogReplication (6.52s)
    Error Trace:    raft_test.go:261
            memory.go:178
            raft_test.go:256
            raft_test.go:359
    Error:      "[]" should have 1 item(s), but has 0


panic: runtime error: index out of range [recovered]
    panic: runtime error: index out of range

goroutine 31 [running]:
panic(0xb701a0, 0xc82000e140)
    /usr/local/go/src/runtime/panic.go:464 +0x3e6
testing.tRunner.func1(0xc8200d4900)
    /usr/local/go/src/testing/testing.go:467 +0x192
panic(0xb701a0, 0xc82000e140)
    /usr/local/go/src/runtime/panic.go:426 +0x4e9
github.com/docker/swarm-v2/manager/state.checkValue.func1(0x2b3de8236cb0, 0xc820cf6640, 0x0, 0x0)
    /home/ubuntu/.go_workspace/src/github.com/docker/swarm-v2/manager/state/raft_test.go:262 +0x2c1
github.com/docker/swarm-v2/manager/state.(*MemoryStore).View(0xc820832ab0, 0xc82147de68, 0x0, 0x0)
    github.com/docker/swarm-v2/manager/state/_test/_obj_test/memory.go:178 +0x134
github.com/docker/swarm-v2/manager/state.checkValue(0xc8200d4900, 0xc8209a0d20, 0xc821317500)
    /home/ubuntu/.go_workspace/src/github.com/docker/swarm-v2/manager/state/raft_test.go:256 +0x61
github.com/docker/swarm-v2/manager/state.TestRaftLogReplication(0xc8200d4900)
    /home/ubuntu/.go_workspace/src/github.com/docker/swarm-v2/manager/state/raft_test.go:359 +0x313
testing.tRunner(0xc8200d4900, 0x1040108)
    /usr/local/go/src/testing/testing.go:473 +0x98
created by testing.RunTests
    /usr/local/go/src/testing/testing.go:582 +0x892
FAIL    github.com/docker/swarm-v2/manager/state    9.132s

@aaronlehmann aaronlehmann mentioned this issue Apr 6, 2016
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@abronan @mrjana