Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data race when starting server #7419

Closed
rleungx opened this issue Nov 23, 2023 · 0 comments · Fixed by #7420
Closed

Data race when starting server #7419

rleungx opened this issue Nov 23, 2023 · 0 comments · Fixed by #7420
Labels
severity/minor type/bug The issue is confirmed as a bug.

Comments

@rleungx
Copy link
Member

rleungx commented Nov 23, 2023

Bug Report

==================
WARNING: DATA RACE
Read at 0x00c000242620 by goroutine 89542:
  github.com/tikv/pd/pkg/member.(*EmbeddedEtcdMember).IsLeader()
      /home/runner/work/pd/pd/pkg/member/member.go:114 +0x75
  github.com/tikv/pd/pkg/utils/apiutil/serverapi.(*redirector).ServeHTTP()
      /home/runner/work/pd/pd/pkg/utils/apiutil/serverapi/middleware.go:163 +0x1c4
  github.com/urfave/negroni.middleware.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38 +0x134
  github.com/urfave/negroni.middleware.ServeHTTP-fm()
      <autogenerated>:1 +0x71
  github.com/tikv/pd/pkg/utils/apiutil/serverapi.(*runtimeServiceValidator).ServeHTTP()
      /home/runner/work/pd/pd/pkg/utils/apiutil/serverapi/middleware.go:44 +0x171
  github.com/urfave/negroni.middleware.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38 +0x134
  github.com/urfave/negroni.(*Negroni).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:96 +0x1b8
  github.com/gorilla/mux.(*Router).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 +0x361
  github.com/gorilla/mux.(*Router).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 +0x361
  github.com/urfave/negroni.(*Negroni).UseHandler.Wrap.func1()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:46 +0x66
  github.com/urfave/negroni.HandlerFunc.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:29 +0x51
  github.com/urfave/negroni.middleware.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38 +0x134
  github.com/urfave/negroni.middleware.ServeHTTP-fm()
      <autogenerated>:1 +0x71
  github.com/urfave/negroni.(*Recovery).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/recovery.go:193 +0xe1
  github.com/urfave/negroni.middleware.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38 +0x134
  github.com/urfave/negroni.(*Negroni).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:96 +0x1b8
  net/http.(*ServeMux).ServeHTTP()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:2514 +0xbc
  go.etcd.io/etcd/embed.(*accessController).ServeHTTP()
      /home/runner/go/pkg/mod/go.etcd.io/[email protected]/embed/serve.go:381 +0x94d
  net/http.serverHandler.ServeHTTP()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:2938 +0x2a1
  net/http.(*conn).serve()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:2009 +0xc24
  net/http.(*Server).Serve.func3()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:3086 +0x4f

Previous write at 0x00c000242620 by goroutine 83520:
  github.com/tikv/pd/pkg/member.(*EmbeddedEtcdMember).InitMemberInfo()
      /home/runner/work/pd/pd/pkg/member/member.go:346 +0x9e8
  github.com/tikv/pd/server.(*Server).startServer()
      /home/runner/work/pd/pd/server/server.go:434 +0x939
  github.com/tikv/pd/server.(*Server).Run()
      /home/runner/work/pd/pd/server/server.go:586 +0x1c7
  github.com/tikv/pd/tests.(*TestServer).Run()
      /home/runner/work/pd/pd/tests/cluster.go:125 +0x1e4
  github.com/tikv/pd/tests/integrations/mcs/scheduling.(*serverTestSuite).TestAllocIDAfterLeaderChange()
      /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/server_test.go:101 +0x135
  github.com/pingcap/failpoint.parseTerm()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:149 +0x364
  github.com/pingcap/failpoint.parse()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:105 +0x276
  github.com/pingcap/failpoint.Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:225 +0x9c
  github.com/tikv/pd/tests/integrations/mcs/scheduling.(*serverTestSuite).TestAllocIDAfterLeaderChange()
      /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/server_test.go:98 +0x9d
  runtime.call16()
      /opt/hostedtoolcache/go/1.21.4/x64/src/runtime/asm_amd64.s:747 +0x42
  reflect.Value.Call()
      /opt/hostedtoolcache/go/1.21.4/x64/src/reflect/value.go:380 +0xb5
  github.com/stretchr/testify/suite.Run.func1()
      /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:197 +0x684
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1648 +0x44

Goroutine 89542 (running) created at:
  net/http.(*Server).Serve()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:3086 +0x80c
  go.etcd.io/etcd/embed.(*serveCtx).serve.func3()
      /home/runner/go/pkg/mod/go.etcd.io/[email protected]/embed/serve.go:141 +0x4f

Goroutine 83520 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1648 +0x82a
  github.com/stretchr/testify/suite.runTests()
      /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:242 +0x186
  github.com/stretchr/testify/suite.Run()
      /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:215 +0x9d2
  github.com/pingcap/failpoint.parse()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:105 +0x276
  github.com/pingcap/failpoint.Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:225 +0x7a
  github.com/tikv/pd/tests/integrations/mcs/scheduling.(*serverTestSuite).SetupSuite()
      /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/server_test.go:62 +0x7b
  github.com/stretchr/testify/suite.Run()
      /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:154 +0x5fe
  github.com/tikv/pd/tests/integrations/mcs/scheduling.TestServerTestSuite()
      /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/server_test.go:56 +0x3c
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1648 +0x44
==================
==================
WARNING: DATA RACE
Read at 0x00c0031c6110 by goroutine 89542:
  sync/atomic.LoadInt64()
      /opt/hostedtoolcache/go/1.21.4/x64/src/runtime/race_amd64.s:208 +0xb
  sync/atomic.LoadPointer()
      <autogenerated>:1 +0x10
  github.com/tikv/pd/pkg/election.(*Leadership).getLease()
      /home/runner/work/pd/pd/pkg/election/leadership.go:87 +0xd2
  github.com/tikv/pd/pkg/election.(*Leadership).Check()
      /home/runner/work/pd/pd/pkg/election/leadership.go:176 +0x6a
  github.com/tikv/pd/pkg/member.(*EmbeddedEtcdMember).IsLeader()
      /home/runner/work/pd/pd/pkg/member/member.go:114 +0x84
  github.com/tikv/pd/pkg/utils/apiutil/serverapi.(*redirector).ServeHTTP()
      /home/runner/work/pd/pd/pkg/utils/apiutil/serverapi/middleware.go:163 +0x1c4
  github.com/urfave/negroni.middleware.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38 +0x134
  github.com/urfave/negroni.middleware.ServeHTTP-fm()
      <autogenerated>:1 +0x71
  github.com/tikv/pd/pkg/utils/apiutil/serverapi.(*runtimeServiceValidator).ServeHTTP()
      /home/runner/work/pd/pd/pkg/utils/apiutil/serverapi/middleware.go:44 +0x171
  github.com/urfave/negroni.middleware.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38 +0x134
  github.com/urfave/negroni.(*Negroni).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:96 +0x1b8
  github.com/gorilla/mux.(*Router).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 +0x361
  github.com/gorilla/mux.(*Router).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 +0x361
  github.com/urfave/negroni.(*Negroni).UseHandler.Wrap.func1()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:46 +0x66
  github.com/urfave/negroni.HandlerFunc.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:29 +0x51
  github.com/urfave/negroni.middleware.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38 +0x134
  github.com/urfave/negroni.middleware.ServeHTTP-fm()
      <autogenerated>:1 +0x71
  github.com/urfave/negroni.(*Recovery).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/recovery.go:193 +0xe1
  github.com/urfave/negroni.middleware.ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38 +0x134
  github.com/urfave/negroni.(*Negroni).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:96 +0x1b8
  net/http.(*ServeMux).ServeHTTP()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:2514 +0xbc
  go.etcd.io/etcd/embed.(*accessController).ServeHTTP()
      /home/runner/go/pkg/mod/go.etcd.io/[email protected]/embed/serve.go:381 +0x94d
  net/http.serverHandler.ServeHTTP()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:2938 +0x2a1
  net/http.(*conn).serve()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:2009 +0xc24
  net/http.(*Server).Serve.func3()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:3086 +0x4f

Previous write at 0x00c0031c6110 by goroutine 83520:
  github.com/tikv/pd/pkg/election.NewLeadership()
      /home/runner/work/pd/pd/pkg/election/leadership.go:75 +0x86e
  github.com/tikv/pd/pkg/member.(*EmbeddedEtcdMember).InitMemberInfo()
      /home/runner/work/pd/pd/pkg/member/member.go:346 +0x6f5
  github.com/tikv/pd/server.(*Server).startServer()
      /home/runner/work/pd/pd/server/server.go:434 +0x939
  github.com/tikv/pd/server.(*Server).Run()
      /home/runner/work/pd/pd/server/server.go:586 +0x1c7
  github.com/tikv/pd/tests.(*TestServer).Run()
      /home/runner/work/pd/pd/tests/cluster.go:125 +0x1e4
  github.com/tikv/pd/tests/integrations/mcs/scheduling.(*serverTestSuite).TestAllocIDAfterLeaderChange()
      /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/server_test.go:101 +0x135
  github.com/pingcap/failpoint.parseTerm()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:149 +0x364
  github.com/pingcap/failpoint.parse()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:105 +0x276
  github.com/pingcap/failpoint.Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:225 +0x9c
  github.com/tikv/pd/tests/integrations/mcs/scheduling.(*serverTestSuite).TestAllocIDAfterLeaderChange()
      /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/server_test.go:98 +0x9d
  runtime.call16()
      /opt/hostedtoolcache/go/1.21.4/x64/src/runtime/asm_amd64.s:747 +0x42
  reflect.Value.Call()
      /opt/hostedtoolcache/go/1.21.4/x64/src/reflect/value.go:380 +0xb5
  github.com/stretchr/testify/suite.Run.func1()
      /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:197 +0x684
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1648 +0x44

Goroutine 89542 (running) created at:
  net/http.(*Server).Serve()
      /opt/hostedtoolcache/go/1.21.4/x64/src/net/http/server.go:3086 +0x80c
  go.etcd.io/etcd/embed.(*serveCtx).serve.func3()
      /home/runner/go/pkg/mod/go.etcd.io/[email protected]/embed/serve.go:141 +0x4f

Goroutine 83520 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1648 +0x82a
  github.com/stretchr/testify/suite.runTests()
      /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:242 +0x186
  github.com/stretchr/testify/suite.Run()
      /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:215 +0x9d2
  github.com/pingcap/failpoint.parse()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:105 +0x276
  github.com/pingcap/failpoint.Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:225 +0x7a
  github.com/tikv/pd/tests/integrations/mcs/scheduling.(*serverTestSuite).SetupSuite()
      /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/server_test.go:62 +0x7b
  github.com/stretchr/testify/suite.Run()
      /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:154 +0x5fe
  github.com/tikv/pd/tests/integrations/mcs/scheduling.TestServerTestSuite()
      /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/server_test.go:56 +0x3c
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.4/x64/src/testing/testing.go:1648 +0x44
==================
@rleungx rleungx added the type/bug The issue is confirmed as a bug. label Nov 23, 2023
@rleungx rleungx mentioned this issue Nov 23, 2023
Merged
ti-chi-bot bot added a commit that referenced this issue Nov 23, 2023
@rleungx @ti-chi-bot
*: fix data race when starting server (#7420)
8198735 
close #7419

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@rleungx rleungx mentioned this issue Nov 23, 2023
Closed
rleungx added a commit to rleungx/pd that referenced this issue Dec 1, 2023
@rleungx @ti-chi-bot
*: fix data race when starting server (tikv#7420)
a088610 
close tikv#7419

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
severity/minor type/bug The issue is confirmed as a bug.
Projects
Questions and Bug Reports
Status: Closed
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

*: fix data race when starting server rleungx/pd
1 participant
@rleungx