Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data Race when running schrddl test #6134

Closed
Tracked by #4674
maxshuang opened this issue Jun 30, 2022 · 7 comments
Closed
Tracked by #4674

Data Race when running schrddl test #6134

maxshuang opened this issue Jun 30, 2022 · 7 comments
Assignees
@hicqu
Labels
area/ticdc Issues or PRs related to TiCDC. may-affects-4.0 may-affects-5.0 may-affects-5.1 may-affects-5.2 may-affects-5.3 may-affects-5.4 may-affects-6.0 may-affects-6.1 severity/minor type/bug The issue is confirmed as a bug.

Comments

@maxshuang
Copy link
Contributor

maxshuang commented Jun 30, 2022 •
edited
Loading

What did you do?

Running this pr #4696 will raise the data race of sink module.

ci:
https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_integration_test/detail/cdc_ghpr_integration_test/13349/pipeline

What did you expect to see?

Run normally

What did you see instead?

Data race of sink module:

github.com/pingcap/tiflow/cdc/sink/mysql.(*mysqlSink).execDDL()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/sink/mysql/mysql.go:322 +0x23a
  github.com/pingcap/tiflow/cdc/sink/mysql.(*mysqlSink).execDDLWithMaxRetries.func1()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/sink/mysql/mysql.go:296 +0x84
  github.com/pingcap/tiflow/pkg/retry.run()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:57 +0x1a4
  github.com/pingcap/tiflow/pkg/retry.Do()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:34 +0x84
  github.com/pingcap/tiflow/cdc/sink/mysql.(*mysqlSink).execDDLWithMaxRetries()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/sink/mysql/mysql.go:295 +0x158
  github.com/pingcap/tiflow/cdc/sink/mysql.(*mysqlSink).EmitDDLEvent()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/sink/mysql/mysql.go:290 +0x84
  github.com/pingcap/tiflow/cdc/owner.(*ddlSinkImpl).run.func1()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/owner/ddl_sink.go:182 +0x1118

Previous write at 0x00c002b13060 by goroutine 88:
  github.com/pingcap/tiflow/cdc/owner.(*changefeed).asyncExecDDLEvent()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/owner/changefeed.go:634 +0xb1
  github.com/pingcap/tiflow/cdc/owner.(*changefeed).asyncExecDDLJob()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/owner/changefeed.go:614 +0xc49
  github.com/pingcap/tiflow/cdc/owner.(*changefeed).handleBarrier()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/owner/changefeed.go:535 +0x772
  github.com/pingcap/tiflow/cdc/owner.(*changefeed).tick()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/owner/changefeed.go:248 +0x88b
  github.com/pingcap/tiflow/cdc/owner.(*changefeed).Tick()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/owner/changefeed.go:160 +0x286
  github.com/pingcap/tiflow/cdc/owner.(*ownerImpl).Tick()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/owner/owner.go:203 +0xb24
  github.com/pingcap/tiflow/pkg/orchestrator.(*EtcdWorker).Run()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/pkg/orchestrator/etcd_worker.go:247 +0x1343
  github.com/pingcap/tiflow/cdc/capture.(*Capture).runEtcdWorker()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/capture/capture.go:430 +0x1a4
  github.com/pingcap/tiflow/cdc/capture.(*Capture).campaignOwner()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/capture/capture.go:403 +0xd51
  github.com/pingcap/tiflow/cdc/capture.(*Capture).run.func2()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/capture/capture.go:279 +0x132

Goroutine 751 (running) created at:
  github.com/pingcap/tiflow/cdc/owner.(*ddlSinkImpl).run()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/owner/ddl_sink.go:124 +0x230
  github.com/pingcap/tiflow/cdc/owner.(*changefeed).initialize()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/owner/changefeed.go:365 +0xe43
  github.com/pingcap/tiflow/cdc/owner.(*changefeed).tick()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/owner/changefeed.go:226 +0x285
  github.com/pingcap/tiflow/cdc/owner.(*changefeed).Tick()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/owner/changefeed.go:160 +0x286
  github.com/pingcap/tiflow/cdc/owner.(*ownerImpl).Tick()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/owner/owner.go:203 +0xb24
  github.com/pingcap/tiflow/pkg/orchestrator.(*EtcdWorker).Run()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/pkg/orchestrator/etcd_worker.go:247 +0x1343
  github.com/pingcap/tiflow/cdc/capture.(*Capture).runEtcdWorker()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/capture/capture.go:430 +0x1a4
  github.com/pingcap/tiflow/cdc/capture.(*Capture).campaignOwner()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/capture/capture.go:403 +0xd51
  github.com/pingcap/tiflow/cdc/capture.(*Capture).run.func2()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/capture/capture.go:279 +0x132

Goroutine 88 (running) created at:
  github.com/pingcap/tiflow/cdc/capture.(*Capture).run()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/capture/capture.go:273 +0x6c5
  github.com/pingcap/tiflow/cdc/capture.(*Capture).Run()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/capture/capture.go:236 +0x2e4
  github.com/pingcap/tiflow/cdc.(*Server).run.func1()
      /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/server.go:255 +0x70
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:74 +0x86

stdout.log

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

Release Version: v6.0.0
Edition: Community
Git Commit Hash: 36a9810441ca0e496cbd22064af274b3be771081

Upstream TiKV version (execute tikv-server --version):

Release Version:   6.2.0-alpha
Edition:           Community
Git Commit Hash:   2e1513c83ffa62fc2edc3d6d28c14cb92e82ddbe

TiCDC version (execute cdc version):

v6.2.0-alpha-184-ge2e77ee17
@maxshuang maxshuang added area/ticdc Issues or PRs related to TiCDC. type/bug The issue is confirmed as a bug. labels Jun 30, 2022
@maxshuang
Copy link
Contributor Author

/severity major

@maxshuang maxshuang mentioned this issue Jun 30, 2022
Open
57 tasks
@maxshuang
Copy link
Contributor Author

I suspect the data race may be raised by closing the cdc process forcely, not by normal DDL/DML sink, because I met another data race with the same check_table_exists timeout.

I will set the severity to minor temporary.

@maxshuang
Copy link
Contributor Author

/severity minor

@maxshuang
Copy link
Contributor Author

/remove-severity major

@maxshuang
Copy link
Contributor Author

/remove-may-affects-4.0

@hicqu
Copy link
Contributor

hicqu commented Jul 1, 2022

/assign hicqu

@asddongmen
Copy link
Contributor

Close it since it's stale.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hicqu hicqu

Labels
area/ticdc Issues or PRs related to TiCDC. may-affects-4.0 may-affects-5.0 may-affects-5.1 may-affects-5.2 may-affects-5.3 may-affects-5.4 may-affects-6.0 may-affects-6.1 severity/minor type/bug The issue is confirmed as a bug.
Projects
Question and Bug Reports
Status: Done
Milestone
No milestone
Development

No branches or pull requests
4 participants
@hicqu @maxshuang @asddongmen @ti-chi-bot