Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug] sacle 2 to 3 ,mysql-operator restart #368

Closed
acekingke opened this issue Jan 17, 2022 · 1 comment · Fixed by #370
Closed

[bug] sacle 2 to 3 ,mysql-operator restart #368

acekingke opened this issue Jan 17, 2022 · 1 comment · Fixed by #370
Assignees
Labels
bug Something isn't working
Milestone

Comments

@acekingke
Copy link
Contributor

Describe the problem
When sacle 2 to 3 . It has error as following:

image

2022-01-17T04:41:34.255Z ERROR mysqlcluster.syncer failed to check slave status {"node": "sample-mysql-2.sample-mysql.default", "error": "Last_SQL_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 1 failed executing transaction '731d75c2-7749-11ec-b541-ba013c392c06:1' at master log mysql-bin.000002, end_log_pos 321. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any."}

github.com/radondb/radondb-mysql-kubernetes/mysqlcluster/syncer.(*StatusSyncer).updateNodeStatus

/workspace/mysqlcluster/syncer/status.go:209

github.com/radondb/radondb-mysql-kubernetes/mysqlcluster/syncer.(*StatusSyncer).Sync

/workspace/mysqlcluster/syncer/status.go:144

github.com/presslabs/controller-util/syncer.Sync

/go/pkg/mod/github.com/presslabs/[email protected]/syncer/syncer.go:82

github.com/radondb/radondb-mysql-kubernetes/controllers.(*StatusReconciler).Reconcile

/workspace/controllers/status_controller.go:100

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler

/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem

/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2

/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214

2022-01-17T04:41:34.277Z ERROR controller-runtime.manager.controller.mysqlcluster.controllers.Status failed to update cluster status {"reconciler group": "mysql.radondb.com", "reconciler kind": "MysqlCluster", "name": "sample", "namespace": "default", "error": "Operation cannot be fulfilled on mysqlclusters.mysql.radondb.com \"sample\": the object has been modified; please apply your changes to the latest version and try again"}

To Reproduce

  1. create a cluster with 2 replicas
  2. modify yaml file to set replicas to 3, then apply it
  3. manger pod got errors like above

Expected behavior

Environment:

  • RadonDB MySQL version: mysql-operator 2.1.1
@acekingke acekingke added the bug Something isn't working label Jan 17, 2022
@acekingke acekingke self-assigned this Jan 17, 2022
@acekingke acekingke added this to the v2.1.2 milestone Jan 17, 2022
@acekingke
Copy link
Contributor Author

Reason: The new pod is given priority to clone init from the follower pod,
and the slave information such as relaybinlog of the new pod still exists.
If new pod do not wait for xenon to execute purge gtid,
it will continue to execute with slave info, and this error will occur

acekingke added a commit to acekingke/radondb-mysql-kubernetes that referenced this issue Jan 17, 2022
andyli029 added a commit that referenced this issue Jan 27, 2022
sidecar: sacle 2 to 3 ,mysql-operator restart #368
zhl003 pushed a commit to zhl003/radondb-mysql-kubernetes that referenced this issue Aug 17, 2022
zhl003 pushed a commit to zhl003/radondb-mysql-kubernetes that referenced this issue Aug 17, 2022
sidecar: sacle 2 to 3 ,mysql-operator restart radondb#368
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant