feat: support concurrent apply dml binlog

[shaohk]

Description

This PR is support concurrency apply dml binlog to _gho table.

In case this PR introduced Go code changes:

• contributed code is using same conventions as original code
• script/cibuild returns with no formatting errors, build errors or unit test errors.

[morgo]

I believe this is only safe to do if you verify that the unique key that you are using is memory-comparable. i.e. a VARCHAR(255) PK will have collations. So from MySQL's perspective 'A' == 'a', but in the go-lang code they will have two entries in the map/will be treated as different keys.

[shaohk]

I believe this is only safe to do if you verify that the unique key that you are using is memory-comparable. i.e. a VARCHAR(255) PK will have collations. So from MySQL's perspective 'A' == 'a', but in the go-lang code they will have two entries in the map/will be treated as different keys.

Yes, U R right, I have added a restriction that only allows concurrent operations when the unique index column of the chunk data is of type 'int'; otherwise, concurrency is not allowed.

[dnovitski]

Am looking with interest to this PR as our migrations can only progress at night due to heavy write load during the day, even though our database has more than enough capacity to perform the DMLs from the binlogs concurrently instead of 1-by-1.

Would be great if we can also progress during daytime with this.

@shaohk Any progress or help needed?

[shaohk]

Am looking with interest to this PR as our migrations can only progress at night due to heavy write load during the day, even though our database has more than enough capacity to perform the DMLs from the binlogs concurrently instead of 1-by-1.

Would be great if we can also progress during daytime with this.

@shaohk Any progress or help needed?

@dnovitski Are you sure that parallel replay of binlog events can meet your needs? From what you've described, there's heavy write activity during the day, so tasks can only run at night. Does that mean when the tasks run at night, serial replay of binlog events can't catch up with the binlog? So, you're considering parallel replay?

[dnovitski]

@dnovitski Are you sure that parallel replay of binlog events can meet your needs? From what you've described, there's heavy write activity during the day, so tasks can only run at night. Does that mean when the tasks run at night, serial replay of binlog events can't catch up with the binlog? So, you're considering parallel replay?

During the night binlogs are caught up, and so table copy can continue mostly only at night for us. We've been testing with increasing dml-batch-size to 2000 and higher (patching the hardcoded MaxEventsSize value), and increasing this also allows us to occassionally catch up with binlogs even during the daytime. So we expect even better performance with parallel binlog apply, allowing us even to continue the table copy during daytime 100% of the time.

[shaohk]

@dnovitski Are you sure that parallel replay of binlog events can meet your needs? From what you've described, there's heavy write activity during the day, so tasks can only run at night. Does that mean when the tasks run at night, serial replay of binlog events can't catch up with the binlog? So, you're considering parallel replay?

During the night binlogs are caught up, and so table copy can continue mostly only at night for us. We've been testing with increasing dml-batch-size to 2000 and higher (patching the hardcoded MaxEventsSize value), and increasing this also allows us to occassionally catch up with binlogs even during the daytime. So we expect even better performance with parallel binlog apply, allowing us even to continue the table copy during daytime 100% of the time.

Got it.