Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perf: Optimize SQL transaction commits in binlog replication applier #8303

Merged
merged 1 commit into from
Aug 29, 2024

Conversation

fulghum
Copy link
Contributor

@fulghum fulghum commented Aug 27, 2024

The binlog replication applier process is responsible for applying the replicated updates from a MySQL source to the Dolt replica. Previously, when committing SQL transactions after applying a change, the applier would attempt to commit on all known databases. This worked when the number of databases was small, but when the server is handling hundreds of databases, it adds noticeable and unnecessary delay. This change switches to use the dirty tracking data of the session to more efficiently commit to only the databases that have had changes applied.

Performance Comparison

The old SQL transaction commit logic scaled exponentially, while the new logic scales linearly.

Num Databases Code Path Replication Time (s)
100 Old 11.02
300 Old 144.34
500 Old 633.44
100 New 6.79
300 New 20.51
500 New 36.04

…pplier thread so that we don't have to commit in all databases
@fulghum fulghum force-pushed the fulghum/replication-bugfix branch from 9dd4a14 to 4c4e9bc Compare August 28, 2024 18:49
@coffeegoddd
Copy link
Contributor

@fulghum DOLT

comparing_percentages
100.000000 to 100.000000
version result total
4c4e9bc ok 5937457
version total_tests
4c4e9bc 5937457
correctness_percentage
100.0

@dolthub dolthub deleted a comment from coffeegoddd Aug 28, 2024
@fulghum fulghum requested a review from jycor August 29, 2024 17:39
Copy link
Contributor

@jycor jycor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@fulghum fulghum merged commit e871e9c into main Aug 29, 2024
21 checks passed
@fulghum fulghum deleted the fulghum/replication-bugfix branch August 29, 2024 17:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants