Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: implement the OrderedBatchProducer #4134

Merged
merged 27 commits into from
Jun 18, 2024
Merged

feat: implement the OrderedBatchProducer #4134

merged 27 commits into from
Jun 18, 2024

Conversation

WenyXu
Copy link
Member

@WenyXu WenyXu commented Jun 12, 2024 •
edited
Loading

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

  1. Implement the OrderedBatchProducer, which attempts to aggregate multiple produce requests together.
  2. Refactor append_entires method, using OrderedBatchProducer to produce entries.
  3. Remove unused benchmarks crate

Performance

OrderedBatchProducer
Screenshot 2024-06-13 at 16 09 10

Baseline
Screenshot 2024-06-13 at 16 11 15

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR requires documentation updates.

@github-actions github-actions bot added the docs-not-required This change does not impact docs. label Jun 12, 2024
@WenyXu WenyXu force-pushed the feat/ordered-producer branch 2 times, most recently from acb2792 to f7afc13 Compare June 12, 2024 08:57
@WenyXu WenyXu marked this pull request as ready for review June 12, 2024 08:57
@WenyXu WenyXu requested a review from v0y4g3r as a code owner June 12, 2024 08:57
@WenyXu WenyXu requested review from fengjiachun and evenyag June 12, 2024 08:58
@codecov Codecov
Copy link

codecov bot commented Jun 12, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 98.12500% with 9 lines in your changes missing coverage. Please review.

Project coverage is 84.84%. Comparing base (558272d) to head (826e6d9).
Report is 5 commits behind head on main.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #4134      +/-   ##
==========================================
- Coverage   84.89%   84.84%   -0.06%     
==========================================
  Files        1022     1021       -1     
  Lines      179006   179241     +235     
==========================================
+ Hits       151970   152070     +100     
- Misses      27036    27171     +135

fengjiachun
fengjiachun reviewed Jun 12, 2024
View reviewed changes
src/log-store/src/kafka/producer/batch.rs Outdated Show resolved Hide resolved
src/log-store/src/kafka/producer/batch.rs Outdated Show resolved Hide resolved
src/log-store/src/kafka/producer/batch.rs Outdated Show resolved Hide resolved
src/log-store/src/kafka/producer/batch.rs Outdated Show resolved Hide resolved
@WenyXu WenyXu mentioned this pull request Jun 12, 2024
Closed
3 tasks
@WenyXu WenyXu marked this pull request as draft June 13, 2024 06:44
@WenyXu WenyXu force-pushed the feat/ordered-producer branch from 082d640 to 86d335d Compare June 13, 2024 08:43
@WenyXu WenyXu marked this pull request as ready for review June 13, 2024 08:44
@WenyXu WenyXu force-pushed the feat/ordered-producer branch from 86d335d to 6de1925 Compare June 13, 2024 08:51
@WenyXu WenyXu requested a review from a team as a code owner June 13, 2024 09:33
@WenyXu WenyXu force-pushed the feat/ordered-producer branch 2 times, most recently from 1401437 to 1bb5aa3 Compare June 14, 2024 06:09
@WenyXu WenyXu force-pushed the feat/ordered-producer branch from 07fb3b9 to 1a02836 Compare June 17, 2024 06:13
@WenyXu WenyXu force-pushed the feat/ordered-producer branch from 1a02836 to 4931fa6 Compare June 17, 2024 06:18
evenyag
evenyag reviewed Jun 17, 2024
View reviewed changes
config/config.md Outdated Show resolved Hide resolved
config/datanode.example.toml Outdated Show resolved Hide resolved
src/common/wal/src/config/kafka/datanode.rs Show resolved Hide resolved
src/common/wal/src/config/kafka/standalone.rs Outdated Show resolved Hide resolved
src/log-store/src/kafka/client_manager.rs Outdated Show resolved Hide resolved
src/log-store/src/kafka/log_store.rs Outdated Show resolved Hide resolved
src/log-store/src/kafka/producer.rs Outdated Show resolved Hide resolved
src/log-store/src/kafka/log_store.rs Outdated Show resolved Hide resolved
src/log-store/src/kafka/producer.rs Show resolved Hide resolved
src/log-store/src/kafka/producer.rs Outdated Show resolved Hide resolved
fengjiachun
fengjiachun reviewed Jun 17, 2024
View reviewed changes
src/log-store/src/kafka/client_manager.rs Outdated Show resolved Hide resolved
src/log-store/src/kafka/client_manager.rs Outdated Show resolved Hide resolved
src/log-store/src/kafka/log_store.rs Outdated Show resolved Hide resolved
src/log-store/src/kafka/log_store.rs Outdated Show resolved Hide resolved
src/log-store/src/kafka/log_store.rs Outdated Show resolved Hide resolved
@WenyXu WenyXu mentioned this pull request Jun 17, 2024
Closed
@WenyXu WenyXu force-pushed the feat/ordered-producer branch from 158f565 to 5246dbd Compare June 17, 2024 09:56
@WenyXu
Copy link
Member Author

WenyXu commented Jun 18, 2024

@fengjiachun @evenyag PTAL

fengjiachun
fengjiachun approved these changes Jun 18, 2024
View reviewed changes
Copy link
Collaborator

@fengjiachun fengjiachun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@WenyXu WenyXu force-pushed the feat/ordered-producer branch from 66571e3 to 66c0e36 Compare June 18, 2024 03:41
@fengjiachun fengjiachun added this pull request to the merge queue Jun 18, 2024
Merged via the queue into GreptimeTeam:main with commit ea2d067 Jun 18, 2024
57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fengjiachun fengjiachun fengjiachun approved these changes

@evenyag evenyag evenyag approved these changes

@v0y4g3r v0y4g3r Awaiting requested review from v0y4g3r v0y4g3r is a code owner

Assignees
No one assigned
Labels
docs-not-required This change does not impact docs.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@WenyXu @fengjiachun @evenyag