Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ISSUE #696]🚀Support pull message consume-5 #697

Merged
merged 1 commit into from
Jun 26, 2024

Conversation

TeslaRustor
Copy link
Collaborator

@TeslaRustor TeslaRustor commented Jun 26, 2024

Which Issue(s) This PR Fixes(Closes)

Fixes #696

Brief Description

How Did You Test This Change?

Summary by CodeRabbit

  • New Features

    • Added capability to check if a message is in memory by commit offset.
    • Introduced methods for iterating over mapped buffer results.
  • Improvements

    • Enhanced ConsumeQueueExt with a new get method.
    • Updated CqExtUnit with public access methods.
    • Adjusted several functions to return optional values for better error handling.

Copy link
Contributor

coderabbitai bot commented Jun 26, 2024

Walkthrough

This update introduces enhancements and new features to various components of RocketMQ, with a primary focus on supporting pull message consumption. Significant changes include adding methods for checking if messages are in memory or cache, updating buffer selection logic, and enhancing access methods for struct properties.

Changes

File Path Change Summary
rocketmq-remoting/src/protocol/header/... Added import statement for RpcRequestHeader and initialized topic_request.
rocketmq-store/src/base/select_result.rs Added method is_in_mem to SelectMappedBufferResult.
rocketmq-store/src/consume_queue/consume_queue_ext.rs Updated CqExtUnit struct with public access methods.
rocketmq-store/src/log_file/mapped_file/default_impl.rs Changed is_in_cache property to true and is_loaded method to return true.
rocketmq-store/src/message_store/default_message_store.rs Added method check_in_mem_by_commit_offset.
rocketmq-store/src/queue.rs Added default implementation for CqUnit and modified get function.
rocketmq-store/src/queue/batch_consume_queue.rs Updated get function to return Option<CqUnit>.
rocketmq-store/src/queue/consume_queue_ext.rs Added get method to ConsumeQueueExt struct.
rocketmq-store/src/queue/single_consume_queue.rs Added imports, methods, and ConsumeQueueIterator struct.

Assessment against linked issues

Objective Addressed Explanation
Support pull message consume-5 (#696)

Poem

In the heart of code that soars,
Messages pull, in mem they store.
Buffers mapped and units defined,
Secrets of the queue aligned.
From cache to queue, the rabbit's tale,
In RocketMQ, none shall fail. 🌕🚀


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@github-actions github-actions bot requested review from mxsm and SpaceXCN June 26, 2024 08:00
@mxsm
Copy link
Owner

mxsm commented Jun 26, 2024

🚀Thanks for your contribution🎉. CodeRabbit(AI) will review your code first🔥

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 56c572e and 0699231.

Files selected for processing (9)
  • rocketmq-remoting/src/protocol/header/pull_message_request_header.rs (2 hunks)
  • rocketmq-store/src/base/select_result.rs (2 hunks)
  • rocketmq-store/src/consume_queue/consume_queue_ext.rs (1 hunks)
  • rocketmq-store/src/log_file/mapped_file/default_impl.rs (3 hunks)
  • rocketmq-store/src/message_store/default_message_store.rs (2 hunks)
  • rocketmq-store/src/queue.rs (2 hunks)
  • rocketmq-store/src/queue/batch_consume_queue.rs (1 hunks)
  • rocketmq-store/src/queue/consume_queue_ext.rs (1 hunks)
  • rocketmq-store/src/queue/single_consume_queue.rs (7 hunks)
Additional comments not posted (6)
rocketmq-store/src/consume_queue/consume_queue_ext.rs (1)

49-63: Public access methods added for CqExtUnit fields.

The methods added for accessing the fields of CqExtUnit are correctly implemented and follow Rust's conventions for getter methods. These changes enhance encapsulation by allowing controlled access to the fields.

rocketmq-store/src/queue.rs (1)

93-105: Default implementation for CqUnit.

The default implementation for CqUnit is provided, setting reasonable default values for its fields. This implementation follows Rust's conventions and helps in creating instances of CqUnit with default values easily.

rocketmq-remoting/src/protocol/header/pull_message_request_header.rs (2)

28-28: Import Statement Added: RpcRequestHeader

The addition of the RpcRequestHeader import is appropriate given the initialization of topic_request with a RpcRequestHeader instance. This change aligns with the new feature requirements.


243-246: Initialization of topic_request with RpcRequestHeader

The initialization of topic_request with a default RpcRequestHeader is crucial for supporting the new pull message consume-5 feature. This change is well-implemented.

Ensure that this new initialization is compatible and does not introduce inconsistencies with other parts of the application.

Verification successful

Initialization of topic_request with RpcRequestHeader

The initialization of topic_request with a default RpcRequestHeader is crucial for supporting the new pull message consume-5 feature. This change appears well-implemented and consistent across the application.

  • rocketmq-remoting/src/rpc/rpc_request.rs
  • rocketmq-remoting/src/rpc/rpc_request_header.rs
  • rocketmq-remoting/src/rpc/topic_request_header.rs
  • rocketmq-remoting/src/protocol/header/unregister_client_request_header.rs
  • rocketmq-remoting/src/protocol/header/get_consumer_listby_group_request_header.rs
  • rocketmq-remoting/src/protocol/header/namesrv/topic_operation_header.rs
  • rocketmq-remoting/src/protocol/header/query_consumer_offset_request_header.rs

Ensure that this new initialization is compatible and does not introduce inconsistencies with other parts of the application.

Scripts executed

The following scripts were executed for the analysis:


Script:

#!/bin/bash
# Description: Verify if the new initialization of `topic_request` is consistent across the application.

# Test: Search for usage of `RpcRequestHeader` and `topic_request` across all Rust files to ensure consistency.
rg --type rust $'RpcRequestHeader|topic_request'

Length of output: 7531

rocketmq-store/src/queue/single_consume_queue.rs (2)

24-24: Approved the addition of new imports for byte manipulation.

These imports are essential for handling byte-level data manipulation which is typical in message queuing systems.


504-508: Approved the modification in get method to return Option<CqUnit>.

This change enhances safety by properly handling cases where the queue item is not found, avoiding potential panics.

Comment on lines +321 to +333
pub fn get_index_buffer(&self, start_index: i64) -> Option<SelectMappedBufferResult> {
let mapped_file_size = self.mapped_file_size;
let offset = start_index * CQ_STORE_UNIT_SIZE as i64;
if offset >= self.get_min_logic_offset() {
if let Some(mapped_file) = self
.mapped_file_queue
.find_mapped_file_by_offset(offset, false)
{
return mapped_file.select_mapped_buffer((offset % mapped_file_size as i64) as i32);
}
}
None
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add error handling in get_index_buffer method.

Consider adding more robust error handling and validation for out-of-bounds or invalid index values to prevent potential runtime errors.

if offset < self.get_min_logic_offset() {
    return None; // Early return if offset is out of bounds
}

Comment on lines +818 to +879
struct ConsumeQueueIterator {
smbr: Option<SelectMappedBufferResult>,
relative_pos: i32,
counter: i32,
consume_queue_ext: Option<ConsumeQueueExt>,
}

impl ConsumeQueueIterator {
fn get_ext(&self, offset: i64, cq_ext_unit: &CqExtUnit) -> bool {
match self.consume_queue_ext.as_ref() {
None => false,
Some(value) => value.get(offset, cq_ext_unit),
}
}
}

impl Iterator for ConsumeQueueIterator {
type Item = CqUnit;

fn next(&mut self) -> Option<Self::Item> {
match self.smbr.as_ref() {
None => None,
Some(value) => {
if self.counter * CQ_STORE_UNIT_SIZE >= value.size {
return None;
}
let mmp = value.mapped_file.as_ref().unwrap().get_mapped_file();
let start =
value.start_offset as usize + (self.counter * CQ_STORE_UNIT_SIZE) as usize;
self.counter += 1;
let end = start + CQ_STORE_UNIT_SIZE as usize;
let mut bytes = Bytes::copy_from_slice(&mmp[start..end]);
let pos = bytes.get_i64();
let size = bytes.get_i32();
let tags_code = bytes.get_i64();
let mut cq_unit = CqUnit {
queue_offset: start as i64 / CQ_STORE_UNIT_SIZE as i64,
size,
pos,
tags_code,
..CqUnit::default()
};

if ConsumeQueueExt::is_ext_addr(cq_unit.tags_code) {
let cq_ext_unit = CqExtUnit::default();
let ext_ret = self.get_ext(cq_unit.tags_code, &cq_ext_unit);
if ext_ret {
cq_unit.tags_code = cq_ext_unit.tags_code();
cq_unit.cq_ext_unit = Some(cq_ext_unit);
} else {
error!(
"[BUG] can't find consume queue extend file content! addr={}, \
offsetPy={}, sizePy={}",
cq_unit.tags_code, cq_unit.pos, cq_unit.pos,
);
}
}
Some(cq_unit)
}
}
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactor ConsumeQueueIterator for better error handling and state management.

The iterator should robustly handle cases where the underlying data might be modified during iteration, preventing inconsistent states or crashes.

if let Some(mmp) = value.mapped_file.as_ref() {
    // Proceed with existing logic
} else {
    error!("Mapped file missing during iteration.");
    return None;
}

Comment on lines +441 to +446
fn check_in_mem_by_commit_offset(&self, offset_py: i64, size: i32) -> bool {
let message = self.commit_log.get_message(offset_py, size);
match message {
None => false,
Some(msg) => msg.is_in_mem(),
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tip

Codebase Verification

Potential issue with get_message method implementation

The get_message method in commit_log.rs currently returns None, which will cause the check_in_mem_by_commit_offset method to always return false. Ensure that get_message is fully implemented and correctly retrieves messages from the commit log.

  • rocketmq-store/src/log_file/commit_log.rs: get_message method needs implementation.
Analysis chain

Method check_in_mem_by_commit_offset looks good but verify dependent methods.

The implementation uses Rust's pattern matching effectively. Ensure that the get_message and is_in_mem methods of CommitLog and Message respectively are optimized and correctly handle edge cases.

Scripts executed

The following scripts were executed for the analysis:


Script:

#!/bin/bash
# Description: Verify the implementations of `get_message` and `is_in_mem`.

# Test: Search for the implementations of these methods. Expect: Efficient and correct implementations.
ast-grep --lang rust --pattern $'impl CommitLog {
  $$$
  get_message($_, $_) {
    $$$
  }
  $$$  
}'
ast-grep --lang rust --pattern $'impl Message {
  $$$
  is_in_mem() {
    $$$
  }
  $$$  
}'

Length of output: 1169



Script:

#!/bin/bash
# Description: Verify the implementations of `get_message` and `is_in_mem`.

# Corrected pattern for `get_message` method in `CommitLog` implementation.
ast-grep --lang rust --pattern 'impl CommitLog {
  fn get_message($_, $_) -> $_ {
    $$$
  }
}'

# Corrected pattern for `is_in_mem` method in `Message` implementation.
ast-grep --lang rust --pattern 'impl Message {
  fn is_in_mem(&self) -> $_ {
    $$$
  }
}'

Length of output: 192



Script:

#!/bin/bash
# Description: Search for the implementations of `get_message` and `is_in_mem` methods across the codebase.

# Search for `get_message` method in the codebase.
rg 'fn get_message'

# Search for `is_in_mem` method in the codebase.
rg 'fn is_in_mem'

Length of output: 1278



Script:

#!/bin/bash
# Extract and review the implementation of `get_message` method from commit_log.rs.
rg -A 10 -B 2 'fn get_message' rocketmq-store/src/log_file/commit_log.rs

# Extract and review the implementation of `is_in_mem` method from select_result.rs.
rg -A 10 -B 2 'fn is_in_mem' rocketmq-store/src/base/select_result.rs

Length of output: 1322

@@ -366,7 +366,7 @@ impl MappedFile for DefaultMappedFile {
start_offset: self.file_from_offset + pos as u64,
size,
mapped_file: Some(self),
is_in_cache: false,
is_in_cache: true,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Property is_in_cache Set to Always true

The property is_in_cache has been set to always return true. This change could potentially improve the performance by reducing disk access times, but it may also lead to data consistency issues if not handled correctly across different scenarios.

Consider adding checks or conditions that ensure this property does not lead to stale data being served, especially in environments with high concurrency or frequent data updates.

- is_in_cache: true,
+ is_in_cache: self.check_cache_validity(),

Also applies to: 391-391

@@ -588,7 +588,7 @@
}*/

fn is_loaded(&self, position: i64, size: usize) -> bool {
todo!()
true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Method is_loaded Now Always Returns true

Setting is_loaded to always return true simplifies the interface but risks masking issues where data may not be properly loaded. This could lead to serious application errors.

It's recommended to implement a more robust checking mechanism that verifies the actual load status of the data.

- true
+ self.verify_data_integrity()

Committable suggestion was skipped due to low confidence.

Comment on lines +43 to +51
pub fn is_in_mem(&self) -> bool {
match self.mapped_file.as_ref() {
None => true,
Some(inner) => {
let pos = self.start_offset - inner.get_file_from_offset();
inner.is_loaded(pos as i64, self.size as usize)
}
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review the logic of is_in_mem method.

The method is_in_mem uses a match statement to determine if the mapped file is in memory based on the start_offset. The logic seems correct, but the return value when mapped_file is None is set to true, which might be incorrect as it assumes the buffer is in memory when there's no associated file. This could lead to potential bugs.

- None => true,
+ None => false,
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
pub fn is_in_mem(&self) -> bool {
match self.mapped_file.as_ref() {
None => true,
Some(inner) => {
let pos = self.start_offset - inner.get_file_from_offset();
inner.is_loaded(pos as i64, self.size as usize)
}
}
}
pub fn is_in_mem(&self) -> bool {
match self.mapped_file.as_ref() {
None => false,
Some(inner) => {
let pos = self.start_offset - inner.get_file_from_offset();
inner.is_loaded(pos as i64, self.size as usize)
}
}
}

Comment on lines +96 to +98
pub fn get(&self, address: i64, cq_ext_unit: &CqExtUnit) -> bool {
unimplemented!()
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Placeholder method get in ConsumeQueueExt.

The method get is currently marked with unimplemented!. This is a typical placeholder in Rust for functionality that is yet to be implemented. Ensure that this method is implemented before merging the PR or handling it properly to avoid runtime panics.

@@ -205,7 +205,7 @@ impl ConsumeQueueTrait for BatchConsumeQueue {
todo!()
}

fn get(&self, index: i64) -> CqUnit {
fn get(&self, index: i64) -> Option<CqUnit> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Placeholder method get in BatchConsumeQueue.

The method get is currently marked with unimplemented!. This is a typical placeholder in Rust for functionality that is yet to be implemented. Ensure that this method is implemented before merging the PR or handling it properly to avoid runtime panics.

Copy link

codecov bot commented Jun 26, 2024

Codecov Report

Attention: Patch coverage is 0% with 110 lines in your changes missing coverage. Please review.

Project coverage is 27.26%. Comparing base (56c572e) to head (0699231).

Files Patch % Lines
rocketmq-store/src/queue/single_consume_queue.rs 0.00% 53 Missing ⚠️
...q-store/src/message_store/default_message_store.rs 0.00% 23 Missing ⚠️
...ketmq-store/src/consume_queue/consume_queue_ext.rs 0.00% 14 Missing ⚠️
rocketmq-store/src/base/select_result.rs 0.00% 7 Missing ⚠️
rocketmq-store/src/queue.rs 0.00% 6 Missing ⚠️
...src/protocol/header/pull_message_request_header.rs 0.00% 4 Missing ⚠️
rocketmq-store/src/queue/consume_queue_ext.rs 0.00% 2 Missing ⚠️
rocketmq-store/src/queue/batch_consume_queue.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #697      +/-   ##
==========================================
- Coverage   27.42%   27.26%   -0.16%     
==========================================
  Files         255      255              
  Lines       19760    19868     +108     
==========================================
- Hits         5419     5417       -2     
- Misses      14341    14451     +110     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved PR has approved auto merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Support pull message consume-5
2 participants