Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using store_offset with OwnedMessage #368

Closed
bruceg opened this issue May 31, 2021 · 3 comments · Fixed by #376
Closed

Using store_offset with OwnedMessage #368

bruceg opened this issue May 31, 2021 · 3 comments · Fixed by #376

Comments

@bruceg
Copy link
Contributor

bruceg commented May 31, 2021

I would like to use the Consumer::store_offset interface to update the offset asynchronously after a message has been fully processed. Currently, this interface takes a BorrowedMessage parameter, but I have not figured a way to use this type due to needing a static lifetime on the async block. I could detach the borrowed message into an OwnedMessage to satisfy the static lifetime requirement, but store_offset takes a BorrowedMessage parameter and there is no way to produce one from an OwnedMessage. Even if this was possible, it is far from ideal, as store_offset only actually requires the topic, partition, and offset data, and storing the entire message just to access those is rather inefficient.

Is there another way to achieve this process?

@bruceg
Copy link
Contributor Author

bruceg commented May 31, 2021

IMO ideally Consumer::store_offset would take a generic parameter that is Into some intermediate type that stores references to the three pieces of data required. This intermediate type could then be implemented for BorrowedMessage, OwnedMessage, and hopefully a new type that just stored the topic, partition, and offset. This would make it a different interface, but one that should only break code that implements trait Consumer, which I would expect to be rare in consumers of this crate.

@binarylogic
Copy link

@fede1024 thanks for your work on this library. It looks like @blt, another Vector engineer, filed a similar issue (#89). If you could approve this direction we'd be happy to contribute the change.

@benesch
Copy link
Collaborator

benesch commented Jun 30, 2021

If you have a minute, #376 should do what you need!

benesch added a commit to benesch/rust-rdkafka that referenced this issue Oct 16, 2021
benesch added a commit to benesch/rust-rdkafka that referenced this issue Oct 16, 2021
benesch added a commit to benesch/rust-rdkafka that referenced this issue Oct 16, 2021
rodoyle pushed a commit to getditto/rust-rdkafka that referenced this issue Oct 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants