Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Dataflow Streaming] Optimize WindmillStateUtil.encodeKey by appending directly to ByteStringOutputStream #33735

Merged
merged 6 commits into from
Jan 29, 2025

Conversation

arunpandianp
Copy link
Contributor

@arunpandianp arunpandianp commented Jan 23, 2025

OutputStreamWriter is thread safe and has synchronization overhead. Each OutputStreamWriter allocates a 512 byte buffer which is not useful when writing in-memory ByteStrings.

The buffer allocations show up starkly on cpu profiles.

Screenshot 2025-01-22 at 8 37 55 PM

#33578

…gAppendable

OutputStreamWriter is thread safe and has synchronization overhead.
Each OutputStreamWriter allocates a 512 byte buffer which is
useful when writing inmemory ByteStrings.
@arunpandianp
Copy link
Contributor Author

R: @scwhittle

Copy link
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

@arunpandianp arunpandianp changed the title [Dataflow Streaming] replace OutputStreamWriter with custom ByteStringAppendable [Dataflow Streaming] replace OutputStreamWriter with custom ByteStringAppendable in encodeKey Jan 23, 2025
@arunpandianp
Copy link
Contributor Author

Run Java PreCommit

@arunpandianp
Copy link
Contributor Author

Run Java PerCommit

@arunpandianp
Copy link
Contributor Author

Run Java PreCommit

@scwhittle
Copy link
Contributor

needs resolving

@arunpandianp
Copy link
Contributor Author

resolved conflicts

@scwhittle scwhittle changed the title [Dataflow Streaming] replace OutputStreamWriter with custom ByteStringAppendable in encodeKey [Dataflow Streaming] encodeKey Jan 27, 2025
@scwhittle scwhittle changed the title [Dataflow Streaming] encodeKey [Dataflow Streaming] Optimize WindmillStateUtil.encodeKey by appending directly to ByteStringOutputStream Jan 27, 2025
@arunpandianp
Copy link
Contributor Author

Run Java PreCommit

2 similar comments
@arunpandianp
Copy link
Contributor Author

Run Java PreCommit

@arunpandianp
Copy link
Contributor Author

Run Java PreCommit

@arunpandianp
Copy link
Contributor Author

Run Java PreCommit

@scwhittle
Copy link
Contributor

unrelated failures

@scwhittle scwhittle merged commit e688295 into apache:master Jan 29, 2025
22 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants