Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement chunk size alignment #336

Open
rohiievych opened this issue Jul 23, 2022 · 0 comments
Open

Implement chunk size alignment #336

rohiievych opened this issue Jul 23, 2022 · 0 comments

Comments

@rohiievych
Copy link
Member

Problem description

Stream sources can produce chunks with variable sizes. This behavior leads to uncontrolable websocket message sizes, what makes 2 problems:

  1. If chunks are too small -> a lot of messages -> poorer performance due to high fragmentation.
  2. If chunks are too big -> small number of messages ->
    2.1. In case of reconnection we'll have to resend big messages.
    2.2. A lot of connections can take really big amounts of memory to handle such chunks.

Describe the solution you'd like

There must be a configurable chunk size, to which source chunks will be aligned. In any case writable stream should send chunks with constant size, except the last one with remaining data. Big chunks have to be divided into smaller messages and small chunks have to be batched into bigger messages.

Possible solution:

  1. Create a Uint8Array with configured constant size and fill it up with data using DataView.
  2. Send constant chunks while queue is not empty and writing is in progress.

Additional context

Configurable chunk size can be investigated and set to optimal default value.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants