Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] Support max concurrency specified by client side in HDFS storage #414

Closed
3 tasks done
zuston opened this issue Dec 13, 2022 · 1 comment
Closed
3 tasks done

[FEATURE] Support max concurrency specified by client side in HDFS storage #414

zuston opened this issue Dec 13, 2022 · 1 comment
Labels
help wanted Extra attention is needed

Comments

@zuston
Copy link
Member

zuston commented Dec 13, 2022 •
edited
Loading

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the feature

The HDFS storage support writing single partition concurrently in #396, but the max concurrency config is based on the server side config. If we introduce the client side config to override the server config, it will be more flexible.

Motivation

No response

Describe the solution

No response

Additional context

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@zuston zuston added the help wanted Extra attention is needed label Dec 13, 2022
zuston added a commit that referenced this issue Apr 27, 2023
@zuston
[#414] feat(client): support specifying per-partition's max concurren…
8edefdf 
…cy to write in client side (#815)

### What changes were proposed in this pull request?

1. Support specifying per-partition's max concurrency to write in client side

### Why are the changes needed?

The PR of #396 has introduced the concurrent HDFS writing for one partition, 
but the concurrency is determined by the server client. In order to increase flexibility, 
this PR supports specifying per-partition's max concurrency to write in client side

### Does this PR introduce _any_ user-facing change?

Yes. The client conf of `<client_type>.rss.client.max.concurrency.per-partition.write` and `rss.server.client.max.concurrency.limit.per-partition.write` are introduced.

### How was this patch tested?
1. UTs
@zuston
Copy link
Member Author

zuston commented Apr 28, 2023

#815 Done.

@zuston zuston closed this as completed Apr 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@zuston