Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chunked encoding for cluster state API #92285

Merged

Conversation

DaveCTurner
Copy link
Contributor

Relates #89838

@DaveCTurner DaveCTurner added >non-issue :Distributed Coordination/Network Http and internode communication implementations v8.7.0 labels Dec 12, 2022
@elasticsearchmachine elasticsearchmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Dec 12, 2022
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@DaveCTurner
Copy link
Contributor Author

NB most of these changes are just to add support for XContent fragments throughout. Kind of noisy. I could make it a separate API instead of adding a boolean parameter, maybe that would be nicer.

@DaveCTurner
Copy link
Contributor Author

#92534 will fix the failure; @elasticmachine please run elasticsearch-ci/part-1

@DaveCTurner
Copy link
Contributor Author

@original-brownbear I've merged this with main, adapting it to #92549, so this is worth another look

Copy link
Member

@original-brownbear original-brownbear left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Phew did my best to carefully read all the steps here, couldn't find any spot where the format would change or where we'd still serialize a problematically huge chunk => LGTM :)

metrics.contains(Metric.CUSTOMS)
? Iterators.flatMap(
customs.entrySet().iterator(),
cursor -> Iterators.concat(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: could use org.elasticsearch.common.xcontent.ChunkedToXContentHelper#wrapWithObject here to shorten this a little

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++ thanks, see c5bc46d

* Response listener for REST requests which dispatches the serialization of the response off of the thread on which the response was
* received, since that thread is often a transport thread and XContent serialization might be expensive.
*/
public class DispatchingRestToXContentListener<Response extends ToXContentObject> extends RestActionListener<Response> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

@DaveCTurner DaveCTurner added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Dec 30, 2022
@elasticsearchmachine elasticsearchmachine merged commit f332fc2 into elastic:main Dec 30, 2022
@DaveCTurner DaveCTurner deleted the 2022-12-12-chunked-cluster-state branch December 30, 2022 22:14
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this pull request Dec 31, 2022
The cluster reroute API (optionally) returns the cluster state in its
response, which can therefore be rather large. elastic#92285 enables a chunked
encoding of the cluster state, and this commit adjusts the reroute API
to make use of this encoding too.
elasticsearchmachine pushed a commit that referenced this pull request Jan 24, 2023
The cluster reroute API (optionally) returns the cluster state in its
response, which can therefore be rather large. #92285 enables a chunked
encoding of the cluster state, and this commit adjusts the reroute API
to make use of this encoding too.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) :Distributed Coordination/Network Http and internode communication implementations >non-issue Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. v8.7.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants