Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pagination to diagnostic APIs #87555

Open
Tracked by #77466
original-brownbear opened this issue Jun 9, 2022 · 1 comment
Open
Tracked by #77466

Add pagination to diagnostic APIs #87555

original-brownbear opened this issue Jun 9, 2022 · 1 comment
Assignees
Labels
:Data Management/Stats Statistics tracking and retrieval APIs >enhancement Team:Data Management Meta label for data/management team

Comments

@original-brownbear
Copy link
Member

original-brownbear commented Jun 9, 2022

The Indices stats API does not scale well for very large index counts.
The response size can reach hundreds of MBs for tens of thousands of indices
and response times eventually go into the tens of seconds due to the
CPU time required for creating and serializing very large responses.

We discussed this during the many-shards sync and decided on introducing
pagination into the indices stats API to enable clients to bound the API's
response size to manageable levels.
This can be done by adding two parameters to indices stats that enable pagination
(sorted by index name):

  • size to limit the number of indices that stats are fetched for
  • from to provide the index name to search from (analogous to search_after in search or the from parameter in the get-snapshots API)

relates #77466

Also relates to #88094 (sorting in APIs) and #88095 (filtering in APIs)

@original-brownbear original-brownbear added >enhancement :Data Management/Stats Statistics tracking and retrieval APIs labels Jun 9, 2022
@original-brownbear original-brownbear self-assigned this Jun 9, 2022
@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Jun 9, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@original-brownbear original-brownbear changed the title Add Pagination to the index stats API Add pagination to the index stats API Jun 9, 2022
@dakrone dakrone changed the title Add pagination to the index stats API Add pagination to diagnostic APIs Jun 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Stats Statistics tracking and retrieval APIs >enhancement Team:Data Management Meta label for data/management team
Projects
None yet
Development

No branches or pull requests

2 participants