Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Infra] Filter out frozen and cold tiers from source endpoint queries #201568

Closed
crespocarlos opened this issue Nov 25, 2024 · 1 comment · Fixed by #201804
Closed

[Infra] Filter out frozen and cold tiers from source endpoint queries #201568

crespocarlos opened this issue Nov 25, 2024 · 1 comment · Fixed by #201804
Assignees
@Bluefinger
Labels
bug Fixes for quality problems that affect the customer experience sdh-linked Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team

Comments

@crespocarlos
Copy link
Contributor

Summary

There are at least two queries in Infra that are not time-range bound and could potentially cause timeouts if they hit the frozen tier:

This issue has occurred in APM, and we recently got an SDH reporting the same problem with the /api/metrics/source/default and /api/metrics/source/hasData endpoints.

The solution could follow the approach used in APM #192373

AC

  • The queries above filter out frozen and cold tiers.
@crespocarlos crespocarlos added bug Fixes for quality problems that affect the customer experience sdh-linked labels Nov 25, 2024
@botelastic botelastic bot added the needs-team Issues missing a team label label Nov 25, 2024
@crespocarlos crespocarlos added Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team and removed bug Fixes for quality problems that affect the customer experience needs-team Issues missing a team label sdh-linked labels Nov 25, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

@crespocarlos crespocarlos added bug Fixes for quality problems that affect the customer experience sdh-linked labels Nov 25, 2024
@Bluefinger Bluefinger self-assigned this Nov 25, 2024
@Bluefinger Bluefinger mentioned this issue Nov 26, 2024
Merged
Bluefinger added a commit that referenced this issue Dec 3, 2024
@Bluefinger @elasticmachine @kibanamachine
[Infra] Exclude frozen/cold data tiers from source queries (#201804)
ac3b60e 
Closes #201568  

## Summary

Adds the exclude data tiers settings to the
`/api/metrics/source/hasData` and `/api/metrics/source/{sourceId}`
requests. Also applies it to the `getIndexStatus` API call.

---------

Co-authored-by: Elastic Machine <[email protected]>
Co-authored-by: kibanamachine <[email protected]>
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Dec 3, 2024
@Bluefinger
[Infra] Exclude frozen/cold data tiers from source queries (elastic#2…
81aeffa 
…01804)

Closes elastic#201568

## Summary

Adds the exclude data tiers settings to the
`/api/metrics/source/hasData` and `/api/metrics/source/{sourceId}`
requests. Also applies it to the `getIndexStatus` API call.

---------

Co-authored-by: Elastic Machine <[email protected]>
Co-authored-by: kibanamachine <[email protected]>
(cherry picked from commit ac3b60e)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Dec 3, 2024
@Bluefinger
[Infra] Exclude frozen/cold data tiers from source queries (elastic#2…
a5026f8 
…01804)

Closes elastic#201568

## Summary

Adds the exclude data tiers settings to the
`/api/metrics/source/hasData` and `/api/metrics/source/{sourceId}`
requests. Also applies it to the `getIndexStatus` API call.

---------

Co-authored-by: Elastic Machine <[email protected]>
Co-authored-by: kibanamachine <[email protected]>
(cherry picked from commit ac3b60e)
Bluefinger added a commit to Bluefinger/kibana that referenced this issue Dec 4, 2024
@Bluefinger
[Infra] Exclude frozen/cold data tiers from source queries (elastic#2…
e9db945 
…01804)

Closes elastic#201568

Adds the exclude data tiers settings to the
`/api/metrics/source/hasData` and `/api/metrics/source/{sourceId}`
requests. Also applies it to the `getIndexStatus` API call.

---------

Co-authored-by: Elastic Machine <[email protected]>
Co-authored-by: kibanamachine <[email protected]>
(cherry picked from commit ac3b60e)
hop-dev pushed a commit to hop-dev/kibana that referenced this issue Dec 5, 2024
@Bluefinger @elasticmachine
@kibanamachine @hop-dev
[Infra] Exclude frozen/cold data tiers from source queries (elastic#2…
899ed84 
…01804)

Closes elastic#201568  

## Summary

Adds the exclude data tiers settings to the
`/api/metrics/source/hasData` and `/api/metrics/source/{sourceId}`
requests. Also applies it to the `getIndexStatus` API call.

---------

Co-authored-by: Elastic Machine <[email protected]>
Co-authored-by: kibanamachine <[email protected]>
Bluefinger added a commit that referenced this issue Dec 5, 2024
@Bluefinger @kibanamachine @jennypavlova
[8.16] [Infra] Exclude frozen/cold data tiers from source queries (#2…
2dcd9bc 
…01804) (#202924)

# Backport

This will backport the following commits from `main` to `8.16`:
- [[Infra] Exclude frozen/cold data tiers from source queries
(#201804)](#201804)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Gonçalo Rica Pais da
Silva","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-12-03T18:23:13Z","message":"[Infra]
Exclude frozen/cold data tiers from source queries (#201804)\n\nCloses
#201568 \r\n\r\n## Summary\r\n\r\nAdds the exclude data tiers settings
to the\r\n`/api/metrics/source/hasData` and
`/api/metrics/source/{sourceId}`\r\nrequests. Also applies it to the
`getIndexStatus` API call.\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<[email protected]>\r\nCo-authored-by:
kibanamachine
<[email protected]>","sha":"ac3b60ee907436aeee35fc59df7c30bcb9d66bc8","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","ci:project-deploy-observability","Team:obs-ux-infra_services","Team:obs-ux-management","v8.16.0"],"number":201804,"url":"https://github.com/elastic/kibana/pull/201804","mergeCommit":{"message":"[Infra]
Exclude frozen/cold data tiers from source queries (#201804)\n\nCloses
#201568 \r\n\r\n## Summary\r\n\r\nAdds the exclude data tiers settings
to the\r\n`/api/metrics/source/hasData` and
`/api/metrics/source/{sourceId}`\r\nrequests. Also applies it to the
`getIndexStatus` API call.\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<[email protected]>\r\nCo-authored-by:
kibanamachine
<[email protected]>","sha":"ac3b60ee907436aeee35fc59df7c30bcb9d66bc8"}},"sourceBranch":"main","suggestedTargetBranches":["8.16"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/201804","number":201804,"mergeCommit":{"message":"[Infra]
Exclude frozen/cold data tiers from source queries (#201804)\n\nCloses
#201568 \r\n\r\n## Summary\r\n\r\nAdds the exclude data tiers settings
to the\r\n`/api/metrics/source/hasData` and
`/api/metrics/source/{sourceId}`\r\nrequests. Also applies it to the
`getIndexStatus` API call.\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<[email protected]>\r\nCo-authored-by:
kibanamachine
<[email protected]>","sha":"ac3b60ee907436aeee35fc59df7c30bcb9d66bc8"}},{"branch":"8.16","label":"v8.16.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"url":"https://github.com/elastic/kibana/pull/202788","number":202788,"branch":"8.17","state":"OPEN"},{"url":"https://github.com/elastic/kibana/pull/202789","number":202789,"branch":"8.x","state":"OPEN"}]}]
BACKPORT-->

---------

Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: jennypavlova <[email protected]>
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this issue Dec 9, 2024
@Bluefinger @elasticmachine
@kibanamachine @CAWilson94
[Infra] Exclude frozen/cold data tiers from source queries (elastic#2…
c547b31 
…01804)

Closes elastic#201568  

## Summary

Adds the exclude data tiers settings to the
`/api/metrics/source/hasData` and `/api/metrics/source/{sourceId}`
requests. Also applies it to the `getIndexStatus` API call.

---------

Co-authored-by: Elastic Machine <[email protected]>
Co-authored-by: kibanamachine <[email protected]>
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this issue Dec 12, 2024
@Bluefinger @elasticmachine
@kibanamachine @CAWilson94
[Infra] Exclude frozen/cold data tiers from source queries (elastic#2…
9464c79 
…01804)

Closes elastic#201568  

## Summary

Adds the exclude data tiers settings to the
`/api/metrics/source/hasData` and `/api/metrics/source/{sourceId}`
requests. Also applies it to the `getIndexStatus` API call.

---------

Co-authored-by: Elastic Machine <[email protected]>
Co-authored-by: kibanamachine <[email protected]>
kibanamachine added a commit that referenced this issue Dec 12, 2024
@kibanamachine @Bluefinger
[8.17] [Infra] Exclude frozen/cold data tiers from source queries (#2…
bf3bdf2 
…01804) (#202788)

# Backport

This will backport the following commits from `main` to `8.17`:
- [[Infra] Exclude frozen/cold data tiers from source queries
(#201804)](#201804)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Gonçalo Rica Pais da
Silva","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-12-03T18:23:13Z","message":"[Infra]
Exclude frozen/cold data tiers from source queries (#201804)\n\nCloses
#201568 \r\n\r\n## Summary\r\n\r\nAdds the exclude data tiers settings
to the\r\n`/api/metrics/source/hasData` and
`/api/metrics/source/{sourceId}`\r\nrequests. Also applies it to the
`getIndexStatus` API call.\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<[email protected]>\r\nCo-authored-by:
kibanamachine
<[email protected]>","sha":"ac3b60ee907436aeee35fc59df7c30bcb9d66bc8","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","ci:project-deploy-observability","Team:obs-ux-infra_services","Team:obs-ux-management","v8.16.0"],"title":"[Infra]
Exclude frozen/cold data tiers from source
queries","number":201804,"url":"https://github.com/elastic/kibana/pull/201804","mergeCommit":{"message":"[Infra]
Exclude frozen/cold data tiers from source queries (#201804)\n\nCloses
#201568 \r\n\r\n## Summary\r\n\r\nAdds the exclude data tiers settings
to the\r\n`/api/metrics/source/hasData` and
`/api/metrics/source/{sourceId}`\r\nrequests. Also applies it to the
`getIndexStatus` API call.\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<[email protected]>\r\nCo-authored-by:
kibanamachine
<[email protected]>","sha":"ac3b60ee907436aeee35fc59df7c30bcb9d66bc8"}},"sourceBranch":"main","suggestedTargetBranches":["8.16"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/201804","number":201804,"mergeCommit":{"message":"[Infra]
Exclude frozen/cold data tiers from source queries (#201804)\n\nCloses
#201568 \r\n\r\n## Summary\r\n\r\nAdds the exclude data tiers settings
to the\r\n`/api/metrics/source/hasData` and
`/api/metrics/source/{sourceId}`\r\nrequests. Also applies it to the
`getIndexStatus` API call.\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<[email protected]>\r\nCo-authored-by:
kibanamachine
<[email protected]>","sha":"ac3b60ee907436aeee35fc59df7c30bcb9d66bc8"}},{"branch":"8.16","label":"v8.16.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Gonçalo Rica Pais da Silva <[email protected]>
kibanamachine added a commit that referenced this issue Dec 12, 2024
@kibanamachine @Bluefinger
[8.x] [Infra] Exclude frozen/cold data tiers from source queries (#20…
c59e9e2 
…1804) (#202789)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Infra] Exclude frozen/cold data tiers from source queries
(#201804)](#201804)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Gonçalo Rica Pais da
Silva","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-12-03T18:23:13Z","message":"[Infra]
Exclude frozen/cold data tiers from source queries (#201804)\n\nCloses
#201568 \r\n\r\n## Summary\r\n\r\nAdds the exclude data tiers settings
to the\r\n`/api/metrics/source/hasData` and
`/api/metrics/source/{sourceId}`\r\nrequests. Also applies it to the
`getIndexStatus` API call.\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<[email protected]>\r\nCo-authored-by:
kibanamachine
<[email protected]>","sha":"ac3b60ee907436aeee35fc59df7c30bcb9d66bc8","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","ci:project-deploy-observability","Team:obs-ux-infra_services","Team:obs-ux-management","v8.16.0"],"title":"[Infra]
Exclude frozen/cold data tiers from source
queries","number":201804,"url":"https://github.com/elastic/kibana/pull/201804","mergeCommit":{"message":"[Infra]
Exclude frozen/cold data tiers from source queries (#201804)\n\nCloses
#201568 \r\n\r\n## Summary\r\n\r\nAdds the exclude data tiers settings
to the\r\n`/api/metrics/source/hasData` and
`/api/metrics/source/{sourceId}`\r\nrequests. Also applies it to the
`getIndexStatus` API call.\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<[email protected]>\r\nCo-authored-by:
kibanamachine
<[email protected]>","sha":"ac3b60ee907436aeee35fc59df7c30bcb9d66bc8"}},"sourceBranch":"main","suggestedTargetBranches":["8.16"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/201804","number":201804,"mergeCommit":{"message":"[Infra]
Exclude frozen/cold data tiers from source queries (#201804)\n\nCloses
#201568 \r\n\r\n## Summary\r\n\r\nAdds the exclude data tiers settings
to the\r\n`/api/metrics/source/hasData` and
`/api/metrics/source/{sourceId}`\r\nrequests. Also applies it to the
`getIndexStatus` API call.\r\n\r\n---------\r\n\r\nCo-authored-by:
Elastic Machine
<[email protected]>\r\nCo-authored-by:
kibanamachine
<[email protected]>","sha":"ac3b60ee907436aeee35fc59df7c30bcb9d66bc8"}},{"branch":"8.16","label":"v8.16.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Gonçalo Rica Pais da Silva <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Bluefinger Bluefinger

Labels
bug Fixes for quality problems that affect the customer experience sdh-linked Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Infra] Exclude frozen/cold data tiers from source queries Bluefinger/kibana
3 participants
@crespocarlos @Bluefinger @elasticmachine