Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

query/sidecar: same store.limits flags on both components #7613

Open
arnoldyahad opened this issue Aug 8, 2024 · 4 comments
Open

query/sidecar: same store.limits flags on both components #7613

arnoldyahad opened this issue Aug 8, 2024 · 4 comments

Comments

@arnoldyahad
Copy link

Thanos and Prometheus version used:

Thanos 0.34.1
Prometheus v2.51.1

What happened:
We are trying to implement limits on querying to avoid thanos-querier reaching OOM.

We want to utilize these 2 flags:

--store.limits.request-samples
--store.limits.request-series

We see in the docs they both appear for the Sidecar and the querier:
https://thanos.io/tip/components/sidecar.md/#flags
https://thanos.io/tip/components/query.md/#flags

but when setting the flags on querier it doesn't block anything.
but still shows them in the UI as they are set.

only when setting them on the Sidecar it actually blocks.

What you expected to happen:
Flags to work when they set either in sidecar or querier.
Also - if both flags are set in these 2 components - which takes precedence?

How to reproduce it (as minimally and precisely as possible):
set up prometheus + thanos sidecar in one pod and another pod of thanos querier.

set these 2 flags with values other than 0 on the querier:

--store.limits.request-samples=100
--store.limits.request-series=5

set these 2 flags with different values on the sidecar.
try to query prometheus through thanos querier - and see that the values that are set in the querier dont really apply anything.

is this the desire behavior?

@tizki
Copy link
Contributor

tizki commented Sep 1, 2024

I'm having the same issue, tried to debug it locally and it seems that the query doesn't get to the part in the code that checks those limits.
it's either a bug or wrong documentation.

@MichaHoffmann
Copy link
Contributor

You were right, those flags were just ignored during query evaluation since we didnt use the limited proxy store in the queryable that is passed to the promql engine! I opened #7679 to fix it.

@yeya24
Copy link
Contributor

yeya24 commented Sep 3, 2024

Given that #7679 has been merged, can we close this issue?

@arnoldyahad
Copy link
Author

@yeya24 @MichaHoffmann thanks a lot for answering and sorry for the delay,
i would ask to close the issue if i had a release to test it but looking at #7679 - its not a part of any release

is there going to be a release soon where i can test it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants