Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Tools to block and/or drop data which is bad #771

Closed
bboreham opened this issue Mar 26, 2018 · 4 comments
Closed

RFC: Tools to block and/or drop data which is bad #771

bboreham opened this issue Mar 26, 2018 · 4 comments
Labels
type/production Issues related to the production use of Cortex, inc. configuration, alerting and operating.

Comments

@bboreham
Copy link
Contributor

Suppose a user suddenly starts sending thousands of samples where the labels change each time: the ingester will create new series for each of them and quickly grow in memory. (cf #665)

Many installations of Cortex will have an upstream component to validate users, so we perhaps don't need to block inside Cortex, but we may want to have a button to erase all series from the flush queue for a particular user.

@tomwilkie tomwilkie added the type/production Issues related to the production use of Cortex, inc. configuration, alerting and operating. label Aug 24, 2018
@bboreham
Copy link
Contributor Author

Another variant: drop all series for a particular metric as they arrive, because it has been observed to be very high-cardinality.

@gouthamve
Copy link
Contributor

I think this is fixed with the -ingester.min-chunk-length

@bboreham bboreham reopened this Aug 28, 2019
@bboreham
Copy link
Contributor Author

Some aspects are improved, but this feature is to block before the data gets into ingesters.

@gouthamve
Copy link
Contributor

Closing this in favour of #1507

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/production Issues related to the production use of Cortex, inc. configuration, alerting and operating.
Projects
None yet
Development

No branches or pull requests

3 participants