Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Convert line filter expressions to a set of bloom tests #12035

Merged
merged 5 commits into from
Feb 23, 2024

Conversation

salvacorts
Copy link
Contributor

@salvacorts salvacorts commented Feb 22, 2024

What this PR does / why we need it:
This PR introduces:

  • A new interface BloomTest with two test methods that take a bloom as an argument and returns a boolean. True if the test passes with the given bloom.
  • A new function FiltersToBloomTest that given a set of LineFilterExpr, it returns a BloomTest.

The logic in this PR is not yet used anywhere since we need to do some refactoring to be able to use it. These refactorings will be done on a set of follow-up PRs:
1. Pass syntax.LineFilterExpr instead of syntax.LineFilter with logproto.FilterChunkRefRequest
2. Create BloomTest from the line filters passed along through logproto.FilterChunkRefRequest and modify the fused querier to use the resulting BloomTest.
3. Map regex line filters to BloomTest when possible. The logic would be pretty similar to what's done in the simplify function at pkg/logql/log/filter.go. We can probably extract the simplification logic into [pkg/util/regex.go].(https://github.com/grafana/loki/blob/5a08a6bcb9ecc816328cb7859bbdcc7cf9d004c5/pkg/util/regex.go)

Special notes for your reviewer:

  • As explained in the bullets above, regexes are not yet tested. all regex filter expressions are mapped to Noop tests which always return true.

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • CHANGELOG.md updated
    • If the change is worth mentioning in the release notes, add add-to-release-notes label
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@salvacorts salvacorts changed the title Convert line filter expressions to a set of bloom tests feat: Convert line filter expressions to a set of bloom tests Feb 22, 2024
@salvacorts salvacorts marked this pull request as ready for review February 22, 2024 16:07
@salvacorts salvacorts requested a review from a team as a code owner February 22, 2024 16:07
Copy link
Contributor

@chaudum chaudum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking really good!

pkg/storage/bloom/v1/bloom_tester.go Outdated Show resolved Hide resolved
pkg/storage/bloom/v1/bloom_tester.go Outdated Show resolved Hide resolved
pkg/storage/bloom/v1/bloom_tester_test.go Outdated Show resolved Hide resolved
pkg/storage/bloom/v1/bloom_tester_test.go Show resolved Hide resolved
pkg/storage/bloom/v1/tokenizer.go Show resolved Hide resolved
pkg/storage/bloom/v1/tokenizer.go Show resolved Hide resolved
@salvacorts salvacorts merged commit 8193063 into main Feb 23, 2024
11 checks passed
@salvacorts salvacorts deleted the salvacorts/bloom-searches branch February 23, 2024 16:38
@salvacorts salvacorts mentioned this pull request Feb 29, 2024
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants