Pattern acceleration #85

elliVM · 2024-09-09T07:46:57Z

Pattern acceleration feature to activate bloom filtering on set regex pattern. Goal is to limit bloom filtering to certain patterns like UUID.

Flow:

Walker walks XML and for each search term regex checks tokenized term against tables in bloomdb excluding the filtertype table
Collect list of tables with a pattern match record from SQL metadata
For each table a temp table is created
For each temp table generate condition that compares temp table filter against parent table filter using bloommatch UDF
Combine conditions from each table, if all filters were null then also true
Combine conditions from each search term
Left join tables to NestedTopNQuery and use the final combined condition in query

Notes: Tokenizer max token count is set to 0 to get only major tokens since that is what dpf_03 currently uses to tokenize the bloom filter tables

…n, primary constructor for IndexStatementCondition explanatory comments for LatestCondition and EarliestCondition

…quals and notEquals tests

… class, refactor tests

…atementConditionTest

… testing for ConditionWalker bloom search terms

…ply spotless

…t and apply spotless" This reverts commit 7dd5e33.

eemhu · 2024-09-09T11:13:06Z

src/main/java/com/teragrep/pth_06/planner/BloomFilterTempTable.java

+
+    public BloomFilterTempTable(DSLContext ctx, Table<?> parentTable, long bloomTermId, Set<Token> tokenSet) {
+        this.ctx = ctx;
+        this.parentTable = parentTable;


would be ideal if the primary constructor had no code

eemhu · 2024-09-09T11:22:15Z

src/main/java/com/teragrep/pth_06/planner/BloomFilterTempTable.java

+            filterBAOS.close();
+        }
+        catch (IOException e) {
+            throw new UncheckedIOException(e);


perhaps it would be worth it to add some extra context to this exception in addition to the original exception to help debugging in the future.

eemhu · 2024-09-09T12:18:17Z

src/test/java/com/teragrep/pth_06/planner/BloomFilterTempTableTest.java

+
+        Set<Token> tokenSet = new PatternMatch(ctx, "test").tokenSet();
+        BloomFilterTempTable tempTable = new BloomFilterTempTable(ctx, table, 0L, tokenSet);
+        Assertions.assertThrows(RuntimeException.class, tempTable::generateCondition);


check that RuntimeException.getMessage() is of expected value?

eemhu · 2024-09-09T12:30:21Z

src/test/java/com/teragrep/pth_06/walker/ConditionWalkerTest.java

+    }
+
+    @Test
+    void singleTalbePatternMatchTest() {


kortemik

please rebase this on the latest main branch, it has commits that do not belong here. i.e. license changes

elliVM · 2024-09-12T07:37:46Z

Moving to another branch

elliVM added 30 commits August 27, 2024 10:02

refactor walkers

34eb164

unit tests for conditions, replace null assignment in ElementConditio…

d1e71b8

…n, primary constructor for IndexStatementCondition explanatory comments for LatestCondition and EarliestCondition

cleanup walker conditions tests

fa7ecec

use DSL condition instead of Optional, add invalid element name tests

9502de9

conditions equality tests

331be4f

add an override for equals() in walker conditions

bb1b213

add equal pointer check for equals methods for conditions, separate e…

3c20bdd

…quals and notEquals tests

conditions use String parameters instead of getting them from Element…

caee813

… class, refactor tests

add javadoc for IndexStatementCondition equals method

7a5e3f5

add symmetric equality check and null object equality check

be9dfa3

fix IndexCondition equals method

6a7f8c1

remove unnecessary comment

e77c6fb

Rebase to walker-refactor branch initial commit

f23b15f

merge main

1363cb3

fix LatestConditionTest equals test

d7f5a73

remove BloomFilterTempTable cache add equals method and equality tests

308d34f

refactor PatternMatch and add equals method and equality tests

bb22717

apply spotless

f34fcd3

restore localhost for jdbc Driver

c91a51d

make objects final when possible add no table matches test to IndexSt…

ae3037a

…atementConditionTest

use ValidElement class to wrap Element

db9bd64

fix SourceTypeConditionTest

b80dad6

set Tokenizer maxTokenCount 0 to use only major tokens

174335e

add IndexStatementCondition matches to ElementCondition matchSet

a6f8adf

add IndexStatementCondition matches to ElementCondition matchSet, add…

c588e84

… testing for ConditionWalker bloom search terms

add bloom enabled stream query tests for ConditionWalkerTest

75e0ef6

add equality test for BloomFilterTempTable and fix javadoc

9bd2f85

apply spotless

6c763e0

check for year formating in license script, run license script and ap…

7dd5e33

…ply spotless

Revert "check for year formating in license script, run license scrip…

948c2da

…t and apply spotless" This reverts commit 7dd5e33.

update PatternMatch comments

0f2dd5f

elliVM self-assigned this Sep 9, 2024

elliVM changed the title ~~Pattern acceleration refactor~~ Pattern acceleration Sep 9, 2024

elliVM linked an issue Sep 9, 2024 that may be closed by this pull request

Term accelerated searches using bloomfilter #30

Closed

eemhu reviewed Sep 9, 2024

View reviewed changes

src/test/java/com/teragrep/pth_06/walker/ConditionWalkerTest.java Outdated

}

@Test

void singleTalbePatternMatchTest() {

Copy link

Contributor

eemhu Sep 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo

kortemik requested changes Sep 9, 2024

View reviewed changes

elliVM added 2 commits September 10, 2024 08:39

refactor BloomFilterTempTable remove code from constructor

5db6bdb

check exception message, fix name typo

f148f57

elliVM removed a link to an issue Sep 12, 2024

Term accelerated searches using bloomfilter #30

Closed

elliVM closed this Sep 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pattern acceleration #85

Pattern acceleration #85

elliVM commented Sep 9, 2024 •

edited

Loading

eemhu Sep 9, 2024

eemhu Sep 9, 2024

eemhu Sep 9, 2024

eemhu Sep 9, 2024

kortemik left a comment

elliVM commented Sep 12, 2024

Pattern acceleration #85

Pattern acceleration #85

Conversation

elliVM commented Sep 9, 2024 • edited Loading

eemhu Sep 9, 2024

Choose a reason for hiding this comment

eemhu Sep 9, 2024

Choose a reason for hiding this comment

eemhu Sep 9, 2024

Choose a reason for hiding this comment

eemhu Sep 9, 2024

Choose a reason for hiding this comment

kortemik left a comment

Choose a reason for hiding this comment

elliVM commented Sep 12, 2024

elliVM commented Sep 9, 2024 •

edited

Loading