-
Notifications
You must be signed in to change notification settings - Fork 245
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Redesign of the DatePartitionedQueryPlanner (formerly know as the FederatedQueryPlanner) #2717
base: integration
Are you sure you want to change the base?
Conversation
Updated TODOs
what fields are unindexed in each range
This reverts commit 549130b.
warehouse/query-core/src/main/java/datawave/query/planner/DatePartitionedQueryPlanner.java
Outdated
Show resolved
Hide resolved
warehouse/query-core/src/main/java/datawave/query/planner/DatePartitionedQueryPlanner.java
Show resolved
Hide resolved
warehouse/query-core/src/test/java/datawave/query/IndexHoleQueryTest.java
Show resolved
Hide resolved
We should probably adjust the marker concept to be one day earlier than the last old index entry timestamp. Since we usually deploy during the day, any deployment where we ceased indexing something would only be a partial day. I replicated this with a test case and any data from the last indexed date that did not get indexed (i.e. post hypothetical deployment) would be missed. |
In other tests, I'm getting empty ranges. We're still checking the fi for the field with the Hole marker, so the AndIterator appears to be defeating. Haven't fully stepped through, though, but we need more end-to-end tests up in here (up in here). |
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Outdated
Show resolved
Hide resolved
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Show resolved
Hide resolved
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Outdated
Show resolved
Hide resolved
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Outdated
Show resolved
Hide resolved
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Outdated
Show resolved
Hide resolved
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Outdated
Show resolved
Hide resolved
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Outdated
Show resolved
Hide resolved
I changed this to use evaluation only instead of index value hole markers in the query to avoid the field index lookups for unindexed fields. |
warehouse/query-core/src/main/java/datawave/query/planner/DatePartitionedQueryPlanner.java
Outdated
Show resolved
Hide resolved
warehouse/query-core/src/main/java/datawave/query/planner/DatePartitionedQueryPlanner.java
Outdated
Show resolved
Hide resolved
warehouse/query-core/src/main/java/datawave/query/planner/DatePartitionedQueryPlanner.java
Outdated
Show resolved
Hide resolved
warehouse/query-core/src/main/java/datawave/query/planner/FederatedQueryIterable.java
Outdated
Show resolved
Hide resolved
warehouse/query-core/src/main/java/datawave/query/planner/DatePartitionedQueryPlanner.java
Outdated
Show resolved
Hide resolved
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Outdated
Show resolved
Hide resolved
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Show resolved
Hide resolved
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Outdated
Show resolved
Hide resolved
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Show resolved
Hide resolved
warehouse/query-core/src/main/java/datawave/query/planner/DatePartitionedQueryPlanner.java
Outdated
Show resolved
Hide resolved
cases. Refactored part of the JexlStringBuildingVisitor to simplify adding some missing operators.
No description provided.