Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Filter splitting and rendering for related types #1541

Merged
merged 8 commits into from
May 29, 2023

Conversation

jsimnz
Copy link
Member

@jsimnz jsimnz commented May 26, 2023

Relevant issue(s)

Resolves #1539, #1540

Description

Fixes the filter splitting of related types issue. This issue was also directly related to the rendering of these related filters within the explain requests, so this PR covers both.

Tasks

  • I made sure the code is well commented, particularly hard-to-understand areas.
  • I made sure the repository-held documentation is changed accordingly.
  • I made sure the pull request title adheres to the conventional commit style (the subset used in the project can be found in tools/configs/chglog/config.yml).
  • I made sure to discuss its limitations such as threats to validity, vulnerability to mistake and misuse, robustness to invalidation of assumptions, resource requirements, etc...

How has this been tested?

Manually, CI

Extensive manual testing with varied queries, extended integration suite with both query and explain tests. Lastly verified against a complex partner query that caused the investigation into the bug originally.

Specify the platform(s) on which this was tested:

  • Ubuntu (WSL2)

@jsimnz jsimnz added bug Something isn't working action/no-benchmark Skips the action that runs the benchmark. area/planner Related to the planner system labels May 26, 2023
@jsimnz jsimnz added this to the DefraDB v0.6 milestone May 26, 2023
@jsimnz jsimnz requested review from fredcarle and shahzadlone May 26, 2023 08:03
@jsimnz jsimnz self-assigned this May 26, 2023
@jsimnz jsimnz changed the title fix: Filter splitting of related types and explain filter rendering fix: Filter splitting and rendering of related types May 26, 2023
@jsimnz jsimnz changed the title fix: Filter splitting and rendering of related types fix: Filter splitting and rendering for related types May 26, 2023
@codecov
Copy link

codecov bot commented May 26, 2023

Codecov Report

Merging #1541 (38eb4ed) into develop (c056713) will increase coverage by 0.15%.
The diff coverage is 94.33%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #1541      +/-   ##
===========================================
+ Coverage    72.10%   72.26%   +0.15%     
===========================================
  Files          185      185              
  Lines        18295    18374      +79     
===========================================
+ Hits         13192    13278      +86     
+ Misses        4060     4054       -6     
+ Partials      1043     1042       -1     
Impacted Files Coverage Δ
planner/select.go 79.22% <50.00%> (-1.06%) ⬇️
planner/mapper/targetable.go 82.19% <91.66%> (+9.21%) ⬆️
planner/type_join.go 74.93% <94.28%> (+1.67%) ⬆️
planner/count.go 95.08% <100.00%> (+0.48%) ⬆️
planner/delete.go 73.61% <100.00%> (ø)
planner/group.go 87.31% <100.00%> (ø)
planner/scan.go 88.65% <100.00%> (ø)
planner/sum.go 88.97% <100.00%> (+0.37%) ⬆️
planner/update.go 73.52% <100.00%> (ø)

... and 6 files with indirect coverage changes

v := v.([]any)
logicMapEntries := make([]any, len(v))
for i, item := range v {
itemMap := item.(map[connor.FilterKey]any)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Can this ever not be a map[connor.FilterKey]any type?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did have a whole type checks for all the type casting in this function. But there really isn't a way for them, to not be the respective types, unless there is a notable bug, in which case it should be caught much higher in the call stack.

},

// @shahzadlone: Do I need this?
// ExpectedPatterns: []dataMap{basicPattern},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

answer: Yes, you should assert that the order of the plan graph nodes is what we expect.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do

},
},

// @shahzadlone: Do I need this?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

answer: Yes, you should assert that the order of the plan graph nodes is what we expect.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do

Copy link
Member

@shahzadlone shahzadlone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was a good catch, overall looks good. I answered the questions that were left as a comment.

Moreover, you should probably also do the fix on the group filter. More specifically in (n *groupNode) simpleExplain().

@jsimnz
Copy link
Member Author

jsimnz commented May 26, 2023

Was a good catch, overall looks good. I answered the questions that were left as a comment.

Moreover, you should probably also do the fix on the group filter. More specifically in (n *groupNode) simpleExplain().

Good catch, will do

@jsimnz jsimnz force-pushed the jsimnz/fix/I1539-I1540-filter-splitting branch from 4312f46 to b58be38 Compare May 26, 2023 19:24
@jsimnz jsimnz requested review from fredcarle and shahzadlone May 26, 2023 20:42
Copy link
Collaborator

@fredcarle fredcarle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks John!

Copy link
Member

@shahzadlone shahzadlone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some todos. In addition I would add the explain tests you are adding to a new file as the test files are split according to different patterns, the patterns of tests you added are more similar to the type join tests, so make a file called ".../default/type_join_with_filter_test.go" and add these two there.

All the todos and the above suggestion have been done here in this commit, feel free to copy: shahzadlone@e57dff5

tests/integration/explain/default/with_filter_test.go Outdated Show resolved Hide resolved
tests/integration/explain/default/with_filter_test.go Outdated Show resolved Hide resolved
tests/integration/explain/default/with_filter_test.go Outdated Show resolved Hide resolved
tests/integration/explain/default/with_filter_test.go Outdated Show resolved Hide resolved
tests/integration/explain/default/with_filter_test.go Outdated Show resolved Hide resolved
tests/integration/explain/default/with_filter_test.go Outdated Show resolved Hide resolved
@jsimnz jsimnz force-pushed the jsimnz/fix/I1539-I1540-filter-splitting branch from e461607 to 38eb4ed Compare May 29, 2023 12:32
Copy link
Member

@shahzadlone shahzadlone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing the suggestions :)

childExplainGraph[filterLabel] = nil
} else {
childExplainGraph[filterLabel] = c.Filter.ExternalConditions
childExplainGraph[filterLabel] = c.Filter.ToMap(n.documentMapping)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Does this need a new test as well?

@jsimnz jsimnz merged commit 9b01928 into develop May 29, 2023
@jsimnz jsimnz deleted the jsimnz/fix/I1539-I1540-filter-splitting branch May 29, 2023 12:45
shahzadlone pushed a commit to shahzadlone/defradb that referenced this pull request Feb 23, 2024
…1541)

Fixes the filter splitting of related types issue. This issue was also
directly related to the rendering of these related filters within the
explain requests, so this PR covers both.

fixes:
- Filter splitting
- New filter rendering utilities
- Aggregate/group explain filter rendering
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
action/no-benchmark Skips the action that runs the benchmark. area/planner Related to the planner system bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Filter splitting between scan and select nodes doesn't handle multiple related types
3 participants