Disabled Models in schema files #5868

emmyoop · 2022-09-16T15:33:54Z

resolves #3992

Description

throw correct exception for models disabled in schema files
fix bug when chained disabled models incorrectly throw exception
Add tests

Checklist

I have read the contributing guide and understand what's expected of me
I have signed the CLA
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
I have run changie new to create a changelog entry

github-actions · 2022-09-16T15:34:15Z

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

emmyoop · 2022-09-16T16:08:38Z

This feels a little hacky but does resolve the exceptions thrown to be correct. I opened #5869 to explore the possible underlying issue.

gshank

I think this needs a few changes, as described in the comments.

core/dbt/parser/schemas.py

gshank · 2022-09-19T21:26:05Z

core/dbt/parser/schemas.py

+                    # If this yaml file is enabled but the project config is not, we need to move
+                    # the node from disabled to manifest.nodes
+                    if patch.config.get("enabled"):
+                        test_from = {"key": block.target.yaml_key, "name": block.target.name}


The "test_from" variation on add_node is only for tests, not regular nodes. I don't think we should need to do "add_node" plus "remove_node'. I would just do something like popping it from "disabled" and adding it to "nodes". Also we can't really do this if there are multiple disabled nodes, since we wouldn't know which one to enable. So I think we'll have to limit this to cases where there's only one disabled node with this unique_id. You also shouldn't need to do a ref_lookup to get the unique_id, the node will already have a unique_id in it. So there would be an if/else after "if patch.config.get("enabled") with len(found_nodes) == 1, plus throwing an error if len is more than 1.

I think it might be possible to just add the node to ref_lookup by manifest.ref_lookup().add_node(node) too. Also need to remove the node from disabled_lookup. Could either rebuild it or add a function to remove node. Probably simpler to rebuild it.

core/dbt/parser/schemas.py

gshank

On thinking about this more, I think we're not handling some other cases. The precedence order is 1) dbt_project.yml config, 2) schema yaml config, 2) model config. So in theory the config in the model file could override the schema yaml config, and we have no code to move around the disabled/not disabled nodes for that case.

I really think we should switch to not having a separate dictionary for disabled, but that's probably for a later release.

For now I think that it might be best to always apply the patches to disabled nodes, except for the case where enabled is set to True and there is more than 1 matching disabled node (I kind of wish we never allowed that...) where we raise an error. Then when schema parsing is done, before refs are resolved, loop through nodes and disabled and make sure the nodes are in the right dictionary.

In addition, I think there's a hole in the "add_disabled" code that we didn't handle in the ticket for disabling metrics and exposures. It needs to be updated to check for test nodes (which used to be the only nodes from a SchemaSourceFile that could be updated), because the "test_from" piece only applies to tests.

Can you think of any additional holes?

core/dbt/parser/schemas.py

gshank

A comment from last time seems to have gotten lost. The processing in parse_patch looks okay, but I think the 'process_nodes' code still has issues.

gshank · 2022-09-28T19:56:36Z

core/dbt/parser/manifest.py

+                if node.config.enabled:
+                    for dis_index, dis_node in enumerate(disabled):
+                        # Remove node from disabled and unique_id from disabled dict if necessary
+                        enable_nodes[dis_index] = dis_node


I commented on this last time, but it looks like github lost it somehow... Making the index the key in the 'enable_nodes' dictionary means that if you have multiple enabled nodes at the same index, the first one will be overwritten.

I'm not sure why you are saving these in separate structures. It seems like for both looping through the nodes and looping through the disabled you could just move them when encountered.

@gshank I'm saving them as a separate structure because python doesn't like when you modify the dict you are looping through.

disabled on line 947 is the list of nodes for the unique id. Probably worth renaming if it's confusing. So the index is just the index of the list item. There can't be multiple nodes at a single index since each index represents a single node.. Now I see it. Will fix it.

emmyoop · 2022-09-29T01:13:05Z

core/dbt/parser/schemas.py

+                    # There are multiple disabled nodes for this model and the schema file wants to enable one.
+                    # We have no way to know which one to enable.
+                    msg = (
+                        f"Found {len(found_nodes)} matching disabled nodes for '{patch.name}'. "
+                        "Multiple nodes for the same unique id cannot be disabled in the schema "
+                        "file. They must be disabled in `dbt_project.yml` or in the sql files."
+                    )
+                    raise ParsingException(msg)


@jtcohen6 can I get some input on the error message here, please?

Thanks for tagging me in — this looks mostly good to me!

Could we include the resource type? Is that available from the patch?

How about:

Found {len(found_nodes)} matching disabled nodes for {patch.resource_type} '{patch.name}'. Multiple nodes with the same unique_id cannot be disabled in yaml resource properties. If you need to have multiple disabled nodes with the same names, you should instead disable them using in-file config, or resource-path config in `dbt_project.yml`.

core/dbt/parser/schemas.py

gshank · 2022-09-29T15:36:27Z

core/dbt/parser/manifest.py

+        # make sure the nodes are in the manifest.nodes or the disabled dict,
+        # correctly now that the schema files are also parsed
+        disable_node_copy = deepcopy(self.manifest.nodes)
+        for node in disable_node_copy.values():


I'm a little concerned by the overhead of deepcopying the entire nodes dictionary. Maybe we could have a list of disabled unique_ids, leave the 'add_disabled_nofile' in place, and just remove the disabled nodes afterward? I'm not so concerned by the disabled dictionary because it will be much smaller.

Oh, that's cleaner than creating a list of all nodes. 👍

* clean up debugging * reword some comments * changelog * add more tests * move around the manifest.node * fix typos * all tests passing * move logic for moving around nodes * add tests * more cleanup * fix failing pp test * remove comments * add more tests, patch all disabled nodes * fix test for windows * fix node processing to not overwrite enabled nodes * add checking disabled in pp, fix error msg * stop deepcopying all nodes when processing * update error message (cherry picked from commit fa4f9d3)

* clean up debugging * reword some comments * changelog * add more tests * move around the manifest.node * fix typos * all tests passing * move logic for moving around nodes * add tests * more cleanup * fix failing pp test * remove comments * add more tests, patch all disabled nodes * fix test for windows * fix node processing to not overwrite enabled nodes * add checking disabled in pp, fix error msg * stop deepcopying all nodes when processing * update error message (cherry picked from commit fa4f9d3) Co-authored-by: Emily Rockman <[email protected]>

cla-bot bot added the cla:yes label Sep 16, 2022

emmyoop mentioned this pull request Sep 16, 2022

[CT-1197] [Bug] Schema Files not parsed for models at the right time #5869

Closed

2 tasks

emmyoop marked this pull request as ready for review September 16, 2022 16:08

emmyoop requested review from a team as code owners September 16, 2022 16:08

emmyoop requested review from nathaniel-may, VersusFacit and gshank and removed request for nathaniel-may and VersusFacit September 16, 2022 16:08

gshank requested changes Sep 19, 2022

View reviewed changes

gshank reviewed Sep 20, 2022

View reviewed changes

emmyoop added 11 commits September 23, 2022 13:31

WIP

1a68f8a

WIP

4702502

clean up debugging

dfd7af8

reword some comments

21c9d03

changelog

5a98b26

add more tests

61268a1

move around the manifest.node

a750db7

fix typos

3a4bf39

WIP

b6ed9b8

still WIP

1bc9c0f

all tests passing

e464f5d

emmyoop force-pushed the er/ct-232-model-schema-configs branch from bf44806 to e464f5d Compare September 26, 2022 15:49

emmyoop added 4 commits September 26, 2022 13:38

move logic for moving around nodes

42bfdf7

add tests

c5ee79e

more cleanup

1cbb960

fix failing pp test

f4d765b

emmyoop requested a review from gshank September 26, 2022 21:21

remove comments

931c603

gshank requested changes Sep 27, 2022

View reviewed changes

core/dbt/parser/schemas.py Outdated Show resolved Hide resolved

emmyoop added 2 commits September 28, 2022 12:17

add mroe tests, patch all disabled nodes

2374de1

fix test for windows

834e825

emmyoop requested a review from gshank September 28, 2022 18:55

gshank requested changes Sep 28, 2022

View reviewed changes

fix node processing to not overwrite enabled nodes

e9f88aa

emmyoop requested a review from gshank September 29, 2022 01:09

emmyoop commented Sep 29, 2022

View reviewed changes

gshank requested changes Sep 29, 2022

View reviewed changes

add checking disabled in pp, fix error msg

5789d64

gshank reviewed Sep 29, 2022

View reviewed changes

stop deepcopying all nodes when processing

b2a6156

gshank approved these changes Sep 29, 2022

View reviewed changes

update error message

78c58af

emmyoop merged commit fa4f9d3 into main Sep 29, 2022

emmyoop deleted the er/ct-232-model-schema-configs branch September 29, 2022 20:24

leahwicz added the backport 1.3.latest label Oct 3, 2022

github-actions bot mentioned this pull request Oct 3, 2022

[Backport 1.3.latest] Disabled Models in schema files #5984

Merged

aranke mentioned this pull request Jul 19, 2023

Hotfix for INC-372: Use JSONEncoder in json.dumps #8149

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disabled Models in schema files #5868

Disabled Models in schema files #5868

emmyoop commented Sep 16, 2022 •

edited

Loading

github-actions bot commented Sep 16, 2022

emmyoop commented Sep 16, 2022

gshank left a comment

gshank Sep 19, 2022

gshank left a comment

gshank left a comment

gshank Sep 28, 2022

emmyoop Sep 28, 2022 •

edited

Loading

emmyoop Sep 29, 2022

jtcohen6 Sep 29, 2022

gshank Sep 29, 2022

emmyoop Sep 29, 2022

Disabled Models in schema files #5868

Disabled Models in schema files #5868

Conversation

emmyoop commented Sep 16, 2022 • edited Loading

Description

Checklist

github-actions bot commented Sep 16, 2022

emmyoop commented Sep 16, 2022

gshank left a comment

Choose a reason for hiding this comment

gshank Sep 19, 2022

Choose a reason for hiding this comment

gshank left a comment

Choose a reason for hiding this comment

gshank left a comment

Choose a reason for hiding this comment

gshank Sep 28, 2022

Choose a reason for hiding this comment

emmyoop Sep 28, 2022 • edited Loading

Choose a reason for hiding this comment

emmyoop Sep 29, 2022

Choose a reason for hiding this comment

jtcohen6 Sep 29, 2022

Choose a reason for hiding this comment

gshank Sep 29, 2022

Choose a reason for hiding this comment

emmyoop Sep 29, 2022

Choose a reason for hiding this comment

emmyoop commented Sep 16, 2022 •

edited

Loading

emmyoop Sep 28, 2022 •

edited

Loading