Ambiguity in --models flag fails fails to select models correctly

[drewbanin]

Issue

Issue description

The --models selection syntax will prefer package-based selections over path-based selections, whereas it should instead support both. This model selection code uses an if/else if/else, but it should instead be three distinct if statements, I think. In particular, the nested if is_selected_node() branch inside of the package branch "steals" the opportunity for selection away from the else block.

Existing code:

        if len(qualified_name) == 1 and fqn_ish[-1] == qualified_name[0]:
            yield node

        elif qualified_name[0] in package_names:
            if is_selected_node(fqn_ish, qualified_name):
                yield node

        else:
            for package_name in package_names:
                local_qualified_node_name = [package_name] + qualified_name
                if is_selected_node(fqn_ish, local_qualified_node_name):
                    yield node
                    break

Suggested changes (though other implementations will work and might be smarter):

        if len(qualified_name) == 1 and fqn_ish[-1] == qualified_name[0]:
            yield node

        if qualified_name[0] in package_names:
            if is_selected_node(fqn_ish, qualified_name):
                yield node

        for package_name in package_names:
            local_qualified_node_name = [package_name] + qualified_name
            if is_selected_node(fqn_ish, local_qualified_node_name):
                yield node
                break

I think the original if/else if/else implementation was meant as an optimization to short-circuit selection checks that we knew wouldn't match, but it seems like that is not the case.

Results

Models are not selected when they should be.

System information

0.11.1

Steps to reproduce

Given a project file:

# dbt_project.yml

name: my_project

And a model:

-- models/my_project/some_model.sql

select 1 as id

Running:

dbt run --model my_project.some_model

will not select any nodes. The my_project in the selector matches a package name, so dbt thinks the selection a fully qualified name for a model. Instead, it's a "locally" qualified name. You can make this work by doing:

dbt run --model my_project.my_project.some_model

which is the fully-qualified name for this model.

[beckjake]

Is there perhaps something more to this issue? I've set up a project exactly as described in steps to reproduce (ok, I added a version and a profile to the project file, but that shouldn't matter), and can't reproduce this failure on 0.11.1 or the most recent revision of dev/guion-bluford.

[drewbanin]

Meep! Sorry for the inaccurate writeup @beckjake. This actually happens in a slightly different scenario. Given:

models/
└── my_project
    └── subdir
        └── some_model.sql

This works:

dbt run --model my_project.my_project.subdir

This does not work:

dbt run --model my_project.subdir

I think the matching {project_name}.{model_name} is a special case in the code. That why

dbt run --model my_project.some_model

worked in the example above.

[GenDemo]

Having a similar problem:
My main project/workflow is as follows:
models
└─ base
└─ staging
└─ marts

Over time this DAG has become very convoluted. I am therefore rebuilding the project parallel to this one in a folder:
v2_models
└─ 1_base
└─ 2_staging
└─ 3_intermediery
└─ 4_marts

I want the original project to keep running as usual, but not run the new one I am building as part of the main schedule.
I concluded that I should use a select/exclude parameter in the dbt run command. This is where I struggles with this.
Following the documentation (to the best of my understanding), I tried all the below for the new "v2" models, all with errors:

• dbt run v2_models
• dbt run --select v2_models
• dbt run --select v2_models/.*
• dbt run --select v2_models.*
• dbt run --select project_name.v2_models.*
• dbt run --select project_name.v2_models
• dbt run --select v2_models.v2_models.*
• dbt run --select v2_models.v2_models
• dbt run --select v2_models.v2_models+
• dbt run --m v2_models.v2_models
• dbt run --m v2_models
• dbt run --m v2_models+
• dbt run --m v2_models.v2_models
• dbt run --m v2_models.4_marts
• dbt run --m +v2_models.4_marts
• dbt run --m +v2_models.4_marts.*
• dbt run --m +v2_models.4_marts.*

in the end, this worked:
dbt run --m +4_marts.*

Likewise I tried all the below for the old/existing model command, all with errors:

• dbt run --exclude v2_models
• dbt run --exclude v2_models.*
• dbt run --m --exclude v2_models
• dbt run --select models
• dbt run --select models/.*
• dbt run --select models.*
• dbt run --exclude path:models/v2_models

in the end this worked.
dbt run --m +marts.*

I am probably not asking anything - merely stating a difficult I encountered.
Perhaps someone can shed light why some of these options worked? perhaps I just interpreted the documentation incorrectly.
I gather from this, that you cant have two folders with the same names? I find the two command that ended up working in the end to be ... 'not ideal'?

I'll ask on the Slack community as well.