Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-1853] [Bug] When a dependency package is missing the dbt_project.yml, dbt deps in the main project fails with a non-descript error #6663

Closed
2 tasks done
Mathyoub opened this issue Jan 19, 2023 · 3 comments · Fixed by #6685
Labels
bug Something isn't working

Comments

@Mathyoub
Copy link
Contributor

Mathyoub commented Jan 19, 2023

Is this a new bug in dbt-core?

  • I believe this is a new bug in dbt-core
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Main dbt project A calls dependency package B.
Dependency B is not in the package hub, but rather in a customer hosted repo retrieved by git.

When the job runs dbt deps, it fails with a strange error message:

[2022-11-08 00:22:21.759993] ERROR: dbt_cloud: Runtime Error
  no dbt_project.yml found at expected path /tmp/dbt-downloads-eh3_yove/c1592655e2ed8fe15c54c562a0934786/dbt_project.yml

Except that Project A actually has a dbt_project.yml and otherwise runs fine in the IDE.

Further research discovers that Package B is actually missing the dbt_project.yml, and that’s what the error message is actually referring too: a missing .yml in a dependency, not the main project.

Fixing this missing .yml fixes the problem.

However, this error message makes it look more like a system failure (/tmp path) than a user-inflicted error.

Expected Behavior

dbt deps fails, but with an error message stating which dependency (in this case a package URL) has the problem.

Steps To Reproduce

See current behavior

Relevant log output

Here’s the traceback:

Traceback (most recent call last):
  File "/usr/src/app/sinter/clients/dbt.py", line 1396, in call
    dbt_main.handle(command + extra_args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 154, in handle
    res, success = handle_and_check(args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 191, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 238, in run_from_args
    results = task.run()
  File "/usr/local/lib/python3.8/dist-packages/dbt/task/deps.py", line 56, in run
    final_deps = resolve_packages(packages, self.config)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/resolver.py", line 131, in resolve_packages
    target = final[package].resolved().fetch_metadata(config, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/base.py", line 86, in fetch_metadata
    self._cached_metadata = self._fetch_metadata(project, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/git.py", line 103, in _fetch_metadata
    loaded = Project.from_project_root(path, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/config/project.py", line 652, in from_project_root
    partial = cls.partial_load(project_root, verify_version=verify_version)
  File "/usr/local/lib/python3.8/dist-packages/dbt/config/project.py", line 639, in partial_load
    return PartialProject.from_project_root(
  File "/usr/local/lib/python3.8/dist-packages/dbt/config/project.py", line 482, in from_project_root
    project_dict = _raw_project_from(project_root)
  File "/usr/local/lib/python3.8/dist-packages/dbt/config/project.py", line 166, in _raw_project_from
    raise DbtProjectError(
dbt.exceptions.DbtProjectError: Runtime Error
  no dbt_project.yml found at expected path /tmp/dbt-downloads-eh3_yove/c1592655e2ed8fe15c54c562a0934786/dbt_project.yml

Environment

- OS:
- Python:
- dbt:

Which database adapter are you using with dbt?

No response

Additional Context

Workaround: Examine all called deps for a missing dbt_project.yml

@Mathyoub Mathyoub added bug Something isn't working triage labels Jan 19, 2023
@github-actions github-actions bot changed the title [Bug] When a dependency package is missing the dbt_project.yml, dbt deps in the main project fails with a non-descript error [CT-1853] [Bug] When a dependency package is missing the dbt_project.yml, dbt deps in the main project fails with a non-descript error Jan 19, 2023
@dbeatty10 dbeatty10 self-assigned this Jan 20, 2023
@dbeatty10
Copy link
Contributor

Thanks for opening @Mathyoub !

Initial assessment

After an initial reading, it doesn't look like dbt is behaving erroneously, so this actually doesn't feel like a bug to me.

However, I can see how the error message might not have enough specificity in some cases. e.g., it might not be clear to the end user that one of the packages listed within packages.yml (or one of its transitive dependencies!) is missing dbt_project.yml.

Is there a simple solution?

Unfortunately, I don't see a simple way of just updating the error message to say something like "no dbt_project.yml found for package XYZ".

This is because of a few things:

Options moving forward

  1. One option is to make no changes. While not trivial to troubleshoot currently, it also doesn't feel impossible.
  2. Another option is doing a rather invasive refactor in order to inject more specific context of the offending "package" into the error message. This feels overkill in this case.
  3. A final option is updating the current generic message to contain just a bit more guidance of what the user can do to troubleshoot this effectively on their own. I think we should go this route!

Current error message:

dbt.exceptions.DbtProjectError: Runtime Error
  no dbt_project.yml found at expected path /tmp/dbt-downloads-eh3_yove/c1592655e2ed8fe15c54c562a0934786/dbt_project.yml

Proposed error message:

dbt.exceptions.DbtProjectError: Runtime Error
  no dbt_project.yml found at expected path /tmp/dbt-downloads-eh3_yove/c1592655e2ed8fe15c54c562a0934786/dbt_project.yml
  verify that each entry within `packages.yml` (and each their transitive dependencies) contains a file named dbt_project.yml

Can you think of any better ways to phrase this error message in a similarly generic fashion @Mathyoub ?

@dbeatty10 dbeatty10 removed their assignment Jan 20, 2023
@Mathyoub
Copy link
Contributor Author

@dbeatty10 I think that suggested generic error message makes sense! With that being said though I think 1 or 3 are fine options because this seems relatively low priority. Although if 3 is as easy as it seems then maybe we just go ahead and update it?

@dbeatty10
Copy link
Contributor

if 3 is as easy as it seems then maybe we just go ahead and update it?

I put up a PR to update it here! It's marked as "ready for review", but we can still theoretically make updates to the error message if we want.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants