[CT-2457] [Bug] dbt deps commands error our in databricks DBFS #7428
Labels
bug
Something isn't working
dbfs
Issues that only occur when dbt is running on the Databricks file system
deps
dbt's package manager
file_system
How dbt-core interoperates with file systems to read/write data
help_wanted
Trickier changes, with a clear starting point, good for previous/experienced contributors
Is this a new bug in dbt-core?
Current Behavior
dbt deps
command fails to run in databricks DBFS.deps
command internally creating a symlink while running dbt deps command. After more debugging, I found dbt internally creating a symlink when installingdbt deps.
In current code the
supports_symlinks
is based on if pythonlib(os)
has thesymlink
method available or not. https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/clients/system.py#L161As DBT is running on 3.7+ Python we can get rid of
supports_symlinks
method. This will be always true in databricks DBFS becausegetattr(os, "symlink", None)
will never return False with the python versions dbt supportingDBFS can not work with symlink so we should use copytree.
https://stackoverflow.com/questions/64314341/creating-symbolic-link-in-a-databricks-filestores-folder
Expected Behavior
Steps To Reproduce
Relevant log output
No response
Environment
Which database adapter are you using with dbt?
other (mention it in "Additional Context")
Additional Context
No response
The text was updated successfully, but these errors were encountered: