Full support for dbt deps on Windows #778

drewbanin · 2018-05-22T17:50:24Z

OSError: [WinError 17] The system cannot move the file to a different disk drive: 'C:\Users\UserName\AppData\Local\Temp\dbt-downloads\75cce4837e0fd73e2934f79b9a5238a9' -> 'dbt_modules\snowplow'

This can happen on Windows if the temp dir is on a different drive than the dbt project dir

ryanmcdonnell · 2018-05-22T18:00:56Z

Windows 10 Pro version 1803
Python v3.6.3
dbt 0.10.1

packages.yml:

packages:
  - git: "https://github.com/fishtown-analytics/snowplow.git"
    revision: 0.3.8
  - git: "https://github.com/fishtown-analytics/dbt-utils.git"
    revision: 0.1.7
  - git: "https://github.com/fishtown-analytics/bing-ads.git"
    revision: 0.0.1

dbt.log:

2018-05-22 10:45:21,096: Tracking: tracking
2018-05-22 10:45:21,096: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001C92DD12A90>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001C92DD12FD0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001C92DD12C88>]}
2018-05-22 10:45:21,489: Executing "git clone --depth 1 https://github.com/fishtown-analytics/snowplow.git 75cce4837e0fd73e2934f79b9a5238a9"
2018-05-22 10:45:22,609: STDOUT: "b''"
2018-05-22 10:45:22,609: STDERR: "b"Cloning into '75cce4837e0fd73e2934f79b9a5238a9'...\n""
2018-05-22 10:45:22,609: Pulling new dependency 75cce4837e0fd73e2934f79b9a5238a9.
2018-05-22 10:45:22,610: Executing "git rev-parse HEAD"
2018-05-22 10:45:22,641: STDOUT: "b'f77428b270d28c31c776ecd41b5825c96b1521e0\n'"
2018-05-22 10:45:22,641: STDERR: "b''"
2018-05-22 10:45:22,641:   Checking out branch 0.3.8.
2018-05-22 10:45:22,641: Executing "git remote set-branches origin 0.3.8"
2018-05-22 10:45:22,680: STDOUT: "b''"
2018-05-22 10:45:22,680: STDERR: "b''"
2018-05-22 10:45:22,680: Executing "git fetch --tags --depth 1 origin 0.3.8"
2018-05-22 10:45:23,394: STDOUT: "b''"
2018-05-22 10:45:23,394: STDERR: "b'From https://github.com/fishtown-analytics/snowplow\n * tag               0.3.8      -> FETCH_HEAD\n * [new tag]         0.1.0      -> 0.1.0\n * [new tag]         0.2.0      -> 0.2.0\n * [new tag]         0.2.1      -> 0.2.1\n * [new tag]         0.2.2      -> 0.2.2\n * [new tag]         0.2.3      -> 0.2.3\n * [new tag]         0.3.0      -> 0.3.0\n * [new tag]         0.3.1      -> 0.3.1\n * [new tag]         0.3.2      -> 0.3.2\n * [new tag]         0.3.3      -> 0.3.3\n * [new tag]         0.3.4      -> 0.3.4\n * [new tag]         0.3.5      -> 0.3.5\n * [new tag]         0.3.6      -> 0.3.6\n * [new tag]         0.3.7      -> 0.3.7\n * [new tag]         0.3.8      -> 0.3.8\n * [new tag]         0.3.9      -> 0.3.9\n * [new tag]         0.4.0      -> 0.4.0\n'"
2018-05-22 10:45:23,394: Executing "git tag --list"
2018-05-22 10:45:23,432: STDOUT: "b'0.1.0\n0.2.0\n0.2.1\n0.2.2\n0.2.3\n0.3.0\n0.3.1\n0.3.2\n0.3.3\n0.3.4\n0.3.5\n0.3.6\n0.3.7\n0.3.8\n0.3.9\n0.4.0\n'"
2018-05-22 10:45:23,432: STDERR: "b''"
2018-05-22 10:45:23,432: Executing "git reset --hard tags/0.3.8"
2018-05-22 10:45:23,530: STDOUT: "b'HEAD is now at 1499581 Merge pull request #18 from fishtown-analytics/feature/fivetran\n'"
2018-05-22 10:45:23,531: STDERR: "b''"
2018-05-22 10:45:23,531: Executing "git rev-parse HEAD"
2018-05-22 10:45:23,562: STDOUT: "b'14995815b385568d834db7cd496865709a0501da\n'"
2018-05-22 10:45:23,562: STDERR: "b''"
2018-05-22 10:45:23,562:   Checked out at 1499581.
2018-05-22 10:45:23,574: Executing "git clone --depth 1 https://github.com/fishtown-analytics/dbt-utils.git 859fd0fa21176efee375a447937f7a46"
2018-05-22 10:45:24,426: STDOUT: "b''"
2018-05-22 10:45:24,426: STDERR: "b"Cloning into '859fd0fa21176efee375a447937f7a46'...\n""
2018-05-22 10:45:24,426: Pulling new dependency 859fd0fa21176efee375a447937f7a46.
2018-05-22 10:45:24,426: Executing "git rev-parse HEAD"
2018-05-22 10:45:24,457: STDOUT: "b'6117edf2720b38893f7520e016c9a1a54e88c6b0\n'"
2018-05-22 10:45:24,457: STDERR: "b''"
2018-05-22 10:45:24,457:   Checking out branch 0.1.7.
2018-05-22 10:45:24,457: Executing "git remote set-branches origin 0.1.7"
2018-05-22 10:45:24,498: STDOUT: "b''"
2018-05-22 10:45:24,498: STDERR: "b''"
2018-05-22 10:45:24,498: Executing "git fetch --tags --depth 1 origin 0.1.7"
2018-05-22 10:45:25,704: STDOUT: "b''"
2018-05-22 10:45:25,704: STDERR: "b'From https://github.com/fishtown-analytics/dbt-utils\n * tag               0.1.7      -> FETCH_HEAD\n * [new tag]         0.0.1      -> 0.0.1\n * [new tag]         0.1.0      -> 0.1.0\n * [new tag]         0.1.1      -> 0.1.1\n * [new tag]         0.1.2      -> 0.1.2\n * [new tag]         0.1.3      -> 0.1.3\n * [new tag]         0.1.4      -> 0.1.4\n * [new tag]         0.1.5      -> 0.1.5\n * [new tag]         0.1.6      -> 0.1.6\n * [new tag]         0.1.7      -> 0.1.7\n * [new tag]         0.1.8      -> 0.1.8\n * [new tag]         0.1.9      -> 0.1.9\n'"
2018-05-22 10:45:25,704: Executing "git tag --list"
2018-05-22 10:45:25,742: STDOUT: "b'0.0.1\n0.1.0\n0.1.1\n0.1.10\n0.1.2\n0.1.3\n0.1.4\n0.1.5\n0.1.6\n0.1.7\n0.1.8\n0.1.9\n'"
2018-05-22 10:45:25,742: STDERR: "b''"
2018-05-22 10:45:25,742: Executing "git reset --hard tags/0.1.7"
2018-05-22 10:45:25,872: STDOUT: "b'HEAD is now at 331df26 Merge pull request #54 from fishtown-analytics/feature/last-day\n'"
2018-05-22 10:45:25,872: STDERR: "b''"
2018-05-22 10:45:25,872: Executing "git rev-parse HEAD"
2018-05-22 10:45:25,903: STDOUT: "b'331df26aa543807a8721f243835e0bbeaaf931e5\n'"
2018-05-22 10:45:25,904: STDERR: "b''"
2018-05-22 10:45:25,904:   Checked out at 331df26.
2018-05-22 10:45:25,914: Executing "git clone --depth 1 https://github.com/fishtown-analytics/bing-ads.git 3ef5c85fd464f489424563cc23940fed"
2018-05-22 10:45:26,671: STDOUT: "b''"
2018-05-22 10:45:26,671: STDERR: "b"Cloning into '3ef5c85fd464f489424563cc23940fed'...\n""
2018-05-22 10:45:26,671: Pulling new dependency 3ef5c85fd464f489424563cc23940fed.
2018-05-22 10:45:26,671: Executing "git rev-parse HEAD"
2018-05-22 10:45:26,703: STDOUT: "b'675c0adb5d9f7085c34f1af8c10751cc2628ad4a\n'"
2018-05-22 10:45:26,703: STDERR: "b''"
2018-05-22 10:45:26,703:   Checking out branch 0.0.1.
2018-05-22 10:45:26,703: Executing "git remote set-branches origin 0.0.1"
2018-05-22 10:45:26,745: STDOUT: "b''"
2018-05-22 10:45:26,745: STDERR: "b''"
2018-05-22 10:45:26,745: Executing "git fetch --tags --depth 1 origin 0.0.1"
2018-05-22 10:45:27,328: STDOUT: "b''"
2018-05-22 10:45:27,328: STDERR: "b'From https://github.com/fishtown-analytics/bing-ads\n * tag               0.0.1      -> FETCH_HEAD\n'"
2018-05-22 10:45:27,329: Executing "git tag --list"
2018-05-22 10:45:27,363: STDOUT: "b'0.0.1\n'"
2018-05-22 10:45:27,363: STDERR: "b''"
2018-05-22 10:45:27,363: Executing "git reset --hard tags/0.0.1"
2018-05-22 10:45:27,425: STDOUT: "b'HEAD is now at 675c0ad updated data types for keyword performance\n'"
2018-05-22 10:45:27,425: STDERR: "b''"
2018-05-22 10:45:27,425: Executing "git rev-parse HEAD"
2018-05-22 10:45:27,457: STDOUT: "b'675c0adb5d9f7085c34f1af8c10751cc2628ad4a\n'"
2018-05-22 10:45:27,457: STDERR: "b''"
2018-05-22 10:45:27,457:   Checked out at 675c0ad.
2018-05-22 10:45:27,468: Installing https://github.com/fishtown-analytics/[email protected]
2018-05-22 10:45:27,469: Executing "git clone --depth 1 https://github.com/fishtown-analytics/snowplow.git 75cce4837e0fd73e2934f79b9a5238a9"
2018-05-22 10:45:27,499: STDOUT: "b''"
2018-05-22 10:45:27,499: STDERR: "b"fatal: destination path '75cce4837e0fd73e2934f79b9a5238a9' already exists and is not an empty directory.\n""
2018-05-22 10:45:27,499: Updating existing dependency 75cce4837e0fd73e2934f79b9a5238a9.
2018-05-22 10:45:27,499: Executing "git rev-parse HEAD"
2018-05-22 10:45:27,528: STDOUT: "b'14995815b385568d834db7cd496865709a0501da\n'"
2018-05-22 10:45:27,529: STDERR: "b''"
2018-05-22 10:45:27,529:   Checking out branch 0.3.8.
2018-05-22 10:45:27,529: Executing "git remote set-branches origin 0.3.8"
2018-05-22 10:45:27,568: STDOUT: "b''"
2018-05-22 10:45:27,568: STDERR: "b''"
2018-05-22 10:45:27,568: Executing "git fetch --tags --depth 1 origin 0.3.8"
2018-05-22 10:45:28,169: STDOUT: "b''"
2018-05-22 10:45:28,169: STDERR: "b'From https://github.com/fishtown-analytics/snowplow\n * tag               0.3.8      -> FETCH_HEAD\n'"
2018-05-22 10:45:28,169: Executing "git tag --list"
2018-05-22 10:45:28,207: STDOUT: "b'0.1.0\n0.2.0\n0.2.1\n0.2.2\n0.2.3\n0.3.0\n0.3.1\n0.3.2\n0.3.3\n0.3.4\n0.3.5\n0.3.6\n0.3.7\n0.3.8\n0.3.9\n0.4.0\n'"
2018-05-22 10:45:28,207: STDERR: "b''"
2018-05-22 10:45:28,207: Executing "git reset --hard tags/0.3.8"
2018-05-22 10:45:28,272: STDOUT: "b'HEAD is now at 1499581 Merge pull request #18 from fishtown-analytics/feature/fivetran\n'"
2018-05-22 10:45:28,272: STDERR: "b''"
2018-05-22 10:45:28,272: Executing "git rev-parse HEAD"
2018-05-22 10:45:28,303: STDOUT: "b'14995815b385568d834db7cd496865709a0501da\n'"
2018-05-22 10:45:28,303: STDERR: "b''"
2018-05-22 10:45:28,303:   Already at 1499581, nothing to do.
2018-05-22 10:45:28,733: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001C92DE227B8>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001C92DE22BA8>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001C92DE2BAC8>]}
2018-05-22 10:45:29,104: Encountered an error:
2018-05-22 10:45:29,105: [WinError 5] Access is denied: 'C:\\Users\\Ryan\\AppData\\Local\\Temp\\dbt-downloads\\75cce4837e0fd73e2934f79b9a5238a9\\.git\\objects\\05\\f1220c8c8d13e365deb6c38e54d3d9ef9e8c0c'
2018-05-22 10:45:29,110: Traceback (most recent call last):
  File "c:\program files\python36\lib\shutil.py", line 544, in move
    os.rename(src, real_dst)
OSError: [WinError 17] The system cannot move the file to a different disk drive: 'C:\\Users\\Ryan\\AppData\\Local\\Temp\\dbt-downloads\\75cce4837e0fd73e2934f79b9a5238a9' -> 'dbt_modules\\snowplow'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\program files\python36\lib\site-packages\dbt\main.py", line 41, in main
    results, succeeded = handle_and_check(args)
  File "c:\program files\python36\lib\site-packages\dbt\main.py", line 85, in handle_and_check
    task, res = run_from_args(parsed)
  File "c:\program files\python36\lib\site-packages\dbt\main.py", line 139, in run_from_args
    results = run_from_task(task, proj, parsed)
  File "c:\program files\python36\lib\site-packages\dbt\main.py", line 147, in run_from_task
    result = task.run()
  File "c:\program files\python36\lib\site-packages\dbt\task\deps.py", line 424, in run
    package.install(self.project)
  File "c:\program files\python36\lib\site-packages\dbt\task\deps.py", line 228, in install
    shutil.move(self._checkout(project), dest_path)
  File "c:\program files\python36\lib\shutil.py", line 556, in move
    rmtree(src)
  File "c:\program files\python36\lib\shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\program files\python36\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\program files\python36\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\program files\python36\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\program files\python36\lib\shutil.py", line 389, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "c:\program files\python36\lib\shutil.py", line 387, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\Ryan\\AppData\\Local\\Temp\\dbt-downloads\\75cce4837e0fd73e2934f79b9a5238a9\\.git\\objects\\05\\f1220c8c8d13e365deb6c38e54d3d9ef9e8c0c'

atharvai · 2018-06-05T21:52:41Z

I spoke with Drew and suggested a workaround being able to override the temp dir. Of course this is no fix or solution but general good idea of having it configurable

drewbanin · 2018-06-28T17:51:15Z

Combining issue #766

dbt deps - symlink fails on windows (symbolic link privilege not held)

Via slack:

2018-05-09 15:52:04,681: Tracking: do not track
2018-05-09 15:52:04,692: Installing ../
2018-05-09 15:52:04,693:   Creating symlink to local dependency.
2018-05-09 15:52:04,693: Encountered an error:
2018-05-09 15:52:04,693: symbolic link privilege not held
2018-05-09 15:52:04,695: Traceback (most recent call last):
  File "c:\users\bene\appdata\local\programs\python\python36-32\lib\site-packages\dbt\main.py", line 41, in main
    results, succeeded = handle_and_check(args)
  File "c:\users\bene\appdata\local\programs\python\python36-32\lib\site-packages\dbt\main.py", line 85, in handle_and_check
    task, res = run_from_args(parsed)
  File "c:\users\bene\appdata\local\programs\python\python36-32\lib\site-packages\dbt\main.py", line 139, in run_from_args
    results = run_from_task(task, proj, parsed)
  File "c:\users\bene\appdata\local\programs\python\python36-32\lib\site-packages\dbt\main.py", line 147, in run_from_task
    result = task.run()
  File "c:\users\bene\appdata\local\programs\python\python36-32\lib\site-packages\dbt\task\deps.py", line 424, in run
    package.install(self.project)
  File "c:\users\bene\appdata\local\programs\python\python36-32\lib\site-packages\dbt\task\deps.py", line 272, in install
    dbt.clients.system.make_symlink(src_path, dest_path)
  File "c:\users\bene\appdata\local\programs\python\python36-32\lib\site-packages\dbt\clients\system.py", line 106, in make_symlink
    return os.symlink(source, link_path)
OSError: symbolic link privilege not held

Windows 10 Pro (developer mode enabled, can call mklink without elevated privileges)
Using powershell
dbt 10.0.0
It works in an admin shell

We should either fix this, or add an option to not create a symlink

Via Ben Edwards in Slack:

some other research that I have done, python's os.symlink will never work in windows unless run by an elevated user, even if the regular user can create symlinks (by enabling developer mode). The check in system.py only checks if the os supports symlink, which windows does, you just cant use it. A possible workaround I've seen is to detect windows, start a shell and call mklink from there.

brian-bk · 2018-08-01T20:00:36Z

Could just set the TMPDIR env var

atharvai · 2018-10-19T16:39:46Z

Here are some investigation scenarios that show that if the TMP dir and dbt project dir are on the OS drive the dbt deps is successful but not if they are on different drives https://gist.github.com/atharvai/822ef904c29bbd304bac07bb37233dab

Fix dbt deps issues (#778 #994 #895)

beckjake · 2018-11-14T20:33:02Z

fixed in #1110

chapmanjacobd · 2021-11-19T16:33:13Z

I think this is the reason dbt has trouble installing deps on WSL2... is there a way to skip using the symlinks? when you only have 1 or 2 dbt projects it causes much much more headaches than the few MBs it saves

If dbt deps runs on the Linux side for some reason it's unable to access the files that it creates

If dbt deps is run in the Windows side then linux can't access the modules unless you copy the files outside the folder then delete the hardlink and reboot WSL then create a normal dbt_modules folder and copy the files back in. So there is a workaround if one is technically minded but for a new user this might be unknown. I feel like more people use WSL than the Windows CLI but I might be mistaken

drewbanin added bug Something isn't working enhancement New feature or request and removed bug Something isn't working labels Jun 12, 2018

drewbanin changed the title ~~Error running dbt deps on windows~~ Full support for dbt deps on Windows Jun 28, 2018

drewbanin mentioned this issue Jun 28, 2018

dbt deps - symlink fails on windows (symbolic link privilege not held) #766

Closed

drewbanin added the windows Everyone's favorite OS that's sometimes a little weird label Jun 28, 2018

drewbanin mentioned this issue Aug 1, 2018

Dbt downloads directory same across different users #895

Closed

jthandy added this to the Stephen Girard milestone Oct 25, 2018

drewbanin modified the milestones: Stephen Girard, Grace Kelly Nov 2, 2018

beckjake mentioned this issue Nov 6, 2018

Fix dbt deps issues (#778 #994 #895) #1110

Merged

beckjake added a commit that referenced this issue Nov 14, 2018

Merge pull request #1110 from fishtown-analytics/fix/dbt-deps-issues

735ff88

Fix dbt deps issues (#778 #994 #895)

beckjake closed this as completed Nov 14, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Full support for dbt deps on Windows #778

Full support for dbt deps on Windows #778

drewbanin commented May 22, 2018

ryanmcdonnell commented May 22, 2018

atharvai commented Jun 5, 2018

drewbanin commented Jun 28, 2018

brian-bk commented Aug 1, 2018

atharvai commented Oct 19, 2018

beckjake commented Nov 14, 2018

chapmanjacobd commented Nov 19, 2021 •

edited

Loading

Full support for dbt deps on Windows #778

Full support for dbt deps on Windows #778

Comments

drewbanin commented May 22, 2018

ryanmcdonnell commented May 22, 2018

atharvai commented Jun 5, 2018

drewbanin commented Jun 28, 2018

dbt deps - symlink fails on windows (symbolic link privilege not held)

brian-bk commented Aug 1, 2018

atharvai commented Oct 19, 2018

beckjake commented Nov 14, 2018

chapmanjacobd commented Nov 19, 2021 • edited Loading

chapmanjacobd commented Nov 19, 2021 •

edited

Loading