Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dvc pull: fails with PermissionError: [WinError 32] #9749

Open
AchimGaedkeLynker opened this issue Jul 20, 2023 · 23 comments
Open

dvc pull: fails with PermissionError: [WinError 32] #9749

AchimGaedkeLynker opened this issue Jul 20, 2023 · 23 comments
Labels
P: windows Related to the Platform: Windows product: VSCode Integration with VSCode extension

Comments

@AchimGaedkeLynker
Copy link

AchimGaedkeLynker commented Jul 20, 2023 •
edited
Loading

Bug Report

(Rehash of comments from #6403 (comment) and following).

Comment #9749 (comment) contains instructions to reproduce the WinError 32 problems from scratch.

Description

On Windows in a DVC project on the console while editing this project in VSCode.

(base) PS C:\Users\achimgaedke\Code\xxx> dvc pull
ERROR: unexpected error - [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\achimgaedke\\Code\\xxx\\train-out\\model_19.pth'

I would expect DVC to pull in my models from S3.

A successful pull looks like this:

(base) PS C:\Users\achimgaedke\Code\xxx> $Env:GIT_OPTIONAL_LOCKS=0
(base) PS C:\Users\achimgaedke\Code\xxx> dvc pull
M       train-out\
1 file modified
(base) PS C:\Users\achimgaedke\Code\xxx>

Reproduce

For me, it reproduces like this (intermittently though):

  • Start working on my project in VSCode
  • Delete the model files in train-out
  • Open the terminal
  • Run dvc pull

The 20ish files pulled are each 300MB. Later (once I have more time) I'll try to create a sharable project.
(Yeah, it is hard to report bugs from ongoing work).

Debug information

dvc_pull_-v.log

Note the line:

2023-07-20 14:39:20,177 DEBUG: failed to create 'C:\Users\achimgaedke\Code\xxx_c14_p113\train-out\model_4.pth' from 'C:\Users\achimgaedke\Code\xxx_c14_p113\.dvc\cache2\files\md5\c9\e8b9140bc7187529a5fcb33fe29702' - [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\achimgaedke\\Code\\xxx_c14_p113\\train-out\\.PLs2vrX8fXNan9LjzfiH3V.tmp' -> 'C:\\Users\\achimgaedke\\Code\\xxx_c14_p113\\train-out\\model_4.pth'

It looks like the creation of the file in the target directory train-out from the cache directory fails (i.e. the caching itself works).

  • I've disabled the virus scan and indexer for the Code directory (following the instructions at https://dvc.org/doc/user-guide/how-to/run-dvc-on-windows)
  • I am running VSCode with the git and dvc extensions - git status might interfere.
  • I've installed dvc, git and VSCode with scoop.
  • I have enabled long pathname and symlink support in windows (following the advice of scoop checkup).

Which process would be so nosy to pick up new file immediately? Maybe git status if this dir/file is not listed in gitignore?)

In my experience this error does not appear when setting $Env:GIT_OPTIONAL_LOCKS=0. (see https://git-scm.com/docs/git#Documentation/git.txt-codeGITOPTIONALLOCKScode).

Environment information

Output of dvc doctor:

dvc doctor
DVC version: 3.5.1
------------------
Platform: Python 3.10.11 on Windows-10-10.0.22621-SP0
Subprojects:

Supports:
        azure (adlfs = 2023.4.0, knack = 0.10.1, azure-identity = 1.13.0),
        gdrive (pydrive2 = 1.16.1),
        gs (gcsfs = 2023.6.0),
        hdfs (fsspec = 2023.6.0, pyarrow = 12.0.1),
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        oss (ossfs = 2021.8.0),
        s3 (s3fs = 2023.6.0, boto3 = 1.26.161),
        ssh (sshfs = 2023.4.1),
        webdav (webdav4 = 0.9.8),
        webdavs (webdav4 = 0.9.8),
        webhdfs (fsspec = 2023.6.0)
Config:
        Global: C:\Users\achimgaedke\AppData\Local\iterative\dvc
        System: C:\ProgramData\iterative\dvc
Cache types: hardlink, symlink
Cache directory: NTFS on C:\
Caches: local
Remotes: s3
Workspace directory: NTFS on C:\
Repo: dvc, git
Repo.site_cache_dir: C:\ProgramData\iterative\dvc\Cache\repo\a5786f5395aa8da08287147b62b67b3b
scoop list
Installed apps:

Name       Version      Source Updated             Info
----       -------      ------ -------             ----
7zip       23.01        main   2023-06-28 15:33:15     
aws        2.13.2       main   2023-07-20 08:59:51     
aws-vault  7.2.0        main   2023-06-23 09:17:31     
dark       3.11.2       main   2023-06-12 08:34:04
dbeaver    23.1.2       extras 2023-07-10 08:48:20
dvc        3.5.1        main   2023-07-12 07:30:56
firefox    115.0.2      extras 2023-07-12 07:30:03
gh         2.32.0       main   2023-07-12 13:12:19
git        2.41.0.3     main   2023-07-18 20:50:20
innounp    0.50         main   2023-07-07 09:19:53
keepass    2.54         extras 2023-06-20 10:34:13
mambaforge 23.1.0-4     extras 2023-07-17 08:10:19
sourcetree 3.4.14       extras 2023-06-29 08:06:09
sudo       0.2020.01.26 main   2023-07-11 07:30:39
terraform  1.5.3        main   2023-07-13 15:37:19
vscode     1.80.1       extras 2023-07-17 08:10:59
@AchimGaedkeLynker AchimGaedkeLynker changed the title dvc pull: fails with PermissionError: [WinError 32] The process cannot access the file because it is being used by another process dvc pull: fails with PermissionError: [WinError 32] Jul 20, 2023
@AchimGaedkeLynker
Copy link
Author

Just updated to version 3.6, debug output looks different in the details, result is the same...

dvc-3.6_pull_-v.log

@efiop efiop added the P: windows Related to the Platform: Windows label Jul 20, 2023
@efiop
Copy link
Contributor

efiop commented Jul 20, 2023

Looks like it downloaded it successfully, but it is checkout (pull actually equals fetch + checkout) that is failing.

I see that you are using symlink/hardlink, that might be what is triggering it. Try using copy, just so we can narrow it down. Also if your files are not that big, maybe copy would be acceptable for you overall, since it is safer than symlink/hardlink, esp on windows where it has more nuances.

@efiop efiop added the awaiting response we are waiting for your reply, please respond! :) label Jul 20, 2023
@AchimGaedkeLynker
Copy link
Author

AchimGaedkeLynker commented Jul 21, 2023 •
edited
Loading

as requested...

dvc config cache.type copy
dvc checkout --relink

and then deleted the directory and issued dvc pull -v.

dvc_pull_-v_copy_cache.log

The reporting about the cache type seems misleading or the config isn't read. The config file and dvc config cache.type both state copy.

Another way of avoiding this ordeal is to close VSCode and only run the commands on the terminal... Defeats the purpose of VSCode though....

@efiop
Copy link
Contributor

efiop commented Jul 21, 2023

@AchimGaedkeLynker Something is not right. Please show dvc config --list

@AchimGaedkeLynker
Copy link
Author 

> dvc config --list
core.analytics=false
remote.aws-storage.url=s3://xxxx/dvc-storage/
core.remote=aws-storage
core.autostage=true
cache.type=copy

@AchimGaedkeLynker
Copy link
Author

AchimGaedkeLynker commented Jul 23, 2023 •
edited
Loading

All right, I've created a new project to reproduce similar (or the same) trouble:

Copy the python program create_data_dir.py into an empty directory (e.g. dvc_stress_test):

import pathlib
import random
import shutil


def create_big_random_file(seed: int, path: pathlib.Path, mbsize: int):
    mbibyte = 1 << 20
    random_data = random.Random(seed).randbytes(mbibyte)
    with path.open(mode="wb") as f:
        for _ in range(mbsize):
            f.write(random_data)


data_dir = pathlib.Path("data")
shutil.rmtree(data_dir, ignore_errors=True)
data_dir.mkdir()

for file_idx in range(10):
    create_big_random_file(
        file_idx + 42, data_dir / f"big_random_file{file_idx:02d}", 1000
    )

and then execute:

git init
dvc init
dvc config cache.type copy
dvc checkout --relink
dvc stage add -n create_data -d create_data_dir.py -o data python create_data_dir.py
dvc repro -v

Do this:

  • in the PowerShell (outside VSCode) - works fine
  • using VSCode with the (git/DVC extensions installed) - this fails, the output is copied below:
PS C:\Users\achimgaedke\extCode\dvc_stress_test> dvc init
>> dvc config cache.type copy
>> dvc checkout --relink
>> 
>> dvc stage add -n create_data -d create_data_dir.py -o data python create_data_dir.py
>> dvc repro -v
Initialized DVC repository.

You can now commit the changes to git.

What's next?
------------
- Check out the documentation: <https://dvc.org/doc>
- Get help and share ideas: <https://dvc.org/chat>
- Star us on GitHub: <https://github.com/iterative/dvc>
Relinked successfully
Added stage 'create_data' in 'dvc.yaml'

To track the changes with git, run:

        git add dvc.yaml

To enable auto staging, run:

        dvc config core.autostage true
2023-07-23 21:35:55,624 DEBUG: v3.7.0, CPython 3.10.11 on Windows-10-10.0.22621-SP0
2023-07-23 21:35:55,625 DEBUG: command: repro -v
2023-07-23 21:35:56,018 DEBUG: Lockfile for 'dvc.yaml' not found
2023-07-23 21:35:56,028 DEBUG: Lockfile for 'dvc.yaml' not found
2023-07-23 21:35:56,152 DEBUG: Dependency 'create_data_dir.py' of stage: 'create_data' changed because it is 'modified'.
2023-07-23 21:35:56,154 DEBUG: stage: 'create_data' changed.
2023-07-23 21:35:56,157 DEBUG: Removing output 'data' of stage: 'create_data'.
2023-07-23 21:35:56,158 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data'
2023-07-23 21:35:56,165 DEBUG: {'create_data_dir.py': 'modified'}
Running stage 'create_data':
> python create_data_dir.py
2023-07-23 21:36:04,216 DEBUG: Lockfile for 'dvc.yaml' not found
2023-07-23 21:36:17,502 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-07-23 21:36:17,504 DEBUG: Computed stage: 'create_data' md5: 'f245734c5a8cc0588b72210046fab8da'
2023-07-23 21:36:17,526 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-07-23 21:36:17,529 DEBUG: Preparing to transfer data from 'memory://dvc-staging-md5/60dc2168a9f5c260e9d6f8ab6a80dee5e8805025a4f7d1335e0d78f177eb17a1' to 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-07-23 21:36:17,530 DEBUG: Preparing to collect status from 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-07-23 21:36:17,531 DEBUG: Collecting status from 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-07-23 21:36:17,538 DEBUG: Preparing to collect status from 'memory://dvc-staging-md5/60dc2168a9f5c260e9d6f8ab6a80dee5e8805025a4f7d1335e0d78f177eb17a1'
2023-07-23 21:36:17,539 DEBUG: transfer dir: md5: 625be7c4ce6f1264a616c17b0e648bdc.dir with 10 files
2023-07-23 21:36:17,556 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-07-23 21:36:17,563 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\.DRmFuaMqh6KghWvzF29XLz.tmp'
2023-07-23 21:36:17,564 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5\.cmZL9BuWGVHrxL6Rr2tyNz.tmp'
2023-07-23 21:36:17,567 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file04'                                            
2023-07-23 21:36:18,336 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file06'                                            
2023-07-23 21:36:19,146 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file09'                                            
2023-07-23 21:36:19,150 ERROR: failed to reproduce 'create_data': [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\achimgaedke\\extCode\\dvc_stress_test\\data\\big_random_file09'
Traceback (most recent call last):
  File "dvc_objects\fs\utils.py", line 89, in _unlink
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\achimgaedke\\extCode\\dvc_stress_test\\data\\big_random_file09'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dvc\repo\reproduce.py", line 199, in _reproduce
  File "dvc\repo\reproduce.py", line 129, in _reproduce_stage
  File "funcy\decorators.py", line 47, in wrapper
  File "dvc\stage\decorators.py", line 43, in rwlocked
  File "funcy\decorators.py", line 68, in __call__
  File "dvc\stage\__init__.py", line 433, in reproduce
  File "dvc\stage\decorators.py", line 43, in rwlocked
  File "funcy\decorators.py", line 68, in __call__
  File "dvc\stage\__init__.py", line 626, in run
  File "funcy\decorators.py", line 47, in wrapper
  File "dvc\stage\decorators.py", line 43, in rwlocked
  File "funcy\decorators.py", line 68, in __call__
  File "dvc\stage\__init__.py", line 549, in commit
  File "dvc\output.py", line 793, in commit
  File "dvc\output.py", line 752, in _checkout
  File "dvc_data\hashfile\checkout.py", line 276, in checkout
  File "dvc_data\hashfile\checkout.py", line 223, in _checkout
  File "dvc_data\hashfile\checkout.py", line 88, in _checkout_file
  File "dvc_data\hashfile\checkout.py", line 59, in _relink
  File "dvc_data\hashfile\checkout.py", line 55, in _remove
  File "dvc_objects\fs\base.py", line 470, in rm
  File "dvc_objects\fs\local.py", line 119, in rm
  File "dvc_objects\fs\utils.py", line 101, in remove
  File "dvc_objects\fs\utils.py", line 91, in _unlink
  File "dvc_objects\fs\utils.py", line 84, in _chmod
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\achimgaedke\\extCode\\dvc_stress_test\\data\\big_random_file09'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "dvc\cli\__init__.py", line 209, in main
  File "dvc\cli\command.py", line 26, in do_run
  File "dvc\commands\repro.py", line 13, in run
  File "dvc\repo\__init__.py", line 64, in wrapper
  File "dvc\repo\scm_context.py", line 151, in run
  File "dvc\repo\reproduce.py", line 260, in reproduce
  File "dvc\repo\reproduce.py", line 203, in _reproduce
  File "dvc\repo\reproduce.py", line 167, in _raise_error
dvc.exceptions.ReproductionError: failed to reproduce 'create_data'

2023-07-23 21:36:19,160 DEBUG: Analytics is disabled.
PS C:\Users\achimgaedke\extCode\dvc_stress_test>

Here some diagnostics

dvc config --list
core.analytics=false
cache.type=copy

dvc doctor
DVC version: 3.7.0
------------------
Platform: Python 3.10.11 on Windows-10-10.0.22621-SP0
Subprojects:

Supports:
        azure (adlfs = 2023.4.0, knack = 0.10.1, azure-identity = 1.13.0),
        gdrive (pydrive2 = 1.16.1),
        gs (gcsfs = 2023.6.0),
        hdfs (fsspec = 2023.6.0, pyarrow = 12.0.1),
        http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
        oss (ossfs = 2021.8.0),
        s3 (s3fs = 2023.6.0, boto3 = 1.26.161),
        ssh (sshfs = 2023.7.0),
        webdav (webdav4 = 0.9.8),
        webdavs (webdav4 = 0.9.8),
        webhdfs (fsspec = 2023.6.0)
Config:
        Global: C:\Users\achimgaedke\AppData\Local\iterative\dvc
        System: C:\ProgramData\iterative\dvc
Cache types: hardlink, symlink
Cache directory: NTFS on C:\
Caches: local
Remotes: None
Workspace directory: NTFS on C:\
Repo: dvc, git
Repo.site_cache_dir: C:\ProgramData\iterative\dvc\Cache\repo\c9f823bbd557bbd5b830efcb5e947c49

The harddrive has at the time 116GB free space (yeah, I cleared out some of my "real files" >>10GB).

@AchimGaedkeLynker
Copy link
Author

(ah, more trouble can be experienced with a repeat dvc repro or dvc destroy after the first dvc repro failed.)

@dberenbaum dberenbaum added product: VSCode Integration with VSCode extension and removed awaiting response we are waiting for your reply, please respond! :) labels Aug 1, 2023
@dberenbaum
Copy link
Collaborator

@AchimGaedkeLynker Have you been able to determine if GIT_OPTIONAL_LOCKS makes a difference?

@dberenbaum dberenbaum added the awaiting response we are waiting for your reply, please respond! :) label Aug 1, 2023
@AchimGaedkeLynker
Copy link
Author

AchimGaedkeLynker commented Aug 5, 2023 •
edited
Loading

@dberenbaum - no it doesn't seem to help. The error message also changed with the latest version/s:

PS C:\Users\achimgaedke\extCode\dvc_stress_test> $Env:GIT_OPTIONAL_LOCKS=0
PS C:\Users\achimgaedke\extCode\dvc_stress_test> dvc repro -v
2023-08-05 16:00:55,168 DEBUG: v3.11.1, CPython 3.10.11 on Windows-10-10.0.22621-SP0
2023-08-05 16:00:55,168 DEBUG: command: repro -v
2023-08-05 16:00:55,431 DEBUG: Lockfile for 'dvc.yaml' not found
2023-08-05 16:00:55,443 DEBUG: Lockfile for 'dvc.yaml' not found
2023-08-05 16:00:55,543 DEBUG: Dependency 'create_data_dir.py' of stage: 'create_data' changed because it is 'modified'.
2023-08-05 16:00:55,543 DEBUG: stage: 'create_data' changed.
2023-08-05 16:00:55,545 DEBUG: Removing output 'data' of stage: 'create_data'.
2023-08-05 16:00:55,546 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data'
2023-08-05 16:00:55,549 DEBUG: {'create_data_dir.py': 'modified'}
Running stage 'create_data':
> python create_data_dir.py
2023-08-05 16:01:03,181 DEBUG: Lockfile for 'dvc.yaml' not found
2023-08-05 16:01:15,126 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'                                                                                                                                                                              
2023-08-05 16:01:15,129 DEBUG: Computed stage: 'create_data' md5: '26237bb165f9c74df6920096edc70031'
2023-08-05 16:01:15,139 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-08-05 16:01:15,142 DEBUG: Preparing to transfer data from 'memory://dvc-staging-md5/60dc2168a9f5c260e9d6f8ab6a80dee5e8805025a4f7d1335e0d78f177eb17a1' to 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-08-05 16:01:15,143 DEBUG: Preparing to collect status from 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-08-05 16:01:15,143 DEBUG: Collecting status from 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-08-05 16:01:15,146 DEBUG: Preparing to collect status from 'memory://dvc-staging-md5/60dc2168a9f5c260e9d6f8ab6a80dee5e8805025a4f7d1335e0d78f177eb17a1'
2023-08-05 16:01:15,148 DEBUG: transfer dir: md5: 625be7c4ce6f1264a616c17b0e648bdc.dir with 10 files
2023-08-05 16:01:15,179 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'                                                                                                                                                                              
2023-08-05 16:01:15,185 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\.C3iskTZZHBAtgstpgrxdsh.tmp'
2023-08-05 16:01:15,187 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5\.BKbBCL5G3EqF9RMsrbggke.tmp'
2023-08-05 16:01:15,191 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file08'
2023-08-05 16:01:16,086 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file05'                                                                                                                                                        
2023-08-05 16:01:17,202 ERROR: failed to reproduce 'create_data': No possible cache link types for 'C:\Users\achimgaedke\extCode\dvc_stress_test\data'. See <https://dvc.org/doc/user-guide/troubleshooting#cache-types> for more information.                       
Traceback (most recent call last):
  File "dvc\repo\reproduce.py", line 199, in _reproduce
  File "dvc\repo\reproduce.py", line 129, in _reproduce_stage
  File "funcy\decorators.py", line 47, in wrapper
  File "dvc\stage\decorators.py", line 43, in rwlocked
  File "funcy\decorators.py", line 68, in __call__
  File "dvc\stage\__init__.py", line 433, in reproduce
  File "funcy\decorators.py", line 47, in wrapper
  File "dvc\stage\decorators.py", line 43, in rwlocked
  File "funcy\decorators.py", line 68, in __call__
  File "dvc\stage\__init__.py", line 626, in run
  File "funcy\decorators.py", line 47, in wrapper
  File "dvc\stage\decorators.py", line 43, in rwlocked
  File "funcy\decorators.py", line 68, in __call__
  File "dvc\stage\__init__.py", line 556, in commit
dvc.exceptions.CacheLinkError: No possible cache link types for 'C:\Users\achimgaedke\extCode\dvc_stress_test\data'. See <https://dvc.org/doc/user-guide/troubleshooting#cache-types> for more information.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "dvc\cli\__init__.py", line 209, in main
  File "dvc\cli\command.py", line 26, in do_run
  File "dvc\commands\repro.py", line 13, in run
  File "dvc\repo\__init__.py", line 64, in wrapper
  File "dvc\repo\scm_context.py", line 151, in run
  File "dvc\repo\reproduce.py", line 260, in reproduce
  File "dvc\repo\reproduce.py", line 203, in _reproduce
  File "dvc\repo\reproduce.py", line 167, in _raise_error
dvc.exceptions.ReproductionError: failed to reproduce 'create_data'

2023-08-05 16:01:17,208 DEBUG: Analytics is disabled.
PS C:\Users\achimgaedke\extCode\dvc_stress_test>

Here's a successful run outside VSCode

PS C:\Users\achimgaedke\extCode\dvc_stress_test> dvc repro -v
2023-08-05 16:11:47,283 DEBUG: v3.11.1, CPython 3.10.11 on Windows-10-10.0.22621-SP0
2023-08-05 16:11:47,283 DEBUG: command: repro -v
2023-08-05 16:11:47,543 DEBUG: Lockfile for 'dvc.yaml' not found
2023-08-05 16:11:47,543 DEBUG: Lockfile for 'dvc.yaml' not found
2023-08-05 16:11:47,625 DEBUG: Dependency 'create_data_dir.py' of stage: 'create_data' changed because it is 'modified'.
2023-08-05 16:11:47,634 DEBUG: stage: 'create_data' changed.
2023-08-05 16:11:47,635 DEBUG: Removing output 'data' of stage: 'create_data'.
2023-08-05 16:11:47,635 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data'
2023-08-05 16:11:47,639 DEBUG: {'create_data_dir.py': 'modified'}
Running stage 'create_data':
> python create_data_dir.py
2023-08-05 16:11:54,435 DEBUG: Lockfile for 'dvc.yaml' not found
2023-08-05 16:12:10,717 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-08-05 16:12:10,717 DEBUG: Computed stage: 'create_data' md5: '26237bb165f9c74df6920096edc70031'
2023-08-05 16:12:10,748 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-08-05 16:12:10,764 DEBUG: Preparing to transfer data from 'memory://dvc-staging-md5/60dc2168a9f5c260e9d6f8ab6a80dee5e8805025a4f7d1335e0d78f177eb17a1' to 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-08-05 16:12:10,764 DEBUG: Preparing to collect status from 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-08-05 16:12:10,764 DEBUG: Collecting status from 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-08-05 16:12:10,768 DEBUG: Preparing to collect status from 'memory://dvc-staging-md5/60dc2168a9f5c260e9d6f8ab6a80dee5e8805025a4f7d1335e0d78f177eb17a1'
2023-08-05 16:12:10,768 DEBUG: transfer dir: md5: 625be7c4ce6f1264a616c17b0e648bdc.dir with 10 files
2023-08-05 16:12:10,780 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-08-05 16:12:10,780 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\.Vx4YGUwbLkyeMQPraGAQxj.tmp'
2023-08-05 16:12:10,780 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5\.49odRazVARr5mYNmzgWFgN.tmp'
2023-08-05 16:12:10,795 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file00'
2023-08-05 16:12:11,925 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file01'
2023-08-05 16:12:13,588 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file08'
2023-08-05 16:12:14,388 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file06'
2023-08-05 16:12:15,518 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file02'
2023-08-05 16:12:16,977 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file04'
2023-08-05 16:12:18,451 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file05'
2023-08-05 16:12:19,910 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file09'
2023-08-05 16:12:21,306 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file03'
2023-08-05 16:12:22,797 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file07'
2023-08-05 16:12:24,319 DEBUG: stage: 'create_data' was reproduced
Generating lock file 'dvc.lock'
Updating lock file 'dvc.lock'

To track the changes with git, run:

        git add dvc.lock

To enable auto staging, run:

        dvc config core.autostage true
Use `dvc push` to send your updates to remote storage.
2023-08-05 16:12:24,335 DEBUG: Analytics is disabled.

@dberenbaum dberenbaum removed the awaiting response we are waiting for your reply, please respond! :) label Aug 5, 2023
@dberenbaum
Copy link
Collaborator

cc @mattseddon for visibility

@mattseddon
Copy link
Contributor

In the extension, we can add GIT_OPTIONAL_LOCKS to all of our Git calls. We can also reduce the overall number by introducing a dependency on the Git extension and pulling information from there. The second part will require a few days of work.

The issue in the VS Code repo is iterative/vscode-dvc#4111

@mattseddon mattseddon mentioned this issue Aug 7, 2023
Merged
@mattseddon
Copy link
Contributor

After making some updates to the extension (available in 1.0.35) I can run the script from #9749 (comment) without error (on a Mac):

Screen.Recording.2023-08-08.at.4.02.15.pm.mov 
❯ dvc doctor
DVC version: 3.13.2 (pip)
-------------------------
Platform: Python 3.10.10 on macOS-13.5-arm64-arm-64bit
Subprojects:
        dvc_data = 2.12.1
        dvc_objects = 0.24.1
        dvc_render = 0.5.3
        dvc_task = 0.3.0
        scmrepo = 1.1.0
Supports:
        http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3)
Config:
        Global: /Users/mattseddon/Library/Application Support/dvc
        System: /Library/Application Support/dvc
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk3s1s1
Caches: local
Remotes: None
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc, git
Repo.site_cache_dir: /Library/Caches/dvc/repo/ae1da75f7e11ee74c1386b2bc6f713c8

@mattseddon mattseddon mentioned this issue Aug 8, 2023
Merged
@AchimGaedkeLynker
Copy link
Author

AchimGaedkeLynker commented Aug 12, 2023 •
edited
Loading

Still the same trouble with DVC 3.14.0 when using VSCode - error message as described last week in #9749 (comment)

If / When being pedantic about issue management, one could say: The permission error is resolved (close this issue?) and open a new one of cache linking failing under MS Windows/VSCode...

@dberenbaum
Copy link
Collaborator

@AchimGaedkeLynker Does that issue still only happen when using VS Code?

@dberenbaum dberenbaum added the awaiting response we are waiting for your reply, please respond! :) label Aug 15, 2023
@AchimGaedkeLynker
Copy link
Author

Sorry, still happening with 3.15.3

2023-08-21 18:55:53,903 ERROR: failed to reproduce 'create_data': No possible cache link types for 'C:\Users\achimgaedke\extCode\dvc_test\data'. See <https://dvc.org/doc/user-guide/troubleshooting#cache-types> for more information.

@dberenbaum
Copy link
Collaborator

@AchimGaedkeLynker Does it only happen when using VS Code?

@AchimGaedkeLynker
Copy link
Author

AchimGaedkeLynker commented Sep 1, 2023 •
edited
Loading

Apologies to take that long. And thanks for keeping pushing.

Sorry, the error changes, but still fails when running in VSCode (powershell terminal in VSCode), but not when executing on the standalone powershell terminal (successful logs are attached below).

Is there a way I can make those error reports more productive / repducible for you? e.g. create a screen recording or provide some other setup / windows config information.

2023-09-01 12:05:09,185 DEBUG: v3.17.0, CPython 3.10.11 on Windows-10-10.0.22621-SP0
2023-09-01 12:05:09,205 DEBUG: command: repro -v
2023-09-01 12:05:09,720 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-01 12:05:09,736 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-01 12:05:09,932 DEBUG: Dependency 'create_data_dir.py' of stage: 'create_data' changed because it is 'modified'.
2023-09-01 12:05:09,938 DEBUG: stage: 'create_data' changed.
2023-09-01 12:05:09,946 DEBUG: Removing output 'data' of stage: 'create_data'.
2023-09-01 12:05:09,946 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest_vscode\data'
2023-09-01 12:05:09,954 DEBUG: {'create_data_dir.py': 'modified'}
Running stage 'create_data':
> python create_data_dir.py
2023-09-01 12:05:19,419 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-01 12:05:39,773 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-09-01 12:05:39,773 DEBUG: Computed stage: 'create_data' md5: '26237bb165f9c74df6920096edc70031'
2023-09-01 12:05:39,841 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-09-01 12:05:39,848 DEBUG: Preparing to transfer data from 'memory://dvc-staging-md5/f9cae4fd742c4aa89bc9eaca61a72290e723fd0bda6cc4c7f7504540702660a6' to 'C:\Users\achimgaedke\Code\dvc_stresstest_vscode\.dvc\cache\files\md5'
2023-09-01 12:05:39,848 DEBUG: Preparing to collect status from 'C:\Users\achimgaedke\Code\dvc_stresstest_vscode\.dvc\cache\files\md5'
2023-09-01 12:05:39,857 DEBUG: Collecting status from 'C:\Users\achimgaedke\Code\dvc_stresstest_vscode\.dvc\cache\files\md5'
2023-09-01 12:05:39,859 DEBUG: Preparing to collect status from 'memory://dvc-staging-md5/f9cae4fd742c4aa89bc9eaca61a72290e723fd0bda6cc4c7f7504540702660a6'  
2023-09-01 12:05:39,865 DEBUG: transfer dir: md5: 625be7c4ce6f1264a616c17b0e648bdc.dir with 10 files
2023-09-01 12:05:39,895 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'                                                                     
2023-09-01 12:05:39,904 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest_vscode\.UwV6y5bogFkMPtL8DMjVu3.tmp'
2023-09-01 12:05:39,911 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest_vscode\.dvc\cache\files\md5\.VabhQt785HcgHUPzPG6NVh.tmp'
2023-09-01 12:05:39,919 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest_vscode\data\big_random_file09'                                             
2023-09-01 12:05:39,927 ERROR: failed to reproduce 'create_data': [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\achimgaedke\\Code\\dvc_stresstest_vscode\\data\\big_random_file09'
Traceback (most recent call last):
  File "dvc_objects\fs\utils.py", line 89, in _unlink
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\achimgaedke\\Code\\dvc_stresstest_vscode\\data\\big_random_file09'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dvc\repo\reproduce.py", line 199, in _reproduce
  File "dvc\repo\reproduce.py", line 129, in _reproduce_stage
  File "funcy\decorators.py", line 47, in wrapper
  File "dvc\stage\decorators.py", line 43, in rwlocked
  File "funcy\decorators.py", line 68, in __call__
  File "dvc\stage\__init__.py", line 424, in reproduce
  File "funcy\decorators.py", line 47, in wrapper
  File "dvc\stage\decorators.py", line 43, in rwlocked
  File "funcy\decorators.py", line 68, in __call__
  File "dvc\stage\__init__.py", line 617, in run
  File "funcy\decorators.py", line 47, in wrapper
  File "dvc\stage\decorators.py", line 43, in rwlocked
  File "funcy\decorators.py", line 68, in __call__
  File "dvc\stage\__init__.py", line 540, in commit
  File "dvc\output.py", line 805, in commit
  File "dvc\output.py", line 759, in _checkout
  File "dvc_data\hashfile\checkout.py", line 270, in checkout
  File "dvc_data\hashfile\checkout.py", line 217, in _checkout
  File "dvc_data\hashfile\checkout.py", line 88, in _checkout_file
  File "dvc_data\hashfile\checkout.py", line 59, in _relink
  File "dvc_data\hashfile\checkout.py", line 55, in _remove
  File "dvc_objects\fs\base.py", line 476, in rm
  File "dvc_objects\fs\local.py", line 121, in rm
  File "dvc_objects\fs\utils.py", line 101, in remove
  File "dvc_objects\fs\utils.py", line 91, in _unlink
  File "dvc_objects\fs\utils.py", line 84, in _chmod
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\achimgaedke\\Code\\dvc_stresstest_vscode\\data\\big_random_file09'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "dvc\cli\__init__.py", line 209, in main
  File "dvc\cli\command.py", line 26, in do_run
  File "dvc\commands\repro.py", line 13, in run
  File "dvc\repo\__init__.py", line 61, in wrapper
  File "dvc\repo\scm_context.py", line 151, in run
  File "dvc\repo\reproduce.py", line 260, in reproduce
  File "dvc\repo\reproduce.py", line 203, in _reproduce
  File "dvc\repo\reproduce.py", line 167, in _raise_error
dvc.exceptions.ReproductionError: failed to reproduce 'create_data'

2023-09-01 12:05:39,984 DEBUG: Analytics is disabled.
 dvc doctor
DVC version: 3.17.0
-------------------
Platform: Python 3.10.11 on Windows-10-10.0.22621-SP0
Subprojects:

Supports:
        azure (adlfs = 2023.8.0, knack = 0.11.0, azure-identity = 1.14.0),
        gdrive (pydrive2 = 1.17.0),
        gs (gcsfs = 2023.6.0),
        hdfs (fsspec = 2023.6.0, pyarrow = 13.0.0),
        http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
        oss (ossfs = 2023.8.0),
        s3 (s3fs = 2023.6.0, boto3 = 1.28.17),
        ssh (sshfs = 2023.7.0),
        webdav (webdav4 = 0.9.8),
        webdavs (webdav4 = 0.9.8),
        webhdfs (fsspec = 2023.6.0)
Config:
        Global: C:\Users\achimgaedke\AppData\Local\iterative\dvc
        System: C:\ProgramData\iterative\dvc
Cache types: hardlink, symlink
Cache directory: NTFS on C:\
Caches: local
Remotes: None
Workspace directory: NTFS on C:\
Repo: dvc, git
Repo.site_cache_dir: C:\ProgramData\iterative\dvc\Cache\repo\b0057e00cc7a7743d344d53d5de1969f

@AchimGaedkeLynker
Copy link
Author

This is the log of the successful run in the windows powershell (executed with the same software versions as the above failed VSCode run):

 dvc repro -v
2023-09-01 11:58:11,500 DEBUG: v3.17.0, CPython 3.10.11 on Windows-10-10.0.22621-SP0
2023-09-01 11:58:11,500 DEBUG: command: repro -v
2023-09-01 11:58:11,704 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-01 11:58:11,704 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-01 11:58:11,799 DEBUG: Dependency 'create_data_dir.py' of stage: 'create_data' changed because it is 'modified'.
2023-09-01 11:58:11,799 DEBUG: stage: 'create_data' changed.
2023-09-01 11:58:11,799 DEBUG: Removing output 'data' of stage: 'create_data'.
2023-09-01 11:58:11,799 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data'
2023-09-01 11:58:11,799 DEBUG: {'create_data_dir.py': 'modified'}
Running stage 'create_data':
> python create_data_dir.py
2023-09-01 11:58:18,705 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-01 11:58:45,230 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-09-01 11:58:45,230 DEBUG: Computed stage: 'create_data' md5: '26237bb165f9c74df6920096edc70031'
2023-09-01 11:58:45,278 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-09-01 11:58:45,278 DEBUG: Preparing to transfer data from 'memory://dvc-staging-md5/a600f1d29891094036239a26abcb5b30f33f64940a2ad1e321e19f81b5e556ea' to 'C:\Users\achimgaedke\Code\dvc_stresstest\.dvc\cache\files\md5'
2023-09-01 11:58:45,278 DEBUG: Preparing to collect status from 'C:\Users\achimgaedke\Code\dvc_stresstest\.dvc\cache\files\md5'
2023-09-01 11:58:45,278 DEBUG: Collecting status from 'C:\Users\achimgaedke\Code\dvc_stresstest\.dvc\cache\files\md5'
2023-09-01 11:58:45,293 DEBUG: Preparing to collect status from 'memory://dvc-staging-md5/a600f1d29891094036239a26abcb5b30f33f64940a2ad1e321e19f81b5e556ea'
2023-09-01 11:58:45,293 DEBUG: transfer dir: md5: 625be7c4ce6f1264a616c17b0e648bdc.dir with 10 files
2023-09-01 11:58:45,314 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-09-01 11:58:45,314 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\.F6MBueW2D2BXetwFYaGhKX.tmp'
2023-09-01 11:58:45,314 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\.dvc\cache\files\md5\.6rGPkf75aJ9SNfFwnRqvSr.tmp'
2023-09-01 11:58:45,326 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file01'
2023-09-01 11:58:46,420 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file07'
2023-09-01 11:58:47,497 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file08'
2023-09-01 11:58:48,541 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file04'
2023-09-01 11:58:49,950 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file05'
2023-09-01 11:58:51,432 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file02'
2023-09-01 11:58:52,901 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file03'
2023-09-01 11:58:54,308 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file00'
2023-09-01 11:58:55,831 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file09'
2023-09-01 11:58:57,293 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file06'
2023-09-01 11:58:58,818 DEBUG: stage: 'create_data' was reproduced
Generating lock file 'dvc.lock'
Updating lock file 'dvc.lock'

To track the changes with git, run:

        git add dvc.lock

To enable auto staging, run:

        dvc config core.autostage true
Use `dvc push` to send your updates to remote storage.
2023-09-01 11:58:58,828 DEBUG: Analytics is disabled.

And some diagnostics:

PS C:\Users\achimgaedke\Code\dvc_stresstest> dvc doctor
DVC version: 3.17.0
-------------------
Platform: Python 3.10.11 on Windows-10-10.0.22621-SP0
Subprojects:

Supports:
        azure (adlfs = 2023.8.0, knack = 0.11.0, azure-identity = 1.14.0),
        gdrive (pydrive2 = 1.17.0),
        gs (gcsfs = 2023.6.0),
        hdfs (fsspec = 2023.6.0, pyarrow = 13.0.0),
        http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
        oss (ossfs = 2023.8.0),
        s3 (s3fs = 2023.6.0, boto3 = 1.28.17),
        ssh (sshfs = 2023.7.0),
        webdav (webdav4 = 0.9.8),
        webdavs (webdav4 = 0.9.8),
        webhdfs (fsspec = 2023.6.0)
Config:
        Global: C:\Users\achimgaedke\AppData\Local\iterative\dvc
        System: C:\ProgramData\iterative\dvc
Cache types: hardlink, symlink
Cache directory: NTFS on C:\
Caches: local
Remotes: None
Workspace directory: NTFS on C:\
Repo: dvc, git
Repo.site_cache_dir: C:\ProgramData\iterative\dvc\Cache\repo\c8b5cfd28ee3665dc9ba223b1457822b

@dberenbaum
Copy link
Collaborator

@AchimGaedkeLynker Thanks for the detailed report. Do you have the DVC extension installed in VS Code? If so, which version? Do you get the error if you uninstall it?

@mattseddon Any other ideas?

@AchimGaedkeLynker
Copy link
Author

Hi there!

The VSCode extension is DVC v1.0.58. Removing the extension avoids the failure.

With the extension installed and using the VSCode terminal: I get this error message from dvc repro -v:

2023-09-30 12:30:57,513 DEBUG: v3.23.0, CPython 3.10.11 on Windows-10-10.0.22621-SP0
2023-09-30 12:30:57,513 DEBUG: command: repro -v
2023-09-30 12:30:57,736 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-30 12:30:57,742 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-30 12:30:57,843 DEBUG: Dependency 'create_data_dir.py' of stage: 'create_data' changed because it is 'modified'.
2023-09-30 12:30:57,843 DEBUG: stage: 'create_data' changed.
2023-09-30 12:30:57,853 DEBUG: Removing output 'data' of stage: 'create_data'.
2023-09-30 12:30:57,853 DEBUG: Removing 'C:\Users\achimgaedke\dvc_stress_test_vscode\data'
2023-09-30 12:30:57,863 DEBUG: {'create_data_dir.py': 'modified'}
Running stage 'create_data':
> python create_data_dir.py
2023-09-30 12:33:10,522 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-30 12:33:16,442 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-09-30 12:33:16,442 DEBUG: Computed stage: 'create_data' md5: 'f245734c5a8cc0588b72210046fab8da'
2023-09-30 12:33:16,490 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-09-30 12:33:16,495 DEBUG: Preparing to transfer data from 'memory://dvc-staging-md5/6f37f50b836981cf8ae7965ff10b89389b3a1dd67f2083d52d7c47882a45c101' to 'C:\Users\achimgaedke\dvc_stress_test_vscode\.dvc\cache\files\md5'
2023-09-30 12:33:16,495 DEBUG: Preparing to collect status from 'C:\Users\achimgaedke\dvc_stress_test_vscode\.dvc\cache\files\md5'
2023-09-30 12:33:16,499 DEBUG: Collecting status from 'C:\Users\achimgaedke\dvc_stress_test_vscode\.dvc\cache\files\md5'
2023-09-30 12:33:16,505 DEBUG: Preparing to collect status from 'memory://dvc-staging-md5/6f37f50b836981cf8ae7965ff10b89389b3a1dd67f2083d52d7c47882a45c101'
2023-09-30 12:33:16,509 DEBUG: transfer dir: md5: 625be7c4ce6f1264a616c17b0e648bdc.dir with 10 files
2023-09-30 12:33:16,544 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-09-30 12:33:16,553 DEBUG: Removing 'C:\Users\achimgaedke\dvc_stress_test_vscode\.Lv9wDsT9dbEcJkkoyPcPmt.tmp'
2023-09-30 12:33:16,556 DEBUG: Removing 'C:\Users\achimgaedke\dvc_stress_test_vscode\.dvc\cache\files\md5\.3ZQxAU2E7qFJ2NpWP57iHF.tmp'
2023-09-30 12:33:16,561 DEBUG: Removing 'C:\Users\achimgaedke\dvc_stress_test_vscode\data\big_random_file05'
2023-09-30 12:33:32,173 ERROR: failed to reproduce 'create_data': No possible cache link types for 'C:\Users\achimgaedke\dvc_stress_test_vscode\data'. See <https://dvc.org/doc/user-guide/troubleshooting#cache-types> for more information.
Traceback (most recent call last):
  File "dvc\repo\reproduce.py", line 199, in _reproduce
  File "dvc\repo\reproduce.py", line 129, in _reproduce_stage
  File "funcy\decorators.py", line 47, in wrapper
  File "dvc\stage\decorators.py", line 43, in rwlocked
  File "funcy\decorators.py", line 68, in __call__
  File "dvc\stage\__init__.py", line 436, in reproduce
  File "funcy\decorators.py", line 47, in wrapper
  File "dvc\stage\decorators.py", line 43, in rwlocked
  File "funcy\decorators.py", line 68, in __call__
  File "dvc\stage\__init__.py", line 629, in run
  File "funcy\decorators.py", line 47, in wrapper
  File "dvc\stage\decorators.py", line 43, in rwlocked
  File "funcy\decorators.py", line 68, in __call__
  File "dvc\stage\__init__.py", line 559, in commit
dvc.exceptions.CacheLinkError: No possible cache link types for 'C:\Users\achimgaedke\dvc_stress_test_vscode\data'. See <https://dvc.org/doc/user-guide/troubleshooting#cache-types> for more information.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "dvc\cli\__init__.py", line 209, in main
  File "dvc\cli\command.py", line 26, in do_run
  File "dvc\commands\repro.py", line 13, in run
  File "dvc\repo\__init__.py", line 61, in wrapper
  File "dvc\repo\scm_context.py", line 151, in run
  File "dvc\repo\reproduce.py", line 260, in reproduce
  File "dvc\repo\reproduce.py", line 203, in _reproduce
  File "dvc\repo\reproduce.py", line 167, in _raise_error
dvc.exceptions.ReproductionError: failed to reproduce 'create_data'

2023-09-30 12:33:32,173 DEBUG: Analytics is disabled.

@dberenbaum
Copy link
Collaborator

@AchimGaedkeLynker What about with the extension installed but running outside of VS Code?

@thomaswp
Copy link

Just wanted to add I'm experiencing the same issue on Windows 10 with the VSCode Extension, and running outside of VSCode (with it closed) did work. Thanks!

@skshetry skshetry removed the awaiting response we are waiting for your reply, please respond! :) label Mar 25, 2024
@12michi34
Copy link

just wondering if there is any update on this. I get the same error but outside VSCode (command console, power console, "as administrator" all with the same result). Only happens when doing an "dvc import" on a Windows PC. "dvc get" works fine (so I assume its again the checkout stage that causes issues). No such problem on linux clients observed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
P: windows Related to the Platform: Windows product: VSCode Integration with VSCode extension
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@thomaswp @dberenbaum @efiop @skshetry @12michi34 @mattseddon @AchimGaedkeLynker