Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

open_dataset not closing NetCDF file (Windows) #1629

Closed
jennirinker opened this issue Oct 12, 2017 · 4 comments
Closed

open_dataset not closing NetCDF file (Windows) #1629

jennirinker opened this issue Oct 12, 2017 · 4 comments

Comments

@jennirinker
Copy link

MWE

Entering

test_ds = xr.open_dataset(path_to_nc)
os.remove(path_to_nc)

results in

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: (path_to_nc)

If I use the option autoclose=True, I can delete the file without this error, but then when I try to access test_ds I get OSError: No such file or directory. (This second part might be expected behavior depending on how autoclose is supposed to function.)

conda info

Current conda install:

               platform : win-64
          conda version : 4.3.27
       conda is private : False
      conda-env version : 4.3.27
    conda-build version : not installed
         python version : 3.6.1.final.0
       requests version : 2.14.2
       root environment : C:\Users\rink\AppData\Local\Continuum\Miniconda3  (wri
table)
    default environment : C:\Users\rink\AppData\Local\Continuum\Miniconda3
       envs directories : C:\Users\rink\AppData\Local\Continuum\Miniconda3\envs
                          C:\Users\rink\AppData\Local\conda\conda\envs
                          C:\Users\rink\.conda\envs
          package cache : C:\Users\rink\AppData\Local\Continuum\Miniconda3\pkgs
                          C:\Users\rink\AppData\Local\conda\conda\pkgs
           channel URLs : https://repo.continuum.io/pkgs/main/win-64
                          https://repo.continuum.io/pkgs/main/noarch
                          https://repo.continuum.io/pkgs/free/win-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/win-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/win-64
                          https://repo.continuum.io/pkgs/pro/noarch
                          https://repo.continuum.io/pkgs/msys2/win-64
                          https://repo.continuum.io/pkgs/msys2/noarch
            config file : None
             netrc file : None
           offline mode : False
             user-agent : conda/4.3.27 requests/2.14.2 CPython/3.6.1 Windows/7 W
indows/6.1.7601
          administrator : True
@fmaussion
Copy link
Member

Thank you for the report! This is expected behavior: users have the responsibility to open and close their files. You can either close it manually:

test_ds.close()

Or (better), use a context manager:

with xr.open_dataset(path_to_nc) as ds:
    # do stuff

@fmaussion
Copy link
Member

I just read the second part of you question: you can't delete the file and then still read data out of it. What is your use case?

@jennirinker
Copy link
Author

Thanks for response.

Basically I've inherited some legacy code that subclasses an xarray dataset, and all of the tests pass on Linux but one fails on window from this issue. I don't think the code ever closes a dataset, nor does it use context managers. We're loading data stored in a netcdf into the subclass instance and then doing a bunch of calculations on it.

Anyway, I agree that the current behavior is actually the ideal scenario, it was just weird at first when the tests passed on Linux and not Windows. I'll have to see if I can patch the code to use context managers, since manually closing the file will probably not always happen.

@shoyer
Copy link
Member

shoyer commented Oct 15, 2017

In my experience, Windows is certainly much pickier than Linux about deleting files that are still open.

LuukBlom added a commit to Deltares/hydromt_sfincs that referenced this issue Jan 8, 2025
roeldegoede added a commit to Deltares/hydromt_sfincs that referenced this issue Jan 10, 2025
* bugfix so that the .nc file is closed automatically when erroring or when leaving the context manager (with block). see pydata/xarray#1629 (comment)

* fixed all occurences of xr.opendataset with the safe open&close pattern

* review comments by Roel

* fix linting

---------

Co-authored-by: roeldegoede <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants