Features/517-netcdf4-features

[ben-bou]

Description

Adding the following features to the save_netcdf method:

• Naming of dimensions
• Creating unlimited dimensions
• Usage of already existing dimensions and variables
• Specifying the Location of data in the netcdf4-File using indices or slices

Issue/s resolved: #517 (partially)

Changes proposed:

Type of change

Due Diligence

• All split configurations tested
• Multiple dtypes tested in relevant functions
• Documentation updated (if needed)
• Updated changelog.md under the title "Pending Additions"

Does this change modify the behaviour of other functions? If so, which?

no

[coquelin77]

some of the formatting is wrong here. this is fixed automatically by pre-commit (most of the time). all you need to do is install pre-commit (pip install pre-commit or conda install -c conda-forge pre-commit) then run pre-commit instal and pre-commit run --all-files

[ben-bou]

Thanks, I ran pre-commit and updated the brach.

[coquelin77]

bump. is this being worked on? to run the tests locally you can use mpirun -np # ...

[ben-bou]

What do you mean, is anything missing?

[coquelin77]

the tests are failing. it looks like something in the IO is hanging. you can see it in the Travis CI - Pull Request details

[ben-bou]

In Travis CI, IO is hanging, but if I do
mpiexec -n 4 coverage run --source=heat --parallel-mode -m pytest heat/ && coverage combine && coverage report && coverage xml, it passes for 1...4 processes.
Any suggestions? Maybe there are incompatibilities in the underlying netcdf Versions?

[coquelin77]

check the versions. it is listed in the log for the travis builds but i have also listed it here:

Package        Version  Location
-------------- -------- --------
atomicwrites   1.3.0    
attrs          19.1.0   
certifi        2019.3.9 
cftime         1.1.2    
chardet        3.0.4    
codecov        2.0.15   
coverage       4.5.3    
future         0.18.2   
h5py           2.10.0   
heat           0.3.0    /heat   
idna           2.8      
more-itertools 7.0.0    
mpi4py         3.0.3    
netCDF4        1.5.2    
numpy          1.18.4   
pip            19.0.3   
pluggy         0.9.0    
py             1.8.0    
pytest         4.4.0    
requests       2.21.0   
setuptools     40.8.0   
six            1.12.0   
torch          1.5.0    
urllib3        1.24.1 

[ben-bou]

Hi, I've got a quick question:

Lets say I've got a dndarray data and a shape out_shape, which is basically the same as data.shape but it might have ones added in between. Is there a way to find out the hypothetical split-axis of data if it was broadcast to out_shape?

[ben-bou]

ok, seems like out_split = data.split + out_shape[: data.split].count(1) - data.shape[: data.split].count(1) does this

[ben-bou]

These formatting errors have nothing to do with my changes. Where do they suddenly come from? Is there a way to suppress them? I don't get any errors when running pre-commit locally.

[codecov]

Codecov Report

Merging #559 (9eedf74) into master (debeaf7) will decrease coverage by 0.05%.
The diff coverage is 88.99%.

@@            Coverage Diff             @@
##           master     #559      +/-   ##
==========================================
- Coverage   97.59%   97.54%   -0.06%     
==========================================
  Files          87       87              
  Lines       18030    18219     +189     
==========================================
+ Hits        17596    17771     +175     
- Misses        434      448      +14     

Impacted Files	Coverage Δ
heat/core/io.py	88.73% <80.16%> (-1.66%)	⬇️
heat/core/tests/test_io.py	94.79% <100.00%> (+1.76%)	⬆️
heat/core/types.py	95.36% <0.00%> (+0.51%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update debeaf7...9eedf74. Read the comment docs.

[coquelin77]

please extend the unit tests to cover the new additions

[Markus-Goetz]

Also, I will have time to look at it next week an make some addition/changes. Sorry for the long delay

[coquelin77]

bump, there was an update which removed the ht_device parameter. see other tests for how to use it

[mtar]

GPU cluster tests are currently disabled on this Pull Request.

[ClaudiaComito]

Hi Ben, please replace _DNDarray__array__ with the new larray for consistency with the rest - also in the follow-up code

[ClaudiaComito]

_DNDarray__array --> larray

[ClaudiaComito]

Thanks a lot @ben-bou for your contribution!