flexible chunks again in dfmt.open_partitioned_dataset()

[veenstrajelmer]

Deltares/xugrid#253 made chunks argument not flexible in some cases. This new code now gives merging errors for some large datafiles.

Todo:

• wait for fix of avoid inconsistent chunks error in merge_partitions() xugrid#279
• set new minimal xugrid version after release (this now properly supports any chunks argument again also for merged partitions, following avoid inconsistent chunks error in merge_partitions() xugrid#279)
• test chunks='auto' in dfmt.open_partitioned_dataset() >> performance for plotting a single field is way slower (code below)
• this should also not raise the warning in suppress chunking warning again #947 anymore, so disable the suppression again if that is indeed the case >> keep the suppression, since we will not change the default chunks argument and for e.g. EPD it is equally fast to plot a single field independent on the chunks.
• also test with merging p:\11210331-004-bes-modellering-2024\4_simulations\hydrodynamica\simulations\abc_000\DFM_OUTPUT_abc\abc_0*_map.nc ("ValueError: Object has inconsistent chunks along dimension Two") >> works with xugrid main branch where avoid inconsistent chunks error in merge_partitions() xugrid#279 is included

Code to test DCSM with different chunks:

import dfm_tools as dfmt
import datetime as dt

file_nc = r'p:\1204257-dcsmzuno\2006-2012\3D-DCSM-FM\A18b_ntsu1\DFM_OUTPUT_DCSM-FM_0_5nm\DCSM-FM_0_5nm_0*_map.nc'

chunks = "auto" # 21% of memory increase
# chunks = {"time":1} # 2% of memory increase

uds = dfmt.open_partitioned_dataset(file_nc, chunks=chunks)

print('>> load single timestep of waterlevels: ', end='')
dtstart = dt.datetime.now()
uds['mesh2d_s1'].isel(time=365).load()
print(f'{(dt.datetime.now()-dtstart).total_seconds():.2f} sec')

Gives:

chunks = "auto"
>> xu.open_dataset() with 20 partition(s): 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 : 11.92 sec
>> xu.merge_partitions() with 20 partition(s): 4.37 sec
>> dfmt.open_partitioned_dataset() total: 16.31 sec
>> load single timestep of waterlevels: 54.47 sec
25% memory increase

chunks = {"time":1}
>> xu.open_dataset() with 20 partition(s): 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 : 11.30 sec
>> xu.merge_partitions() with 20 partition(s): 4.21 sec
>> dfmt.open_partitioned_dataset() total: 15.53 sec
>> load single timestep of waterlevels: 0.48 sec
2% memory increase