Save all files in a task at the same time to avoid recomputing intermediate results

[bouweandela]

Description

Save all files in a task at the same time to avoid recomputing intermediate results.

This change is not backward compatible because it changes the return value of esmvalcore.preprocessor.save, which is part of the public API. Previously this function returned the filename, not it returns None on immediate saves and a dask.delayed.Delayed for delayed saves that can be requested with the compute=False argument.

Closes #2521
Closes #2042

Links to documentation:

• https://esmvaltool--2522.org.readthedocs.build/projects/ESMValCore/en/2522/quickstart/configure.html#logging-configuration
• https://esmvaltool--2522.org.readthedocs.build/projects/ESMValCore/en/2522/api/esmvalcore.preprocessor.html#esmvalcore.preprocessor.save

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.72%. Comparing base (d8ad2f0) to head (e60520a).
Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2522      +/-   ##
==========================================
+ Coverage   94.67%   94.72%   +0.05%     
==========================================
  Files         251      252       +1     
  Lines       14302    14436     +134     
==========================================
+ Hits        13540    13675     +135     
+ Misses        762      761       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[valeriupredoi]

@bouweandela this is very nice, but I have serious concerns related to store and locking, there are many known issues with HDF5 and threads, and even the Dask folk are looking at this type of IO issue, see eg dask/distributed#780 - inherently what iris are doing is also not thread-safe, so I am doubly concerned

[bouweandela]

there are many known issues with HDF5 and threads, and even the Dask folk are looking at this type of IO issue, see eg dask/distributed#780 -

The linked issue is not related to writing to HDF5 files.

inherently what iris are doing is also not thread-safe, so I am doubly concerned

If you have concerns about Iris's capability, I would recommend playing around with it and see if you can get it to crash and report any issues you find on the Iris GitHub page. The code that handles saving to NetCDF with distributed lives in `iris.fileformats.netcdf in case you would like to have a look.

I have serious concerns related to store and locking

Could you elaborate on those and provide an example of a case where it does not work?

[valeriupredoi]

Could you elaborate on those and provide an example of a case where it does not work?

I need to hatch me a few used cases and test for stress points. Not on the priority list though, so let's see if any issues pop up naturally, am not gonna block this PR, just wanted to see if you have any concerns too 🍺

[valeriupredoi]

it seems that a Lock object is indeed in dask.distributed, so first hurdle I was afraid of is alleviated dask/dask#1892 (comment)

[valeriupredoi]

this, though, is a bit scary dask/dask#2488

[bouweandela]

this, though, is a bit scary dask/dask#2488

We're not using the to_hdf5 method, so I'm not worried about that issue.

I need to hatch me a few used cases and test for stress points. Not on the priority list though, so let's see if any issues pop up naturally, am not gonna block this PR, just wanted to see if you have any concerns too 🍺

It would be great if you could give it a try. I've tested this with the recipe in #2300 and there is seems to work well.

[valeriupredoi]

this, though, is a bit scary dask/dask#2488

let's see if this hits us, after we merge - no point holding up a performance imrpovement for something that's putative 👍

[bouweandela]

The conclusion from community consultation at the workshop is that we would like to display progress bars (also for downloads) and will provide a configuration option to disable them.

[bouweandela]

For example, when attached to a non-interactive terminal you could print the percentage completion

@jfrost-mo I implemented this solution for the case where a progress bar is not desired.

[bouweandela]

@schlunma It is now possible to disable the progress bar by setting a value other than 0 for log_progress_interval in the configuration, e.g.

logging:
  log_progress_interval: 10s

If max_parallel_tasks is set to a value other than 1, the progress bar is also disabled to avoid writing multiple overlapping progress bars.

Does it look better now?

[schlunma]

Thanks Bouwe, that look great! I really love the new progress bar ❤️ . It is also nicely rendered in the SLURM log now (I tried VSCode, vim, and cat):

The only "issue" I found is that with a distributed scheduler, the progress bar sometimes does not finish with "100%" (I didn't see this with the default scheduler):

One "issue" with the progress logger is that with the default scheduler, there is a new line character after the bar, this does not happen with a distributed scheduler:

I wonder if it would make sense to add an option to disable the progress logging completely? Maybe with log_progress_interval=-1?

Apart from that, I only have minor comments on the doc. Thank you!!

[schlunma]

Thanks Bouwe, tested this again and all looks as expected now! Cheers!

Will merge now since two people approved. 🚀