You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have had difficulty converting the NWM subset from Zarr to Parquet in parallel using xArray and Dask. See pydata/xarray#6811 and https://dask.discourse.group/t/workers-dont-have-promised-key-error-and-delayed-computation/936. If we cannot resolve our problems by increasing our understanding of Dask by reading documentation and forum posts, we should find a workaround. One potential workaround is to use AWS Batch to run a script in parallel (using the job array functionality in Batch), where each job selects a specific piece of the datasets and saves it as a Parquet file.
The text was updated successfully, but these errors were encountered:
We have had difficulty converting the NWM subset from Zarr to Parquet in parallel using xArray and Dask. See pydata/xarray#6811 and https://dask.discourse.group/t/workers-dont-have-promised-key-error-and-delayed-computation/936. If we cannot resolve our problems by increasing our understanding of Dask by reading documentation and forum posts, we should find a workaround. One potential workaround is to use AWS Batch to run a script in parallel (using the job array functionality in Batch), where each job selects a specific piece of the datasets and saves it as a Parquet file.
The text was updated successfully, but these errors were encountered: