Code

Multiple Zarr to Rule them All#54

Code

Closed

sadamov wants to merge 67 commits intomainmllam/neural-lam:mainfrom feature_dataset_yamlmllam/neural-lam:feature_dataset_yamlCopy head branch name to clipboard

+1,668-1,067

Commits

Commits on May 5, 2024

• yaml_config for cosmo data

  

  Simon Adamov

  committed
  e5c245f

• initial version of single zarr dataset

  

  Simon Adamov

  committed
  33e7ecf

Commits on May 6, 2024

• handling None zarrs

  

  Simon Adamov

  committed
  9936e3b

Commits on May 8, 2024

• removed all dependencies on constants.py

  Show description for 774d16a
  

  Simon Adamov

  committed
  774d16a

• fix linter

  

  Simon Adamov

  committed
  7bb139b

• Fixed calls to new WeatherDataModule Class

  

  Simon Adamov

  committed
  af076fe

• fix linter

  

  Simon Adamov

  committed
  147caec

• upload data config to wandb for history logs

  

  Simon Adamov

  committed
  2b65416

• Improved handling of static data

  

  Simon Adamov

  committed
  ed9ed69

• dask and zarr are required backends to xarray

  

  Simon Adamov

  committed
  0b69f4e

Commits on May 9, 2024

• Implements windowed forcing and boundary

  

  Simon Adamov

  committed
  b76d078

• Some project related stuff (simple setup to pip install -e .)

  

  Simon Adamov

  committed
  5d27a4c

• introducing realistic boundaries

  

  Simon Adamov

  committed
  4dadf29

• Adapted nwp_xy related code to new data loading procedure

  

  Simon Adamov

  committed
  7524c4d

• only state requires units for plotting

  Show description for 45fd375
  

  Simon Adamov

  committed
  45fd375

• small bugfixes and improvements

  

  Simon Adamov

  committed
  812323d

• Calculate stats and store in zarr archive

  Show description for 500f2fb
  

  Simon Adamov

  committed
  500f2fb

• latex support

  

  Simon Adamov

  committed
  9293fe1

• ar_steps for training and eval

  

  Simon Adamov

  committed
  e80aa58

• smaller ammendments

  

  Simon Adamov

  committed
  a86fc07

• Dummy mask was inverted - fixed

  

  Simon Adamov

  committed
  7ae9c87

• replace hardcoded normalization path

  

  Simon Adamov

  committed
  93674a2

Commits on May 23, 2024

• create 3D mesh objects for schol AR

  

  Simon Adamov

  committed
  f2cbc44

Commits on May 28, 2024

• fixed math writing

  sadamov

  committed
  c0e7529

• Merge branch 'feature_dataset_yaml' of https://github.com/joeloskarsson/neural-lam into feature_dataset_yaml

  sadamov

  committed
  af7751a

• cherry-pick with main

  sadamov

  committed
  5f538f9

• bugfixes

  sadamov

  committed
  6685e94

• pre_commits

  sadamov

  committed
  6423fdf

Commits on May 31, 2024

• Merge remote-tracking branch 'origin/main' into feature_dataset_yaml

  sadamov

  committed
  59c4947

• config.py is ready for danra

  sadamov

  committed
  4e457ed

Commits on Jun 1, 2024

• streamlined multi-zarr workflow

  sadamov

  committed
  adc592f

• xarray zarr based data normalization

  sadamov

  committed
  a7bea6b

Commits on Jun 2, 2024

• adjusted pre-processing scripts to new data config workflow

  sadamov

  committed
  1f7cbe8

• plotting update with latest get_xy() function

  sadamov

  committed
  e328152

• making data config more modular

  sadamov

  committed
  cb85cda

• removing boundaries for now

  sadamov

  committed
  eb8c6fb

• small updates

  sadamov

  committed
  0cfbb33

• improved stats and units retrieval

  sadamov

  committed
  59d0c8a

Commits on Jun 3, 2024

• improved zarr-based normalization

  sadamov

  committed
  668dd81

• merge main

  sadamov

  committed
  374d032

• Introduced datetime forcing calculation as seperate script

  sadamov

  committed
  8fa3ca7

• Fixed order of y and x dims to adhere to #52

  sadamov

  committed
  a748903

Commits on Jun 4, 2024

• cleanup: boundary_mask, zarr-opening, utils

  sadamov

  committed
  74b4a10

• Merge remote-tracking branch 'origin/main' into feature_dataset_yaml

  sadamov

  committed
  0a041d1

• adapted tests for zarr-analysis data

  sadamov

  committed
  425123c

• Readme adapted for yaml zarr analysis workflow

  sadamov

  committed
  4dcf671

• samller bugfixes and improvements

  sadamov

  committed
  6d384f0

• Added fixed data config file for testing on Danra

  sadamov

  committed
  12ff4f2

• reducing runtime of tests with smaller sample

  sadamov

  committed
  03f7769

Commits on Jun 6, 2024

• download danra data for test and example (streaming not possible)

  sadamov

  committed
  26f069c

• bugfixes after real-life testcase

  sadamov

  committed
  1f1cbcc

• Merge remote-tracking branch 'origin/main' into feature_dataset_yaml

  sadamov

  committed
  b369306

• organize .zarr in /data

  sadamov

  committed
  0cdc361

• cleanup

  sadamov

  committed
  23ca7b3

• linter

  sadamov

  committed
  81422f1

Commits on Jun 7, 2024

• static dataset doesn't have time dim

  sadamov

  committed
  124541b

• making two complex functions more modular

  sadamov

  committed
  6140fdb

Commits on Jun 8, 2024

• chunk dataset by time

  sadamov

  committed
  db6a912

• create list first for performance

  sadamov

  committed
  1aaa8dc

• converting to_array is very slow

  Show description for 81856b2
  sadamov

  committed
  81856b2

• allow for forcings to not be normalized

  sadamov

  committed
  b3da818

• allow non_normalized_vars to be null

  sadamov

  committed
  7ee5398

• fixed coastlines using new xy_extent function

  sadamov

  committed
  4782103

Commits on Jun 9, 2024

• Some projections return inverted axes (rotatedPole)

  sadamov

  committed
  e0ffc5b

Commits on Jun 13, 2024

• Docstrings added

  sadamov

  committed
  c1f43b7

Commits on Aug 7, 2024

• Merge branch 'main' into feature_dataset_yaml

  sadamov

  committed
  1a1f5b2

• Bugfix training without forcing data

  sadamov

  committed
  5f72fe3