Putting this on hold for now, there was a problem in the live synoptic pipeline ingestion that resulted in not all AIA being downloaded live. This was corrected last week for Heliolab but has yet to be run over the time period July 2023 - July 2024 (inclusive). A almost completed dataset creation script is available at https://github.com/spaceml-org/SDO-FM/blob/webdataset-dataloader/scripts/dataset_construction/make_sdoml_webdataset.py

Things that need to be completed are:

• Download of complete 3min cadence post SDOMLv2a-static end time
• The dataset creation script above needs to be validated against that full set, and dataset constructed. This would likely feed into a live SDOML future
• The great start Richard made in https://github.com/spaceml-org/SDO-FM/blob/webdataset-dataloader/sdofm/datasets/SDOMLWebDataset.py needs to be modified to read in the daily tar format

Nice to haves that would be required for a Phase B:

• The Zarr dataset could be converted
• Eventually the complete 3min cadence downloaded