-
Notifications
You must be signed in to change notification settings - Fork 129
/
datasets.yml
1292 lines (1162 loc) · 48.5 KB
/
datasets.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
# Dataset information
---
datasets:
AERONET:
tier: 3
source: "https://aeronet.gsfc.nasa.gov/"
last_access: 2024-04-06
info: |
Aerosol Optical Depth information from a worldwide network of stations.
AGCD:
tier: 2
source: "http://dx.doi.org/10.25914/6009600786063"
last_access: 2023-11-21
info: |
Australian Gridded Climate Data (AGCD) version 2 is the Bureau of Meteorology's official dataset for climate
analyses covering analysis of monthly rainfall. The dataset provides consistent temporal and spatial analyses
across Australia for each observed data variable. This accounts for spatial and temporal gaps in observations.
Where possible, the gridded analysis techniques provide useful estimates in data-sparse regions
such as central Australia.
Time coverage: Site-based data are used to provide gridded climate data at the monthly timescale for rainfall (1900+).
Reference: Evans, A., Jones, D.A., Smalley, R., and Lellyett, S. 2020. An enhanced gridded rainfall analysis scheme
for Australia. Bureau of Meteorology Research Report. No. 41.
National Computational Infrastructure (NCI) - Catalogue Record: http://dx.doi.org/10.25914/6009600786063.
Data from NCI (National Computing Infrastructure Australia https://nci.org.au/),
requires an NCI account and access to Gadi(Supercomputer in Canberra) and the project found in catalogue record.
Access can be requested through NCI. NCI is an ESGF node (https://esgf.nci.org.au/projects/esgf-nci/)
ANUClimate:
tier: 3
source: "https://dx.doi.org/10.25914/60a10aa56dd1b"
last_access: 2023-11-21
info: |
Data from NCI project requiring an NCI account and access to GADI
ANUClimate 2.0 consists of gridded daily and monthly climate variables across the terrestrial landmass of Australia
from at least 1970 to the present. Rainfall grids are generated from 1900 to the present. The underpinning spatial
models have been developed at the Fenner School of Environment and Society of the Australian National University.
APHRO-MA:
tier: 3
source: "http://aphrodite.st.hirosaki-u.ac.jp/download/"
last_access: 2020-03-06
info: |
Register at http://aphrodite.st.hirosaki-u.ac.jp/download/create/
To use the automatic download set your credentials in your .netrc file.
See https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html.
To download manually, download the following files from
http://aphrodite.st.hirosaki-u.ac.jp/product/:
APHRO_V1808_TEMP/APHRO_MA
025deg_nc/APHRO_MA_TAVE_025deg_V1808.nc.tgz
050deg_nc/APHRO_MA_TAVE_050deg_V1808.nc.tgz
APHRO_V1101/APHRO_MA
025deg_nc/APHRO_MA_025deg_V1101.1951-2007.nc.gz.tar
050deg_nc/APHRO_MA_050deg_V1101.1951-2007.nc.gz.tar
APHRO_V1101EX_R1/APHRO_MA
025deg_nc/APHRO_MA_025deg_V1101_EXR1.nc.tgz
050deg_nc/APHRO_MA_050deg_V1101_EXR1.nc.tgz
Please untar / unzip all *.tar *.tgz *.gz files in the same directory
(no subdirectories!) prior to running the cmorizer!
AURA-TES:
tier: 3
source: https://search.earthdata.nasa.gov/search?q=TL3O3M
last_access: 2018-12-08
info: |
Download and processing instructions
Select the V004 and V005 projects.
Download the script file for both projects.
To download the data use: wget -i <scriptfile>.
From the downloaded data, pick only the *.he5 files and put them in
input_dir_path.
Data is freely available, but a registration is required.
BerkeleyEarth:
tier: 2
source: http://berkeleyearth.org/data/
last_access: 2020-02-25
info: |
Download the following file:
http://berkeleyearth.lbl.gov/auto/Global/Gridded/Land_and_Ocean_LatLong1.nc
CALIPSO-GOCCP:
tier: 2
source: ftp://ftp.climserv.ipsl.polytechnique.fr/
last_access: 2020-01-27
info: |
Download the data from:"
ftp://ftp.climserv.ipsl.polytechnique.fr/
path: /cfmip/GOCCP_v3/3D_CloudFraction/grid_2x2xL40/yyyy/avg/
Put all files under a single directory (no subdirectories with years).
3D_CloudFraction330m_yyyymm_avg_CFMIP2_sat_3.1.2.nc
CALIPSO-ICECLOUD:
tier: 3
source: https://eosweb.larc.nasa.gov/project/CALIPSO/CAL_LID_L3_Ice_Cloud-Standard_V1-00
last_access: 2022-08-04
info: |
Download and processing instructions
(requires EarthData login; see https://urs.earthdata.nasa.gov/)
1) Go to https://eosweb.larc.nasa.gov/project/CALIPSO/CAL_LID_L3_Ice_Cloud-Standard_V1-00
2) Click on "Get Dataset"
3) Select Granules "A" containing both, day and night (Day/Night filter = "Both")
4) Download selected granules
5) Enter EarthData login and password (https://urs.earthdata.nasa.gov/)
6) Follow download instructions in email from EarthData and put all
files in the same directory
CDS-SATELLITE-ALBEDO:
tier: 3
source: https://cds.climate.copernicus.eu/cdsapp#!/dataset/satellite-albedo?tab=form
last_access: 2019-04-01
info: |
You will need to accept the license prior to use the automatic download.
If not, it will fail but will provide you the link to do so.
Download and processing instructions
- Download the data from source to the right directory
using the download script.
- Decompress the files within the directory:
"find . -name '*.tar.gz' -execdir tar -xzvf '{}' \"
CDS-SATELLITE-LAI-FAPAR:
tier: 3
source: https://cds.climate.copernicus.eu/cdsapp#!/dataset/satellite-lai-fapar?tab=form
last_access: 2019-07-03
info: |
You will need to accept the license prior to use the automatic download.
If not, it will fail but will provide you the link to do so.
Download and processing instructions
- Open in a browser the data source as specified above
- Put the right ticks:
- Tick variables LAI and FAPAR
- Tick satellite SPOT (System Pour l'Observation de la Terre)
- Tick sensor VGT (Vegetation)
- Tick horizontal resolution 1km
- Tick product version V1
- Tick all available years
- Tick all available months
- Tick Nominal day 20
- Click 'submit form'
- According to ESMValTool practice, put them in the right rawobsdir folder
- Request might need to be split into chunks to not exceed download limit
- Fails setting standard name for variable FAPAR
CDS-SATELLITE-SOIL-MOISTURE:
tier: 3
source: https://cds.climate.copernicus.eu/cdsapp#!/dataset/satellite-soil-moisture?tab=form
last_access: 2019-03-14
info: |
- Use the download command to retrieve the data.
- Available years: 1992-2019 (ACTIVE) or 1979-2019 (others).
- Versions: v201812.0.0 and v201912.0.0
CDS-UERRA:
tier: 3
source: https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-uerra-europe-soil-levels
last_access: 2019-11-04
info: |
- Open in a browser the data source as specified above
- Put the right ticks:
- Tick Origin UERRA-HARMONIE
- Tick Variable 'Volumetric soil moisture'
- Tick Soil level 1, 2, 3
- Tick all available years
- Tick all available months
- Tick all available days
CDS-XCH4:
tier: 3
source: https://cds.climate.copernicus.eu/cdsapp#!/dataset/satellite-methane?tab=form
last_access: 2019-03-11
info: |
Select Processing level "Level 3", variable "Column-average dry-air mole
fraction of atmospheric methane (XCH4) and related variables", Sensor and
algorithm "MERGED and OBS4MIPS".
A registration is required to download the data.
CDS-XCO2:
tier: 3
source: https://cds.climate.copernicus.eu/cdsapp#!/dataset/satellite-carbon-dioxide?tab=form
last_access: 2019-03-19
info: |
Select Processing level "Level 3", variable "Column-average dry-air mole
fraction of atmospheric carbon dioxide (XCO2) and related variables",
Sensor and algorithm "MERGED and OBS4MIPS".
A registration is required to download the data.
CERES-EBAF:
tier: 2
source: https://ceres-tool.larc.nasa.gov/ord-tool/jsp/EBAFTOA41Selection.jsp
last_access: 2022-07-01
info: |
Select: "TOA Fluxes" (all), "Monthly", "Regional" (0-360, -90-90)
Enter "Email Address" and click on "Get Data"
Wait for the processing to be finished and click on "Download"
CERES-SYN1deg:
tier: 3
source: https://ceres-tool.larc.nasa.gov/ord-tool/jsp/SYN1degSelection.jsp
last_access: 2019-02-07
info: |
Monthly data:
Expand "Compute TOA Fluxes" and select:
Shortwave Flux, Allsky and Clearsky
Longwave Flux, Allsky and Clearsky
Shortwave Flux Down, Allsky
Expand "Computed Surface Fluxes" and select:
Shortwave Flux Up, Allsky and Clearsky
Shortwave Flux Down, Allsky and Clearsky
Longwave Flux Up, Allsky and Clearsky
Longwave Flux Down, Allsky and Clearsky
then click on "Monthly", "Regional" and "Get data". All fields are saved
in CERES_SYN1deg-Month_Terra-Aqua-MODIS_Ed3A_Subset_200003-201702.nc
3hr data:
Select the same fields as above, then click on "Daily 3-Hourly" and
"Get data". All fields are saved in
CERES_SYN1deg-3H_Terra-Aqua-MODIS_Ed3A_Subset_YYYYMMDD-YYYYMMDD.nc
Put all files in input_dir_path (no subdirectories with years).
For orders larger than 2 GB a registration is required.
CLARA-AVHRR:
tier: 3
source: https://wui.cmsaf.eu/
last_access: 2021-03-22
info: |
Download and processing instructions
1) Create ("register") an user account at
https://wui.cmsaf.eu/safira/action/viewLogin?menuName=NUTZER_HOME
2) login (same URL as above)
3) Search data using search form at
https://wui.cmsaf.eu/safira/action/viewProduktHome?menuName=PRODUKT_HOME
- Product group: Climate Data Records
- Product family: CLARA-A ed. 2.1
- Product name: CFC - Factional cloud cover
IWP - Ice water path
LWP - Liquid water path
- Area: Global
- Temporal resolution: Monthly
4) Select "CLARA-A ed. 2.1 AVHRR on polar orbiting satellites" from
list of results.
5) Click on "Add to order cart"
6) Follow download instructions in automatic email received when data
are ready for download.
7) Untar all .tar files into a single directory.
CLOUDSAT-L2:
tier: 3
source: https://www.cloudsat.cira.colostate.edu/
last_access: 2022-08-10
info: |
Download and processing instructions
1) Create an account at the CloudSat Data Processing Center
(https://www.cloudsat.cira.colostate.edu/)
2) Download the datasets '2B-CWC-RO' and '2C-PRECIP-COLUMN' from
www.cloudsat.cira.colostate.edu (via sftp) following the instructions
given here: https://www.cloudsat.cira.colostate.edu/order/sftp-access
3) Put all files for a specific year under a single directory
named like the year (e.g. 2007), no subdirectories with
days etc.
CMAP:
tier: 2
source: https://psl.noaa.gov/data/gridded/data.cmap.html
last_access: 2024-09-09
info: |
To facilitate the download, the links to the https server are provided.
https://downloads.psl.noaa.gov/Datasets/cmap/enh/
precip.mon.mean.nc
CowtanWay:
tier: 2
source: https://www-users.york.ac.uk/~kdc3/papers/coverage2013/series.html
last_access: 2020-02-26
info: |
Download the following files:
'had4_krig_v1_0_0.nc.gz'
'had4_uah_v1_0_0.nc.gz'
'had4_short_krig_v2_0_0.nc.gz'
'had4_short_uah_v2_0_0.nc.gz'
'ghcn_short_krig_v2_0_0.nc.gz'
'ghcn_short_uah_v2_0_0.nc.gz'
'had4sst4_krig_v2_0_0.nc.gz'
'had4_krig_v2_0_0.nc.gz'
CRU:
tier: 2
source: https://crudata.uea.ac.uk/cru/data/hrg/cru_ts_4.07/cruts.2304141047.v4.07/
last_access: 2023-11-06
info: |
Files can be downloaded using the download script (latest version only)
or manually:
{raw_name}/cru_ts4.07.1901.2022.{raw_name}.dat.nc.gz
where {raw_name} is the name of the desired variable(s).
Previous versions can be downloaded from the corresponding folders in
https://crudata.uea.ac.uk/cru/data/hrg/. ESMValTools formatter can be used
for older versions with minor adjustments of
``esmvaltool/cmorizers/data/cmor_config/CRU.yml``
Exact time coordinates and number of stations are not available version
TS4.02.
CT2019:
tier: 2
source: https://www.esrl.noaa.gov/gmd/ccgg/carbontracker/index.php
last_access: 2020-03-23
info: |
Create a new empty directory ``$RAWOBSPATH/Tier2/CT2019`` (where
``$RAWOBSPATH`` is given by your configuration) where the raw
data will be stored. The download of the data is automatically handled by
this script. If data is already present in this directory, the download is
skipped (to force a new download delete your old files).
Duveiller2018:
tier: 2
source: https://ndownloader.figshare.com/files/9969496
last_access: 2019-04-30
info: |
- Download the dataset albedo_IGBPgen.nc and save in the right directory
according to ESMValTool practices.
- Complete the CMOR-config specifications (see instructions in the file
itself)
E-OBS:
tier: 2
source: http://surfobs.climate.copernicus.eu/dataaccess/access_eobs.php#datafiles
last_access: 2020-02-25
info: |
Download the ensemble mean files for:
TG TN TX RR PP
Eppley-VGPM-MODIS:
tier: 2
source: http://orca.science.oregonstate.edu/data/1x2/monthly/eppley.r2018.m.chl.m.sst/hdf
last_access: 2019-05-15
info: |
Download and unpack all the *.tar files under a single directory
(no subdirectories with years) in ${RAWOBS}/Tier2/Eppley-VGPM-MODIS
ERA-Interim-Land:
tier: 3
source: https://apps.ecmwf.int/datasets/data/interim-land/type=fc/
last_access: 2019-11-04
info: |
See script cmorize_obs_era_interim.py
ERA-Interim:
tier: 3
source: http://apps.ecmwf.int/datasets/data/interim-full-moda/
last_access: 2019-09-05
info: |
Select "ERA Interim Fields":
Daily: for daily values
Invariant: for time invariant variables (like land-sea mask)
Monthly Means of Daily Means: for monthly values
Monthly Means of Daily Forecast Accumulation: for accumulated variables
like precipitation or radiation fluxes
Select "Type of level" (Surface or Pressure levels)
Download the data on a single variable and single year basis, and save
them as ERA-Interim_<var>_<mean>_YYYY.nc, where <var> is the ERA-Interim
variable name and <mean> is either monthly or daily. Further download
"land-sea mask" from the "Invariant" data and save it in
ERA-Interim_lsm.nc.
It is also possible to download data in an automated way, see:
https://confluence.ecmwf.int/display/WEBAPI/Access+ECMWF+Public+Datasets
https://confluence.ecmwf.int/display/WEBAPI/Python+ERA-interim+examples
A registration is required for downloading the data.
It is also possible to use the script in:
esmvaltool/cmorizers/data/download_scripts/download_era-interim.py
This cmorization script currently supports daily and monthly data of
the following variables:
10m u component of wind
10m v component of wind
2m dewpoint temperature
2m temperature
evaporation
maximum 2m temperature since previous post processing
mean sea level pressure
minimum 2m temperature since previous post processing
skin temperature
snowfall
surface net solar radiation
surface solar radiation downwards
temperature of snow layer
toa incident solar radiation
total cloud cover
total precipitation
and daily, monthly (not invariant) data of:
Geopotential
and monthly data of:
Fraction of cloud cover (3-dim)
Inst. eastward turbulent surface stress
Inst. northward turbulent surface stress
Net top solar radiation
Net top solar radiation clear-sky
Sea surface temperature
Specific cloud ice water content
Specific cloud liquid water content
Specific humidity
Surface net thermal radiation
Surface latent heat flux
Surface sensible heat flux
Relative humidity
Temperature
Top net thermal radiation clear-sky
Total column water wapour
U component of wind
V component of wind
Vertical integral of cloud condensed water (ice and liquid)
Vertical integral of cloud liquid water
Vertical integral of cloud frozen water
Vertical velocity
Caveats
Make sure to select the right steps for accumulated fluxes, see:
https://confluence.ecmwf.int/pages/viewpage.action?pageId=56658233
https://confluence.ecmwf.int/display/CKB/ERA-Interim%3A+monthly+means
for a detailed explanation.
The data are updated regularly: recent years are added, but also the past
years are sometimes corrected. To have a consistent timeseries, it is
therefore recommended to download the full timeseries and not just add
new years to a previous version of the data.
For further details on obtaining daily values from ERA-Interim,
see:
https://confluence.ecmwf.int/display/CKB/ERA-Interim
https://confluence.ecmwf.int/display/CKB/ERA-Interim+documentation#ERA-Interimdocumentation-Monthlymeans
https://confluence.ecmwf.int/display/CKB/ERA-Interim%3A+How+to+calculate+daily+total+precipitation
ERA5:
tier: 3
source: https://cds.climate.copernicus.eu
last_access: 2021-11-29
info: |
Reformat preprocessor output from the native6 project to follow the OBS6
conventions.
ESACCI-AEROSOL:
tier: 2
source: ftp://anon-ftp.ceda.ac.uk/neodc/esacci/aerosol/data/
last_access: 2019-01-24
info: |
Download the data from:
ATSR2_SU/L3/v4.21/MONTHLY/ (1997-2002)
AATSR_SU/L3/v4.21/MONTHLY/ (2003-2011)
Other years are not considered since they are not complete.
Put all files in input_dir_path (no subdirectories with years).
ESACCI-CLOUD:
tier: 2
source: https://public.satproj.klima.dwd.de/data/ESA_Cloud_CCI/CLD_PRODUCTS/v3.0/
last_access: 2019-02-01
info: |
Download the data from:
L3C/AVHRR-PM/
To fill the gap 199409-199501, also download:
L3C/AVHRR-AM/AVHRR_NOAA-12/1994/ \
199409-ESACCI-L3C_CLOUD-CLD_PRODUCTS-AVHRR_NOAA-12-fv3.0.nc
L3C/AVHRR-AM/AVHRR_NOAA-12/1994/ \
199410-ESACCI-L3C_CLOUD-CLD_PRODUCTS-AVHRR_NOAA-12-fv3.0.nc
L3C/AVHRR-AM/AVHRR_NOAA-12/1994/ \
199411-ESACCI-L3C_CLOUD-CLD_PRODUCTS-AVHRR_NOAA-12-fv3.0.nc
L3C/AVHRR-AM/AVHRR_NOAA-12/1994/ \
199412-ESACCI-L3C_CLOUD-CLD_PRODUCTS-AVHRR_NOAA-12-fv3.0.nc
L3C/AVHRR-AM/AVHRR_NOAA-12/1995/ \
199501-ESACCI-L3C_CLOUD-CLD_PRODUCTS-AVHRR_NOAA-12-fv3.0.nc
Put all files under a single directory (no subdirectories with years).
ESACCI-FIRE:
tier: 2
source: ftp://anon-ftp.ceda.ac.uk/neodc/esacci/fire/data/
last_access: 2019-01-24
info: |
Download the data from:
burned_area/MERIS/grid/v4.1/
Put all files in input_dir_path (no subdirectories with years).
ESACCI-LANDCOVER:
tier: 2
source: ftp://anon-ftp.ceda.ac.uk/neodc/esacci/land_cover/data/pft/v2.0.8/
last_access: 2024-07-11
info: |
Download and processing instructions:
Use the following CLI to download all the files:
esmvaltool data download ESACCI-LANDCOVER
The underlying downloader is located here:
/ESMValTool/esmvaltool/cmorizers/data/downloaders/datasets/esacci_landcover.py
and it will download all the files currently available on CEDA (1992-2020)
under a single directory as follow: ${RAWOBS}/Tier2/ESACCI-LANDCOVER
ESACCI-LST:
tier: 2
source: On CEDA-JASMIN, /gws/nopw/j04/esacci_lst/public
last_access: 2022-01-26
info: |
For access to this JASMIN group workspace please register at:
https://accounts.jasmin.ac.uk/services/group_workspaces/esacci_lst/
Download and processing instructions:
Put all files under a single directory (no subdirectories with years)
in ${RAWOBS}/Tier2/ESACCI-LST
BOTH DAY and NIGHT files are needed for each month
Caveats
Currently set to work with only the MODIS AQUA L3 monthly data
ESACCI-OC:
tier: 2
source: ftp://oceancolour.org/occci-v5.0/geographic/netcdf/monthly/chlor_a/
last_access: 2021-02-02
info: |
In case of issues with data download, check also the information provided at
OceanColour webpage https://esa-oceancolour-cci.org/
Username and password are provided on this website
Put all files under a single directory (no subdirectories with years)
in ${RAWOBS}/Tier2/ESACCI-OC
ESACCI-OZONE:
tier: 2
source: ftp://anon-ftp.ceda.ac.uk/neodc/esacci/ozone/data/
last_access: 2019-02-01
info: |
Download the data from:
total_columns/l3/merged/v0100/
limb_profiles/l3/merged/merged_monthly_zonal_mean/v0002
Put all files under a single directory (no subdirectories with years).
ESACCI-SOILMOISTURE:
tier: 2
source: ftp://anon-ftp.ceda.ac.uk/neodc/esacci/soil_moisture/data/
last_access: 2024-06-19
info: |
Download the data from:
daily_files/COMBINED/v08.1/
ancillary/v08.1/
Put all files under a single directory (no subdirectories with years).
ESACCI-SEA-SURFACE-SALINITY:
tier: 2
source: ftp://anon-ftp.ceda.ac.uk/neodc/esacci/sea_surface_salinity/data
last_access: 2020-09-21
info: |
Download the data from:
v01.8/30days/
v02.31/30days/
Put all files under a single directory (no subdirectories with years).
ESACCI-SST:
tier: 2
source: ftp://anon-ftp.ceda.ac.uk/neodc/esacci/sst/data/
last_access: 2019-02-01
info: |
Download the data from:
lt/Analysis/L4/v01.1/
Put all files under a single directory (no subdirectories with years).
ESACCI-WATERVAPOUR:
tier: 3
source: https://wui.cmsaf.eu/safira/action/viewDoiDetails?acronym=COMBI_V001
last_access: 2024-02-21
info: |
CDR2 requires registration at EUMETSAT CM SAF, the information on how to
download the order will be emailed once the order is ready.
All files need to be in one directory, not in yearly subdirectories.
ESDC:
tier: 2
source: http://data.rsc4earth.de/EarthSystemDataCube/
last_access: 2023-01-26
info: |
It is not necessary to download the data, as the cmorizer script can access
it directly from the cloud if it is not available locally.
To download a dataset, the dataset folder can be explored on the source
website, and downloaded using wget:
```wget -m -nH -np -R "index.html*" http://data.rsc4earth.de/EarthSystemDataCube/v3.0.1/```
ESRL:
tier: 2
source: http://www.esrl.noaa.gov/gmd/dv/data/index.php
last_access: 2020-06-30
info: |
(1) Select the following filters:
Category: Greenhouse Gases.
Parameter name: Carbon Dioxide.
Type: Insitu.
Frequency: Monthly Averages.
Site: BRW, MLO, SMO, SPO.
(2) Get stations properties at http://www.esrl.noaa.gov/gmd/obop/
and compile a station list as comma separated value table containing:
station code, station name, latitude, longitude (0-360), elevation (m)
Example (ESRL_stations.csv):
----------------------------
BRW, Barrow Alaska, 71.3230, 203.3886, 11
THD, Trinidad Head, 41.0541, 235.849, 107
SUM, Summit Greenland, 72.5962, 321.578, 3210
MLO, Mauna Loa, 19.5362, 204.4237, 3397
SMO, American Samoa, -14.2474, 189.4356, 42
SPO, South Pole, -90.00, 301.0, 2840
(3) This list must be manually created (format: comma separated values)
and saved to the path + filename given by "FILE_STAT" below.
(4) Check the period covered by the data and give it below as YEAR1 and
YEAR2.
FLUXCOM:
tier: 3
source: http://www.bgc-jena.mpg.de/geodb/BGI/Home
last_access: 2019-07-27
info: |
From the website, select FLUXCOM as the data choice and click download.
Two files will be displayed. One for Land Carbon Fluxes and one for
Land Energy fluxes. The Land Carbon Flux file (RS + METEO) using
CRUNCEP data file has several data files for different variables.
The data for GPP generated using the
Artificial Neural Network Method will be in files with name:
GPP.ANN.CRUNCEPv6.monthly.*.nc
A registration is required for downloading the data.
Users in the UK with a CEDA-JASMIN account may request access to the jules
workspace and access the data.
Note : This data may require rechunking of the netcdf files.
This constraint will not exist once iris is updated to
version 2.3.0 Aug 2019
GCP2018:
tier: 2
source: https://www.icos-cp.eu/GCP/2018
last_access: 2019-10-17
info: |
Download the following file: '2018 Global Budget v1.0'
GCP2020:
tier: 2
source: https://www.icos-cp.eu/GCP/2020
last_access: 2019-10-17
info: |
Download the following file: '2020 Global Budget v1.0'
GHCN-CAMS:
tier: 2
source: |
https://www.esrl.noaa.gov/psd/data/gridded/data.ghcncams.html
ftp://ftp.cdc.noaa.gov/Datasets/ghcncams/air.mon.mean.nc
last_access: 2020-03-04
info: Use automatic download feature to get the data
GHCN:
tier: 2
source: https://www.esrl.noaa.gov/psd/data/gridded/data.ghcngridded.html
last_access: 2019-03-08
info: |
Download the dataset "precip.mon.total.nc" (precipitation, total, surface,
1900-2015 on a 5x5 grid).
GISTEMP:
tier: 2
source: https://data.giss.nasa.gov/gistemp/ https://data.giss.nasa.gov/pub/gistemp/gistemp250_GHCNv4.nc.gz
last_access: 2020-03-03
info: "Use automatic download feature to get the data"
GLODAP:
tier: 2
source: https://www.glodap.info/index.php/mapped-data-product/
last_access: 2020-03-03
info: "Use automatic download feature to get the data"
GPCC:
tier: 2
source: |
https://opendata.dwd.de/climate_environment/GPCC/html/fulldata-monthly_v2018_doi_download.html
https://opendata.dwd.de/climate_environment/GPCC/full_data_2018/full_data_monthly_v2018_[025 05 10 25].nc.gz
last_access: 2020-02-25
info: |
Download the following files:
full_data_monthly_{version}.nc.gz
GPCP-SG:
tier: 2
source: |
https://psl.noaa.gov/data/gridded/data.gpcp.html
https://downloads.psl.noaa.gov/Datasets/gpcp/precip.mon.mean.nc
last_access: 2023-02-15
info: |
Download the file precip.mon.mean.nc
GRACE:
tier: 3
source: https://podaac.jpl.nasa.gov/dataset/TELLUS_GRAC-GRFO_MASCON_CRI_GRID_RL06_V2
last_access: 2020-11-27
info: |
Download and processing instructions
- Go to the source link
- Click the tab "Data Access"
- Log in with Earthdata account
- Download the following files:
- CLM4.SCALE_FACTOR.JPL.MSCNv02CRI.nc
- GRCTellus.JPL.200204_202004.GLO.RL06M.MSCNv02CRI.nc
- LAND_MASK.CRI.nc
- Download the grace months table which holds important information
on data coverage. Save it in the RAWOBSDIR.
https://podaac-tools.jpl.nasa.gov/drive/files/allData/tellus/L3/docs/GRACE_GRACE-FO_Months_RL06.csv
- Manually inspect and check the months table
HadCRUT3:
tier: 2
source: http://www.metoffice.gov.uk/hadobs/hadcrut3/data/download.html
last_access: 2019-02-21
info: |
Download the HadCRUT3v.nc file (variance adjusted dataset).
Caveats
The HadCRUT3v variance-adjusted dataset for tas is actually the anomaly
with respect to the period 1958-2001.
HadCRUT4:
tier: 2
source: https://crudata.uea.ac.uk/cru/data/temperature/
last_access: 2019-02-08
info: |
Download the dataset "HadCRUT4" (median temperature anomalies) and
the dataset "Absolute" (absolute temperatures for the base period
1961-90 on a 5x5 grid).
Caveats
In contrast to the HadCRUT3 reformat script which produces temperature
anomalies (relative to the 1961-90 climatology), this script calculates
absolute tas by adding the climatology ("absolute.nc") to the anomalies
("HadCRUT.4.6.0.0.median.nc"). It creates 2 output, one with the
temperature time-series and one with the anomaly time-series
HadCRUT5:
tier: 2
source: https://crudata.uea.ac.uk/cru/data/temperature
last_access: 2022-03-28
info: |
Download the following files:
infilling
[Source]/HadCRUT.5.0.1.0.analysis.anomalies.ensemble_mean.nc
no-infilling
[Source]/HadCRUT.5.0.1.0.anomalies.ensemble_mean.nc
climatology
[Source]/absolute_v5.nc
HadISST:
tier: 2
source: http://www.metoffice.gov.uk/hadobs/hadisst/data/download.html
last_access: 2019-02-08
info: |
Download and unzip "HadISST_ice.nc.gz" and "HadISST_sst.nc.gz".
HALOE:
tier: 2
source: Grooss, J.-U. and Russell III, J. M., Atmos. Chem. Phys., 5, 2797-2807, doi:10.5194/acp-5-2797-2005, 2005.
last_access: 2020-03-11
info: |
Download and processing instructions
Download and untar the supplementary material of the above reference.
Caveats
The time period includes incomplete years. According to the given
reference (Section 3): "Between October 1991 and August 2002, a total of
78600 HALOE profiles of O3, CH4, H2O, HCl, and HF mixing ratio were then
combined into bins of 5 degree equivalent latitude for 22 pressure levels
between 316 and 0.1 hPa and 12 months".
In order to create T3M fields a fake longitude coordinate is added.
HWSD:
tier: 3
source: https://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=1247
last_access: 2019-10-15
info: |
Download the following file:
HWSD_SOIL_CLM_RES.nc4
A registration is required
ISCCP-FH:
tier: 2
source: https://isccp.giss.nasa.gov/pub/flux-fh/tar-nc4_MPF/
last_access: 2019-11-07
info: |
To use the automatic download set your credentials in your .netrc file.
See https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html.
To download manually:
1) Go to https://isccp.giss.nasa.gov/projects/flux.html and click on
"FH On-line Data"
2) username and password are provided on this website
3) go to directory tar-nc4_MPF/ (= monthly files)
3) download and unpack files
JMA-TRANSCOM:
tier: 3
source: http://www.globalcarbonatlas.org/en/content/atmospheric-inversions
last_access: 2019-07-02
info: |
To obtain the data sets it is necessary to contact Takashi Maki
(Department of Atmosphere, Ocean and Earth System Modeling Research,
Meteorological Research Institute, Tsukuba City, Japan). See link above
for more information.
JRA-25:
tier: 2
source: https://esgf.nccs.nasa.gov/thredds/fileServer/CREATE-IP/reanalysis/JMA/JRA-25/JRA-25/
last_access: 2022-11-17
info: |
Download the following files:
mon/atmos/clt/clt_Amon_reanalysis_JRA-25_197901-201312.nc
mon/atmos/hus/hus_Amon_reanalysis_JRA-25_197901-201312.nc
mon/atmos/prw/prw_Amon_reanalysis_JRA-25_197901-201312.nc
mon/atmos/rlut/rlut_Amon_reanalysis_JRA-25_197901-201312.nc
mon/atmos/rlutcs/rlutcs_Amon_reanalysis_JRA-25_197901-201312.nc
mon/atmos/rsut/rsut_Amon_reanalysis_JRA-25_197901-201312.nc
mon/atmos/rsutcs/rsutcs_Amon_reanalysis_JRA-25_197901-201312.nc
JRA-55:
tier: 2
source: https://rda.ucar.edu/datasets/ds628.1/
last_access: 2023-03-22
info: |
Create an account on the research data archive (RDA) in order to be able
to download the data (1.25 degree, pressure levels). See
https://rda.ucar.edu/login/register/ for more details.
Kadow2020:
tier: 2
source: http://users.met.fu-berlin.de/~ChristopherKadow/
last_access: 2022-03-29
info: |
Download the following file:
[SOURCE]/HadCRUT.5.0.1.0.anomalies.Kadow_et_al_2020_20crAI-infilled.ensemble_mean_185001-202012.nc
LAI3g:
tier: 3
source: http://cliveg.bu.edu/modismisr/lai3g-fpar3g.html
last_access: 2019-05-03
info: |
To obtain the data sets it is necessary to contact Ranga B. Myneni
(Department of Earth and Environment, Boston University). See link above
for more information.
By default, this dataset is regridded to a 1°x1° grid (original resolution
is 1/12°). If you want to use the original resolution, remove the `regrid`
section in the configuration file (`LAI3g.yml`). Note that in this case,
preprocessing the dataset with ESMValTool (i.e. every time you run the
tool) can take a very long time (> 30 min).
LandFlux-EVAL:
tier: 3
source: https://data.iac.ethz.ch/landflux/
last_access: 2019-05-16
info: |
To use the automatic download set your credentials in your .netrc file.
See https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html.
Download the following files:
LandFluxEVAL.merged.89-05.monthly.all.nc
A registration is required for downloading the data (see
<http://www.iac.ethz.ch/group/land-climate-dynamics/research/
landflux-eval.html>).
Landschuetzer2016:
tier: 2
source: https://www.nodc.noaa.gov/archive/arc0105/0160558/3.3/data/0-data/
last_access: 2019-03-08
info: |
Download the file spco2_1982-2015_MPI_SOM-FFN_v2016.nc
Landschuetzer2020:
tier: 2
source: https://www.ncei.noaa.gov/data/oceans/ncei/ocads/data/0209633/
last_access: 2022-11-02
info: |
Download the file MPI-ULB-SOM_FFN_clim.nc
MAC-LWP:
tier: 3
source: https://search.earthdata.nasa.gov/search?q=MAC-LWP
last_access: 2020-01-30
info: |
Select the MACLWP-mean dataset.
Download the script file for both projects.
Download the data using http either by selecting each granule
individually or by using the option "download all".
Data is freely available, but a registration is required.
MERRA:
tier: 3
source: https://goldsmr3.gesdisc.eosdis.nasa.gov/data/MERRA_MONTHLY/
last_access: 2023-02-01
info: |
Use automatic download. That will download monthly data.
MERRA2:
tier: 3
source: https://goldsmr4.gesdisc.eosdis.nasa.gov/data/MERRA2_MONTHLY/ https://goldsmr5.gesdisc.eosdis.nasa.gov/data/MERRA2_MONTHLY/
last_access: 2022-09-13
info: |
Use automatic download. That will download monthly data but with
yearly granularity. Note that some (most) variables are on the goldsmr4 server,
whereas others are on the goldsmr5 server.
MLS-AURA:
tier: 3
source: https://disc.gsfc.nasa.gov/datasets/ML2RHI_004/summary https://disc.gsfc.nasa.gov/datasets/ML2T_004/summary
last_access: 2020-02-03
info: |
Select "Data Access" -> "Subset/Get Data" -> "Get Data" and follow the
"Instructions for downloading". All *.he5 files need to be saved in the
$RAWOBS/Tier3/MLS-AURA directory, where $RAWOBS refers to the RAWOBS
directory defined in the configuration. Apply this procedure to both
links provided above. The temperature fields are necessary for quality
control of the RHI data (see Data Quality Document for MLS-AURA for more
information).
A registration is required.
MOBO-DIC_MPIM:
tier: 2
source: https://www.ncei.noaa.gov/data/oceans/ncei/ocads/data/0221526/
last_access: 2022-11-03
info: |
Download the file MOBO-DIC_MPIM_monthly_clim.nc
MOBO-DIC2004-2019:
tier: 2
source: https://www.nodc.noaa.gov/archive/arc0211/0277099/2.3/data/0-data/
last_access: 2023-10-09
info: |
Download the file MPI_MOBO-DIC_2004-2019_v2.nc
MODIS:
tier: 3
source: https://ladsweb.modaps.eosdis.nasa.gov/search/order
last_access: 2019-02-09
info: |
In Products: select "MODIS Aqua", "Collection 6.1" and
"L3 Atmosphere Product", click on MYD08_M3.
In Time: select from 2000-01-01 to today.
In Location: skip, the global domain will be applied.
In Files: select all.
Submit the order.
A registration is required to download the data.
Caveats
clwvi and clivi data are in-cloud values whereas CMIP5 models provide
grid-box averages --> multiply MODIS clwvi and clivi values with cloud
fraction as a first guess
MTE:
tier: 3
source: http://www.bgc-jena.mpg.de/geodb/BGI/Home
last_access: 2019-05-07
info: |
Download the following files:
EnsembleGPP_GL.nc
A registration is required for
NCEP-DOE-R2:
tier: 2
source: https://psl.noaa.gov/data/gridded/data.ncep.reanalysis2.html
last_access: 2022-09-06
info: |
To facilitate the download, the links to the https server are provided.
https://downloads.psl.noaa.gov/Datasets/ncep.reanalysis2/Monthlies/
pressure/
rhum.mon.mean.nc
air.mon.mean.nc
omega.mon.mean.nc
https://downloads.psl.noaa.gov/Datasets/ncep.reanalysis2/Monthlies/
gaussian_grid
tcdc.eatm.mon.mean.nc
prate.sfc.mon.mean.nc
uflx.sfc.mon.mean.nc
vflx.sfc.mon.mean.nc
skt.sfc.mon.mean.nc
https://downloads.psl.noaa.gov/Datasets/ncep.reanalysis2/Monthlies/
surface
pr_wtr.eatm.mon.mean.nc
NCEP-NCAR-R1:
tier: 2
source: https://psl.noaa.gov/data/gridded/data.ncep.reanalysis.html
last_access: 2022-11-22
info: |
To facilitate the download, the links to the ftp server are provided.
Since the filenames are sometimes identical across different
save the data in two subdirectories in input_dir_path.
Subdirectory pressure/:
ftp://ftp.cdc.noaa.gov/Projects/Datasets/data.ncep.reanalysis/pressure/
air.mon.mean.nc
hgt.mon.mean.nc
rhum.mon.mean.nc
shum.mon.mean.nc
uwnd.mon.mean.nc