-
Notifications
You must be signed in to change notification settings - Fork 5
Speeding up MERRA data access
Nick edited this page Feb 14, 2022
·
1 revision
MERRA datasets are subdivided into daily files. This means that requests over long time periods must make hundreds of serial server requests. The time to connect to the server is often much greater than the time to download the dataset.
following the instructions here for creating cookie files: if you get a bunch of MERRA URLs using a modified MERRA subsetter, you can download them in (kind-of) parallel with the following:
cat ../merra-wishlist.txt | xargs -n 1 -P 8 wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition
where -P 8
uses 8 threads (could use more - I haven't tried)
then you need to merge them together
cdo mergetime MERRA2_*00.inst3_3d_asm_Np* globsim_merra2.inst3_3d_asm.nc
cdo mergetime MERRA2_*00.inst6_3d_ana_Np* globsim_merra2.inst6_3d_ana_Np.nc
cdo mergetime MERRA2_*00.inst3_3d_asm_Np* globsim_merra2.inst3_3d_asm_Np.nc
cdo mergetime MERRA2_*00.inst1_2d_asm_Nx* globsim_merra2.inst1_2d_asm_Nx.nc
cdo mergetime MERRA2_*00.tavg1_2d_flx_Nx* globsim_merra2.tavg1_2d_flx_Nx.nc
cdo mergetime MERRA2_*00.tavg1_2d_rad_Nx* globsim_merra2.tavg1_2d_rad_Nx.nc
cdo mergetime MERRA2_*00.tavg1_2d_slv_Nx* globsim_merra2.tavg1_2d_slv_Nx.nc
probably want to include the time range here. You could and delete the old ones