Extend the CIME workflow to support user-specified initial conditions flexibly #37
Comments
arunchawla-NOAA
added this to the Internal testing of end to end system using CIME milestone
|
@jedwards4b @rsdunlapiv when workflow is activated the case.run triggers buildnml and buildnml overwrites the files produced by chgres. i am not sure this was the case for the prototype that we had before. i think that case.submit must run the buildnml not case.run. there is no need to run buildnml in every task in the workflow. of course, we could modify the buildnml not to overwrite files produced by chgres but i am not sure about best way. Anyway, in the current version i put a control to the section that copies/links input files to check the files to prevent overwriting the existing files but we could change this design. |
|
@rsdunlapiv i added IC_DIR environment variable to the machine.xml file to point the initial condition directory. The directory must follow the standard indicated as below,
|
|
Post-processing is also implemented. Now, we have initial version of end to end workflow. We still need followings
The ufs-mrweather-app on ESCOMP is updated if you want to test it on Cheyenne. The successful run can be found in /glade/scratch/turuncu/ufs-mrweather-app.v16beta/run |
|
I could run the whole workflow with different initial condition successfully with minor modifications in buildnml. I'll update app after testing different combinations and resolutions. /glade/p/cesmdata/cseg/ufs_inputdata/IC/gfs.20191224/12 |
|
@ufuk Turuncoglu <[email protected]> - that's really good news and great
progress!
…On Thu, Dec 26, 2019 at 1:20 PM Ufuk Turunçoğlu ***@***.***> wrote:
I could run the whole workflow with different initial condition
successfully with minor modifications in buildnml. I'll update app after
testing different combinations and resolutions.
/glade/p/cesmdata/cseg/ufs_inputdata/IC/gfs.20191224/12
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#37?email_source=notifications&email_token=AB4XCEYNMPIB572YYSFD4CLQ2UGZFA5CNFSM4J5JVKIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHWDETA#issuecomment-569127500>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB4XCE6PAJ2MGBWEZNN7BKDQ2UGZFANCNFSM4J5JVKIA>
.
|
|
@uturuncoglu needs to remove the IC_DIR environment variable and implement access to NOMADS server to download raw input files for chgres. This will only support the datasets currently available in NOMADS - the user is responsible for pulling in their own input files for dates not available in NOMADS. @jedwards4b We need to add a DIN_IC_ROOT to env_run.xml. If DIN_IC_ROOT is not set, it will default to $DIN_LOC_ROOT/IC. If the user pulls down their own inputs to chgres, then they can set DIN_IC_ROOT to their own local input directory if needed. |
|
@rsdunlapiv IC_DIR environment variable is removed and access to NOMADS server to download raw input files for chgres is implemented. The model looks the specific location in $DIN_IC_ROOT/prod and try to find desired input folder which is a sub-directory under prod/ as gfs.YYYYMMDD/HH. If it could not find the folder and files that are suitable for model start time, then it goes to nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/ and look for the desired files. |
|
@uturuncoglu to send request to @KateFriedman-NOAA to place default raw inputs to the FTP server that will be used as the default start date. We also need additional testing of the NOMADS retrieval. |
|
@KateFriedman-NOAA Do you have access to Cheyenne or Stampede2? The data is in following directories, Cheyenne: Stampede2: The data need to be placed as prod/gfs.20190909/00. @rsdunlapiv could also copy files to NOAA Hera machine if you don't have access to any of the above platforms. |
|
@uturuncoglu I don't have access to Cheyenne or Stampede2 unfortunately. If @rsdunlapiv could put the files on Hera I can grab them from there, thanks! |
|
@KateFriedman-NOAA and @uturuncoglu I started the copy to Hera. It looks like it's going to take a while. I'll update you when it's complete. |
|
@KateFriedman-NOAA the files are on Hera: There are two files: |
|
@rsdunlapiv Thanks, once Hera is back from today's maintenance I'll copy the files from there to our ftp server. Stay tuned... |
|
@KateFriedman-NOAA I was copying following files from the model source before but the new version of the model does not have those files under pram/ folder. So, we need to place them also in the FTP postxconfig-NT.txt If it is possible could you also put them to FTP or i could put them inside of CIME interface of FV3 (like i did for *_table files) but i am not sure those files changes in time. So, what do you think? |
|
@rsdunlapiv I'm in the process of copying the sample inputs over to WCOSS from Hera and then up to our ftp server. The atmanl file is huge so it's taking a while... @uturuncoglu Those two files are post parm files:
If $GRIBVERSION = 'grib2' & anl file: postxconfig-NT-GFS-ANL.txt Do you have the postxconfig-NT*.txt files in the post associated with the release?
The G2TMPL_SRC variable comes from the g2tmpl module. Is there a copy of the g2tmlp library with the release? |
|
@rsdunlapiv The sample input files are now on the ftp server: https://ftp.emc.ncep.noaa.gov/EIB/UFS/inputdata/prod/gfs.20190909/00/ |
So, i could copy the files from NCEPLIBS but NCEPLIBS installation could change the based on the used platform and i prefer to include those files to CIME. At this point, i just wonder that are those files changes regularly? |
|
@uturuncoglu I can't speak to the params_grib2_tbl_new file but the post postxconfig-*txt files do change from time to time when new fields are added. If the release is a frozen set it's not a concern but if you're setting up something that can evolve with time I recommend not copying the files into CIME but rather establishing links/references like we do in global-workflow. For the params_grib2_tbl_new file from the g2tmpl module we have separate module files for each platform that defines and loads the module so at runtime the g2tmpl module gets loaded and the value of $POSTGRB2TBL is known. See the module_base.* files here in global-workflow for examples: https://github.com/NOAA-EMC/global-workflow/tree/develop/modulefiles |
|
@KateFriedman-NOAA which postxconfig-NT-* file in NCEPLIBS-post/parm/ directory was in the model source param/ directory before? I was copying postxconfig-NT.txt in that time. I think i just need to copy that file because the workflow only post-process FV3 output. |
|
@uturuncoglu What do you mean when you say "model source param/ directory"? Which path are you talking about specifically? The postxconfig-NT-GFS.txt file is what should be used for all forecast hours except for the anl and f000 files. The postxconfig-NT-GFS-ANL.txt file is for processing the anl file and the postxconfig-NT-GFS-F00.txt file is for processing the f000 file. |
|
Thanks for your help. Sorry i am little bit confused. I was using following files for post https://github.com/ufs-community/ufs-weather-model/blob/3bc41ffde579516773549079fa00929f802b218a/parm/postxconfig-NT.txt and those files were included into ufs-weather-model/parm directory before but in the current version of model they are not. So, i am trying to find the correct place to reach those files. As you can see, the file name was postxconfig-NT.txt and it is hard for me to find the correct one from the list of files that are found in the NCEPLIBS-post/parm/. So, if you point me the correct one, i could try to use it. My other concern is that, we design the workflow to process FV3 output using NCEP Post at the end of the simulation. I could both process nemsio and netcdf at this point to create grib files. In this case, do i need to use different postxconfig file for 000 and different for rest of the files such as 001 etc.? There was no such information given to us when we start to design the workflow and this implementation might require additional change in our side. |
|
BTW, there is no ANL file when you run the model. |
|
@uturuncoglu Ah I see, ok so in the post scripts there are if conditions that determine which postxconfig-NT*.txt file to grab and it ends up with the name "postxconfig-NT.txt" after being copied, which is what you're seeing. So you want the postxconfig-NT-GFS.txt file for f001+ and the postxconfig-NT-GFS-F00.txt file for f000. I do not know if the new inline post differentiates between f000 and f001+. Let me see if I can find out... |
|
@uturuncoglu Scratch my comment about inline post, we aren't including inline post in this release. @arunchawla-NOAA @junwang-noaa Does the post for the UFS release differentiate between f000 and the other forecast hours when setting the postxconfig-NT.txt file? Can someone point me to the copy of the post being included in this release? I can check what the scripts are doing. Thanks! |
|
@KateFriedman-NOAA Thanks. That is really valuable information for the person outside of NOAA. I was using single file before. If you don't mind could you put those files to FTP with some kind of versioning. If i try to get them from NCEPLIBS-post/parm/ directory it would break the workflow if someone install the library and delete the source or having some kind of custom installation. It is better to have hem in a public place that everybody could reach it. Then, i'll make the modification in the post script and CIME to use this updated information. Do you think you have any other information related with post that could help us to fix or improve? Yes, the inline post is not the part of the release but we still need those files for post processing of the simulation output. |
|
Current POST allows users to use different post control file by setting the
post control file name in the post configuration file: itag.
…On Thu, Jan 16, 2020 at 2:17 PM Kate Friedman ***@***.***> wrote:
@uturuncoglu <https://github.com/uturuncoglu> Scratch my comment about
inline post, we aren't including inline post in this release.
@arunchawla-NOAA <https://github.com/arunchawla-NOAA> @junwang-noaa
<https://github.com/junwang-noaa> Does the post for the UFS release
differentiate between f000 and the other forecast hours when setting the
postxconfig-NT.txt file? Can someone point me to the copy of the post being
included in this release? I can check what the scripts are doing. Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#37?email_source=notifications&email_token=AI7D6TMWBNXSXQQBBOARA7TQ6CXFLA5CNFSM4J5JVKIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJFGW4Y#issuecomment-575302515>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AI7D6TIZAMUVZUZ6OBBJKHDQ6CXFLANCNFSM4J5JVKIA>
.
|
|
@uturuncoglu I have created a parm folder on the ftp server and placed the two postxconfig-NT-GFS*.txt files there: https://ftp.emc.ncep.noaa.gov/EIB/UFS/parm/postxconfig-NT-GFS.txt |
|
@KateFriedman-NOAA Thanks. Is it possible also put params_grib2_tbl_new to new folder. |
|
@uturuncoglu Done: https://ftp.emc.ncep.noaa.gov/EIB/UFS/parm/params_grib2_tbl_new That is the file from g2tmpl v1.5.0. |
|
@KateFriedman-NOAA Thank you for your kindly help. Do we need to put a versioning to those files at this point? |
|
Yes!
…On Fri, Jan 17, 2020 at 10:16 AM Ufuk Turunçoğlu ***@***.***> wrote:
@KateFriedman-NOAA <https://github.com/KateFriedman-NOAA> Thank you for
your kindly help. Do we need to put a versioning to those files at this
point?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#37?email_source=notifications&email_token=ABOXUGAVNGIECWLTOIP6TETQ6HRVJA5CNFSM4J5JVKIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJILXCI#issuecomment-575716233>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABOXUGBG3SKDKPTB3Y5FTGLQ6HRVJANCNFSM4J5JVKIA>
.
--
Jim Edwards
CESM Software Engineer
National Center for Atmospheric Research
Boulder, CO
|
@junwang-noaa which parameter that could be used to set post control file name? Is it introduced recently? I defined all namelist options in the CIME side using XML file but i could not see any option for it. If you let me know the exact name of the parameter that would be great! |
|
@uturuncoglu I created a lib folder and mimicked the structure of the g2tmpl library on WCOSS. You can now find that parm file here with a version folder included: https://ftp.emc.ncep.noaa.gov/EIB/UFS/lib/g2tmpl/v1.5.0/src/params_grib2_tbl_new Is the g2tmpl library not included in this release package somewhere with NCEPLIBS? |
|
It is included as NCEPLIBS-g2tmpl and it has a params_grib2_tbl_new. |
|
@junwang-noaa I think i found it, it seems it is fileNameFlat and read from stdin |
@uturuncoglu Cool, can CIME get that parm file from there then? I'm anxious about maintaining copies outside of the packaged libraries. |
|
@KateFriedman-NOAA I am totally agree with your concern about keeping parm files outside of the source such as FTP but this particular case, i would like to get them from FTP because it is not easy (or possible) to reach the source directory of library installations all the time. So, FTP will be better solution at this point. |
|
@uturuncoglu Understood, I'll stand down from my concerns. Thanks! @WenMeng-NOAA @HuiyaChuang-NOAA What is the version number of the post being included in the UFS release? Thanks! |
|
@KateFriedman-NOAA The UPP branch "ufs_public_release" is for the UFS release. |
@WenMeng-NOAA Thanks for the branch name but is there a version # I can attach to it (v#.#.#)? Perhaps the associated operational version # that best matches the postxconfig-NT*.txt files in that branch? It has been requested that everything we put on the ftp server for the release include version numbers. I am posting copies of the postxconfig-NT*.txt files. Thanks! |
|
@KateFriedman-NOAA As I know so far, there is no version number created. The DTC UPP team has been working on UPP for the UFS release. I include @fossell. She might provide you further information. |
|
Thanks @WenMeng-NOAA ! |
|
@rsdunlapiv what is the status of this ticket? |
|
I think that this issue is resolved. A new xml variable DIN_LOC_IC was introduced which is the local location of input files to the chgres program. The user sets RUN_STARTDATE in the case. Cime will check the DIN_LOC_IC directory for inputs for that date, if they are not found cime will check the NOMAD server for that date and download the files if available. |
|
I confirm that setting DIN_LOC_IC for custom initial condition path works without any problem. |
A clear mechanism is needed in CIME for how the user should handle specifying which initial conditions to use (e.g., a forecast starting date and a set of input file to chgres) and ensuring overall consistency of the IC settings across the workflow.
This is something we expect the user to change frequently and potentially at different times during the workflow (e.g., the weather model has been built and run with a particular IC and the user wants to perform another run with a new IC - the model would not need to be rebuilt in this case.) The selected IC impacts both chrgres inputs and the model start time.
START_DATEparameter which should be used to determine the ICs. A meaningfulSTART_DATEis needed out of the box. An option here is to add a --startdate option to the create_newcase script (@jedwards4b). There may be other options as well.data_dir_input_gridthat points to the location of the time dependent raw input data$DIN_LOC_ROOT/global/ic/gfs.icdate. The fixed files on a platform will be in$DIN_LOC_ROOT/global/fix/fix_xx/.The text was updated successfully, but these errors were encountered: