Ensure workflow can point to user-created and staged UPP flat file rather than default

[jwolff-ncar]

Generic instructions on how to modify the UPP flat file for different fields/levels/etc are available in the UPP UG (https://upp.readthedocs.io/en/ufs-v1.0.0/InputsOutputs.html section 3.1.2.3). If a user does create a new file, the workflow needs to use it so we need flexibility to stage/point to a specific file. The instructions on how to stage/point to a specific file will need to be documented in the SRW app UG.

[jwolff-ncar]

From @JeffBeck-NOAA: I think a good option would be to create a user-definable variable in config_defaults.sh, maybe a path to the user's new flat file, telling the workflow to use that instead of sourcing from EMC_post.

[fossell]

Tagging @hertneky here too so she can be in the loop.

[jwolff-ncar]

Are there additional changes needed in the workflow at this point or has this been resolved? I thought more conversation had happened regarding this issue but they don't seem to be here. Is there another related issue that this should be pointing to?

[fossell]

There was some additional discussion on the EMC_post issue for SRW release: NOAA-EMC/UPP#157
I'm not sure of the status for this, my understanding was that SRW was going to make some changes in the workflow to point to a custom flat file that user creates. I'm not sure if that code mod was actually made though.

[JeffBeck-NOAA]

@jwolff-ncar @fossell This functionality still needs to be added to the workflow so that, if the user wishes, the workflow points to their own flat file, and not the one that is normally sourced from EMC_post.

[fossell]

@JeffBeck-NOAA - would you like help from UPP team or is this too specific to SRW development? @hertneky - Have you already experimented with doing this for SRW?

[JeffBeck-NOAA]

@fossell If @hertneky feels comfortable working on this, then any help would be greatly appreciated, as we're overwhelmed with development for the release at the moment.

[hertneky]

@JeffBeck-NOAA @fossell Being that it doesn't use CIME and that I have been using it, I feel more comfortable about tackling this than when it was done for the MRW. It might be helpful to have a quick meeting, as I imagine workflow folks might have an idea of how they might envision this customization working. e.g. a parameter that can be set in config.sh for passing to the workflow?

[JeffBeck-NOAA]

@hertneky That's exactly how I envisioned it, something like a "UPP_flat_file_from_user" variable that would have a default "false" value in config_defaults.sh, and, when defined and set to "true" in config.sh, would then activate an "if" statement (based on the value of UPP_flat_file_from_user) around the following line in exregional_run_post.sh:

cp_vrfy ${EMC_POST_DIR}/parm/postxconfig-NT-fv3sar.txt ./postxconfig-NT.txt

We'd also need a user-defined path variable that would then be used in that "if" statement to locate the new flat file and copy it.

If you'd like to meet, I'm happy to do that too! Thanks!

[hertneky]

@JeffBeck-NOAA Glad I'm thinking on the same page. This is definitely something I could do and test easily. There are obvious other questions;

1. Would the user need to use the same filename for the flat file or would they include the filename in the path variable?
2. For the MRW, the user is required to remake the flat file themselves after customizing the postcntrl.xml - would we want the same process here? Assuming so, just making sure.

I thought I had another question, but can't think of it now. I can likely tackle this next week and mock up some documentation.

[JeffBeck-NOAA]

@hertneky

1. My feeling would be that the second user-defined variable would be something like ${UPP_flat_file_path}, where the user can define the path and name of the their flat file however they like, and then if ${UPP_flat_file_from_user} is set to "true", then the "if" statement would activate the following:

cp_vrfy ${UPP_flat_file_path} ./postxconfig-NT.txt

2. Yeah, I would leave that up to the user to do prior to running the SRW App workflow. We would just assume they've already completed that step.

Thanks for your help with this!

[hertneky]

@JeffBeck-NOAA I've tested the customization addition and all tests work fine.

Test 1: run as usual using default config file
Test 2: run using modified config in a from a new path
Test 3: run using modified config, but purposely pointing to incorrect path - to check that it errors properly. I added a file path check in generate_FV3LAM_wflow.sh for the custom config file. I thought this a better place than in exregional_run_post since it won't waste resources trying to run post if the user defines the path incorrectly. Let me know if there is a better/preferred place for this.
If all sounds good - I will put in PR.

[JeffBeck-NOAA]

@hertneky Thanks for your work on this! The Google Doc instructions look great too. Please feel free to create a PR and we'll have the code review committee take a look.

Thanks!

[jwolff-ncar]

@hertneky Is this done? If so, let's go ahead and close this.

[hertneky]

@jwolff-ncar This is done and can be closed. I do not have the capability to close this one myself. Can you go ahead and do it? Thanks!