Skip to content

2022 Software Engineering meeting notes

Gregory Lemieux edited this page Dec 5, 2022 · 294 revisions

December 5, 2022

  • Upcoming meetings (5 min)
  • Weekly synthesis of the PR Status Board (5-10 min)
  • Latest issues (5-10 min)
  • Calibration progress update (5-10 min)
  • CLM meeting recap (5 min)
  • Discussion of Tristan's DA presentation
  • Walk-on topics

Upcoming meetings

  • FATES modeling meeting: Thursday December 8, Xinyuan Wei
  • E3SM LBNL Team Meeting: Wednesday December 7, 9:00 AM PST
  • CLM-FATES grazing meeting: Wednesday December 7, 1:00 PM PST

CLM meeting recap

Discussion of Tristan's DA presentation

November 28, 2022

  • Upcoming meetings (5 min)
  • Weekly synthesis of the PR Status Board (5-10 min)
  • Latest issues (5-10 min)
  • Calibration progress update (5-10 min)
  • Leaf Layer Optimization (5-10 min)
  • Dynamic LU/Pasture CLM meeting prep
  • Walk-on topics

Upcoming meetings

Calibration progress update (5-10 min)

Leaf Layer Optimization (5-10 min)

Dynamic-land use/Pasture CLM meeting prep

  • References
  • clm meeting prep
    • What are key priorities?
      • focus on dynamic LU or pasture?
      • what HLM development can happen that will benefit FATES and Matvey and Sam?
    • Designated a "lead"?
      • Design documentation

November 21, 2022

  • Upcoming meetings (5 min)
  • Weekly synthesis of the PR Status Board (5-10 min)
    • Nutrient update check-in
  • Latest issues (5-10 min)
  • Cori retirement update
  • Calibration progress update (5-10 min)
    • Albedo bias
  • Discussion of Tristan's DA talk
  • Walk-on topics

Upcoming meetings

  • FATES modeling meeting: Thursday December 8, Xinyuan Wei
    • Ryan suggested reaching out to people that have done nutrients work that we haven't included yet - example: LM4 group
    • Rosie to reach out to Nick Smith
  • CLM meeting, Pasture Land Use: Thursday December 1, 12:00 PM PST
  • Updated
  • New
    • 940 (patch fusion error)
      • Charlie: Speculation is that this is an outdated check necessary prior to categorical patches
      • Looking at git blame and tracing to the orignal PR this was used to avoid "numerical havoc".

Cori retirement update

  • Pushed back to March 2023. Next transitions training on Dec 1.

Calibration progress update

  • Rosie recapped the presentation that she made at the clm meeting last week
    • Different data products have different biases. Data assimilation workflow
    • What next? Update parameter defaults with the results? Which one (somewhat arbitrary)?
      • Jennifer: suggested using convergence between this and the analysis from Jessie.
        • Need to increase leaf layers from 30 to 36
      • Consensus is to do this.
    • Ryan: process resolution change? Chop up leaf-layer finer in the radiative transfer model and then upscale
      • If we can increase bin widths as we go vertically, how come we are running into max leaf layers crash?
    • Ryan: 3 canopies versus 2 cases?
      • BCI script generates a 3 layer system.
      • Marcos a long time ago, Charlie more recently to handle grass
      • Recall not important for SP mode since no canopy layers
  • Jessie plotting net carbon uptake by leaf layer using Atkins to look for areas of die-off see last week
    • Charlie: Visualizes the canopy trimming logic. This gives insight into how trimming would occur even when in sp mode

Discussion of Tristan's DA presentation - deferred

November 14, 2022

  • Upcoming meetings (5 min)
  • Weekly synthesis of the PR Status Board (5-10 min)
  • Latest issues (5-10 min)
  • Calibration progress update (5-10 min)
  • HLM use_bedrock (Jackie)
  • Pasture LU development meeting (Jackie, Rosie)
  • Walk-on topics

Upcoming meetings

  • FATES modeling meeting: Thursday December 8, Xinyuan Wei
  • CLM meeting, Pasture Land Use: Thursday December 1, 12:00 PM PST

Calibration progress update

  • Rosie: more study on clumping sensitivity, and has a nice ilamb workflow. Differences are significant in the tropics
  • Jessie: Working on respiration , studying the base rate at Ryan et al. , found that increasing base respiration, improved global biomass, but made negative progress on LAI and GPP. Atkin respiration is showing collapse in certain areas.

HLM use_bedrock (Jackie)

  • Use_bedrock issue

  • Katie test w/ FATES is success w/ recognize change in depth to bedrock

  • not closing old issue: Use_bedrock error

  • Plan on running some large tests, e3sm and ctsm to see if things are working

  • Plan on issueing a PR to switch using bedrock as the default in FATES: use_bedrock=.true. (was =.false.)

    • change will impact roots in active layer and ALT
  • Create a discussion on FATES side

  • This is important for considering with calibrations!

Pasture LU development meeting (Jackie, Rosie)

  • time for group discussion being set by poll
    • Discussion kicked off earlier this morning (8AM PST). Next discussion during CLM meeting, December 1
  • meantime check pasture discussion thread pasture dev

Walk-on topics

  • Recap of Tristan's talk (Ryan) - deferred

November 07, 2022

  • Upcoming meetings (5 min)
  • Weekly synthesis of the PR Status Board (5-10 min)
  • Latest issues (5-10 min)
  • Calibration progress update (5-10 min)
  • python regional subset is working! Jackie to test new PR (Jackie)
  • AD spin-up (Ryan)
  • HLM bedrock depth (Ryan, Jackie)
  • Crop anthropogenic land use labels (Rosie)
  • Respiration and productivity (Jessie)
  • Walk-on topics

Upcoming meetings

  • New: none
  • Ryan had offline discussions via email with Hanlin. Encouraged them to update the relevant discussions.

Calibration progress update

  • Rosie: ILAMB working on ensembles inside Jupyter
    • ILAMB is not really well setup for ensembles. Rosie is considering just grabbing output nc and replicating graphics instead of using native ILAMB graphics.
    • Change in albedo with bin-width updates
      • Thin layers and no clumping reduces the bias
        • What assumptions are being made for the LAI input data during its generation? How does this impact our considerations of clumping?
      • Reminder: There is no native fates albedo output. Post calculate via ratio of upwelling and downwelling radiation

Regional subsetting tool update

  • Jackie: Negin has updated tool PR. Jackie has yet to test. Should be integrated soon.
    • Aside: There was some concern on ctsm side about using dask. Charlie noted that xarray uses dask.
    • subsetting regions PR for testing

AD spin-up

  • Keith and Ryan have been revisiting this
    • Some bugs still to be fixed
      • We're not MPI broadcasting flags
      • Some of the namelist building scripts need to be updated
    • Discussion about total system carbon differences

HLM bedrock depth

  • Use_bedrock issue
  • create a discussion on FATES side?
  • FATES has roots in active layer
  • FATES has depth to soil and thaw depth from HLM
  • w/ shallow bedrock ALT should not be lower than bedrock
  • set to default off. Should it be on?
    • Erik recommended to Katie to try turning on with FATES run. Jackie to check in with her.
    • Consensus is that is should be on by default
      • This is important for considering with calibrations!
      • Rosie will test out turning this on as well

Crop anthropogenic land use labels

  • Discussion impetus: Grazing on pasture in CLM5 or FATES for Mongolia project (Matvey Debolskiy)
    • Ryan: Do we need disturbance logic to enable transitions into/out of crop/pasture?
      • Charlie: currently assuming outcome of harvest is secondary lands, but we could change this (we aren't reading in transitions matrix yet)
    • Jackie: Dave Lawrence and Bill Sacks have been discussing something associated with a crop update. Jackie will pass along the details of this. We should come together about this in the future.

Respiration and productivity

  • ILAMB showing respiration low biased, but forest geo dben productivity looks high
    • Both update growth respiration rates and new model (Atkins) from Charlie tested
      • Increasing respiration rates shows better results, but new model probably should be utilized
        • New model includes new parameters that would need to be included into calibration

Walk on

  • Add e3sm specific discussion line every other week

October 31, 2022

  • Upcoming meetings (5 min)
  • Discussion section progress
  • Weekly synthesis of the PR Status Board (5-10 min)
  • Latest issues (5-10 min)
  • Calibration progress update (5-10 min)
  • MCT - update on end date & region subset
  • Filtering discussion
  • Walk-on topics

Upcoming meetings

Discussion section progress

Calibration progress update

MCT - update on end date & region subset

  • MCT
    • MCT failure with CESM tags, but not yet in CTSM
    • CTSM MCT likely fail with next externals update
      • next externals update weeks to a month out per Erik
  • Region subset
    • CTSM_PR1735 python might get completed python version
      • Negin working on this again, but maybe Teagan King to take it up
    • CTSM_1773 two line
      • depends on CIME for modules see Bill comments in PR1884
      • nco and esmf modules not loaded with the stand alone tool
      • so two-line method not long-term solution

Filtering discussion

October 24, 2022

  • Upcoming meetings (5 min)
  • Weekly synthesis of the PR Status Board (5-10 min)
  • Latest issues (5-10 min)
  • Nutrient coupling thoughts - filters and allocations (5 min)
  • Possibility of having higher order history output in CLM and ELM, abandon multiplexing? Getting goahead and work scoping on the three models?
  • Walk-on topics

Attendance

  • Charlie, Jennifer, Jessie, Marcos, Greg, Jackie, Kjetl, Rosie, Ryan
  • Kjetl (CICERO) joining the meeting for the first time!

Upcoming meetings

Nutrients

Multiplexing discussion

  • This came out of an email discussion between Erik and Ryan (within the context of PR 880)
  • ELM amenable to this change?
    • Jen: There is work going on to change the IO for exascale, so this might be worth discussing within this context
  • Does the group think this is worth pursuing at this point?
    • Worth followup discussion with CLM and ELM folks

Walk-on topics

  • Calibration progress
    • Charlie noted that we had wanted to have weekly updates on the calibration progress (Added to agenda template)
    • Jessie to upload Adrianna's scripts (both original form for Cheyenne and modified for Compy)
    • Rosie: work is ongoing
    • Kjetl and Rosie: ISIMIP
      • Impact focus
      • Uses CMIP output, but bias corrected. Daily mean output. Quite a lot of reprocessing before using in land models
      • Some discussion of overlap in scope
    • Ryan to look through notes from discussion with Keith to see if CLM-side works
      • Works on ELM side, but not sure if CLM side is working
    • Ryan: tropical pfts calibration
      • Marcos: planning to go back to this in Nov due to AGU goal
        • Charlie: we should have a discussion about supporting this within the context NGEET RFA3 meeting
    • Communications: point people to the repo where this work is ongoing. Not ready yet for a progress update to the larger community.

October 17, 2022

  • Upcoming meetings (5 min)
  • Weekly synthesis of the PR Status Board (5-10 min)
  • Latest issues (5-10 min)
  • Lawrencium perl library re-installed (Greg)
  • MCT deprecation update (Jackie)
  • XML patch file workflow demo (Ryan)
  • Walk-on topics

Upcoming meetings

MCT deprecation update

XML patch file workflow demo

  • Rosie: should we get rid of the "clonal" in the reproduction param name?
    • Jackie noted that fates_recruit_seed_dbh_repro_threshold
  • Proposal: create archive to save the previous defaults and the xml patch file
    • Allow people to replicate parameter file from previous version
    • Rosie: how to do pull request workflow?
      • Work through the proposed workflow and then create PR
    • Charlie: filename format proposal discussion
      • datestamp prior any suffix to allow for ease of sorting
      • Marcos: include a fates tag to correlate as well
      • Consensus seems to be note to include descriptive suffixes. Just use notes section in the xml file.
    • History meta data in default?
      • Consensus is that we shouldn't update the history meta data in the default cdl file
  • Testing protocol?
    • Submit PR and conduct ilamb analysis in the PR. Discuss results due to the param file update therein.

October 10, 2022

  • Upcoming meetings (5 min)
  • Google X invite update
  • Weekly synthesis of the PR Status Board (5-10 min)
  • Latest issues (5-10 min)
  • NGEET Executive committee report back (Jackie, 5-10 min)
  • Walk-on topics

Upcoming meetings

  • FATES modeling meeting: Thursday October 13
    • Thursday October 27 TBD
      • Marcos will miss this
      • Rosie suggested we communicate that we are emphesizing global calibration
  • Related: NGEE-Arctic ModEx workshop and meeting: October 17-19
    • "Field to Model" workshop by Shawn Serbin and others utilizing containerized ELM using OLMT
    • Jennifer has been using OLMT for site parameterization, pre-industrial spin up
      • Rosie and Jennifer agree that we should bring it up with an option
  • Ryan and Greg met with Greg Bronevetsky from Google X last week
  • What dates and who to present?
  • Rosie suggested getting ML figure from the Oxford talk that happened recently
  • Main concern is time commitment
  • Dec 13 is during AGU. Pick Jan 31 date
  • Greg to reach out LBL folks for advice on preparation
  • Ryan to come up with proposal outline slide deck. 3-4 topics, no more.
  • Reach out to google contacts for more insight as well
  • Integrated: None
  • Updated:
    • E3SM#5106 (C-based harvests): Ryan started review
    • 891 (Grass PFT), 888 (C-based harvest), 788 (Crown damage) review updates?
      • 888: Charlie and Jen have been discussing with Shijie via email. Updates forthcoming from Shijie
      • 788: testing on going
  • New:
  • Updated: None
  • New
    • 915 (patchno refactor): inconsistency in code; needs refactor
      • Related: Rosie and Charlie agreed with Greg's proposal to review refactoring patch data structure to remove inclusion of zeroth patch. This would help avoid potential issues that PR 914 addresses.
        • Greg to draft design and send out for review
  • Aside: github actions testout on Greg's personal repo. This will be implemented to update the issue triage board in near future.

Walk-on topics

  • Jennifer: typo in temp freezing mortality tolerance. Found durings boreal forest runs. See email thread.
    • Quick fix: update parameter. All agree. Long-term fix: use Marius' code that references his publication.
    • Ryan: put prototype workflow on agenda for next week including these modifications
  • Rosie: Adding CSIRO employee to fates software meeting. Will be working on calibration cascade stuff.

NGEET executive committee report back (Jackie)

  • Deferred due to time

October 03, 2022

Agenda

  • Upcoming meetings (5 min)
  • Weekly synthesis of the PR Status Board (5-10 min)
  • Latest issues (5-10 min)
  • NGEET Annual All-Hands review
  • Project coordination and OSF.io demo
  • Google X invitation
  • Walk-on topics

Upcoming meetings

  • FATES modeling meeting: Thursday October 13
    • Evan Gora. Send reminder next week.
    • Rosie suggested we communicate that we are remphesizing global calibration

NGEET Annual All-Hands review

  • Executive committee report back
    1. Global runs, coupled simulations and calibration
    • Ryan is concerned about still needing reasonable MET data
    • Coupled implies global
    • Use of Regionaly Refined Mesh may be an important feature for FATES-E3SM
    • E3SM is hopeful that 10km grid will provide reasonable amazon rainfall
    • E3SM team has inquired about FATES performance timing. (Recalled that Dave L reported ~6x slowdown on earlier versions of fates compared to big-leaf clm). Revisiting coupled E3SM-FATES runs is of interest.
    1. Need more people to do regional runs
    • Set milestones for simulations
    • Basin-wide scale was essential, hydro, nutrients and disturbance linked
      • Linking damage with fire mortality
    1. More outreach and tutorial (Maria U?)
    • Goal: DEI: capacity building at institution outside the US
    • Consistent with Jessica and Stuart discussions during Cartehana meetings
      • Similar to the NGEE-Arctic model to field tutorial that Shawn is leading
        • This was brought up in exec committee as successful example
    • Funding discussion?
    • Discussed the current status of containerization, and how that might be useful for reaching out to new users
  • Other topics
    • Patch level soil columns

Project coordination and OSF.io demo

  • During house-keeping, we will mention during the FATES meeting that we have a group effort underway to perform calibrations and
  • Discussion of Jennifer's parameter file and where to host
    • Ryan would like to pre-process into xml patch format
      • helps distill the salient differences and param file name changes
      • Jessica succesfully used tool to convert Jennifer's global file
    • Jennifer
      • Two and a half files
        • General improvement to global distributions. Small tweaks to a few params using sensitivity analysis (SLA, Wood density, etc)
          • Jessica used to benchmark with ILAMB and confirmed improved
        • Alaska boreal single site using neural network
          • Also the global + boreal updates
      • Discussed Jennifer's parameterization. Some parameters such as SLA, wood density and others can be released into public versions as soon as ready. Other parameterizations, specifically Boreal parameterizations, are not ready for public release
    • Charlie: Regional specific cases versus global baseline
      • Workflow for integrating regional results into global baseline?
        • This is likely to happen more often
        • Do we want a pull request to the param file?
        • Global calibration workflow
          • Another descriptive layer to describe the way runs were done
          • Way to store all parameter files and configurations that we used for these
    • Hosting plan DBEN github repository
  • Greg demo'd OSF.io to Adrianna and Jessica

Google X invitation

  • Jennifer was contacted by Google X employee to present FATES modelling work at Alphabet-wide climate talk series
    • Jennifer recommended Ryan and Greg since mostly general software experience with ecosystem modelling
  • Respond back with questions about talk
    • Audience? Who else is in the series?
      • Setup time to talk on a call.
    • Sounds like an existing series. There are existing time slots.

September 26, 2022

Canceled due to members travelling for NGEE-Tropics Annual All-hands meeting

September 19, 2022

Agenda

Upcoming meetings

  • FATES modeling meeting: Thursday September 29
    • canceled due to NGEET all hands
    • Rosie suggested Tristan Quaife for potential future talk
  • NGEE-Tropics Annual All-hands meeting: Tuesday, Sep 27 - Thursday, Sep 29
  • FATES Software Meeting: Monday September 26
    • Consensus was to cancel this meeting as Ryan and Jackie will be in transit for NGEET all hands

PR status board (pre-meeting synthesis today)

  • Integrated

    • 887 (laimemory), fates-users-guide#16/#18 (cleanup and link to Discussions)
      • Issue #899 created from PR #887
  • Updated

    • 788 (crown damage): Ryan and Jessica have made progress. Regression testing soon
    • CTSM#1849/1827 (test updates): combining both PRs, testing complete
    • 904 (readme): removes out of date info and adds link to Discussions
    • fates-docs#30 (fates-hydro section): formatting fix in progress
    • E3SM#5106 (c-harvest): Cleanup complete, needs testing
      • 888 (c-harvest): needs deconflict and testing

Latest issues

Global calibration and parameter discussion

  • Prompt: How to conduct/update the combined science and software testing workflow for updating the numbers in the default parameter file?
  • What does the division of labor look like?
  • When do we change the default parameter file?
    • What is the basis for the current parameters? Inherited big leaf CLM parameters.
      • How do we handle clm/elm parameter divergence?
  • Workflow
  • There is a request to have Jennifer and Jessie present ILAMB comparision results
  • Move from "default" to set of specific testing sets?
    • Or maybe we should change to "base" (in conjunction with xml patch files)
  • Start a discussion for this in Github?

September 12, 2022

Agenda

Upcoming meetings

PR status board (pre-meeting synthesis today)

  • 887 (laimemory): PR split to determine non-b4b factor is underway
    • Why is it that satphen mode is not b4b?
      • Reviewing the UpdateCohortLAI routine suggests that we should move sai inside the sp mode check.
  • 888/E3SM#5106 (C-based harvest): Cleanup underway
  • 880 (nutrient dynamics): moved to WIP

Machine updates

  • Cori retirement date announced: Jan 17, 2023
  • Lawrencium: Perl XML::LibXML dependency removed for some reason. LBL IT asking that it be locally installed with cpan.
  • Part of the problem is due to yearly net uptake not being carried over during the restart.

Walk on topics

  • Ryan: Double slash issue causing still exists?
    • Ryan re-discovered that including a double slash in the caseroot definition for your script can cause a build failure.
    • Error message is not very informative. Ryan to discuss with CIME folks.
  • Jessica: Demographic Benchmarking (DBEN) quick update
    • Jessica to send out scheduling email when Charlie and Jennifer get back.
    • Some discussion about starting a NGEET project board for coordinating this effort.
      • Ryan suggested potentially using OSF.io
    • What sort of tooling do we have through ctsm, that we need for e3sm? Particularly with regard to setting up new sites that would be part of this project.

September 05, 2022

No meeting due to Federal holiday

August 29, 2022

Agenda

Upcoming meetings

  • FATES modeling meeting: Thursday September 01
    • Anthony confirmed to present
  • Next meeting (Sep 15): office hours
    • Ryan to send out annoucement and be in charge
    • Next meeting after that (Sep 29): canceled due to NGEET all hands
  • Next fates software meeting (Monday September 5) canceled due to labor day

PR status board (pre-meeting synthesis today)

  • E3SM#5094 (lobata configuration): integrated
  • E3SM#5092 (fates api 24): approved
  • #892 (height metrics): On deck for review and test
  • fates-docs#30: Greg to fix documentation formatting
  • E3SM#5106 (C-based harvest): Ask Shijie to restore machine config and test file changes to master
  • #851 (hydro stability) Greg revived old email thread to check status
  • #891 (grass pft): review ongoing (see agenda item)

PR 891 grass fire discussion

  • Should items raised become new issues? Are they within scope of the PR?
  • Ryan suggesting add graceful failure to the code
  • Wait for feedback from Zachary

Revisiting gitub "disucussions" tab

  • ctsm software group decided to give github discussions a try
    • archiving open-ended issues that are actually discussions
    • converting issues that are questions and tagging "answers"
  • Do we want to give this a try with fates repo as well?
  • Related: Charlie suggested directing people in Slack to this since Slack isn't archived
  • Add links to the CTSM/E3SM github Discussions and CESM forums

August 22, 2022

Agenda

Upcoming meetings

  • FATES modeling meeting: Thursday September 01
    • Ryan and Charlie to follow up with Anthony about presenting September 1.
    • Jackie suggested Thomas Bytnerowicz who attended the cesm tutorial recently
      • Charlie to reach out to him via email thread from a few months ago

PR status board (pre-meeting synthesis today)

  • #887 (laimemory): Satellite phenology global test not bfb with baseline. Greg and Ryan to assess.
  • #788 (crown damage): Ryan to return to this soon.
  • #891 (Grass CWD): Zachary created PR. Jackie reviewed. More reviewers?
  • #880 (nutrient dynamics): work ongoing
  • fates-docs#30: Junyan added fates-hydro section. Ryan tapped for review.

Cori retirement and Perlmutter followup

  • Greg sent out email about successful use of perlmutter
  • Noel Keen is point of contact for the machine
  • Login node mirrors cori, so transition was nearly seamless

Exact restart (ERS) issue and test coverage

  • Charlie found evidence that his cases are not restarting exactly
    • 2 potential issues:
      • non-exact restart when started during the same time of year, but difference is not terribly meaningful scientifically
      • very different restart when started at different time of year
        • Jennifer has seen something like this before due to deciduous timing being off during restart
  • Investigation
    • Greg found that other testmods are failing long-term (one year or greater) tests
    • Ryan and Greg investigated canopy trim timing as a possibility this morning.
    • Testing progressively simpler cases for restart on going
      • SatPhen passes 1 year ERS
  • Take away: adding more long term tests to the fates suite is probably warrented
    • We had been assuming that the one long term ers issue test was specific to a testmod
    • Greg: should we have a very long (100+ year) test that we run occasionally?
      • Charlie notes that would be good for testing effects of parameterization
      • What about ilamb automation? Adrianna has looked into this in the past. Hard part is automating output analysis (pass/fail)

August 15, 2022

Agenda

Upcoming meetings

  • FATES modeling meeting: Thursday August 18
    • Anthony contacted by Charlie. He will present September 1.
    • Adrianna and Charlie to present this week.

PR status board (No pre-meeting synthesis today)

  • #873 (grassfall): On deck to be integrated
  • #887 (laimemory): Testing started
  • E3SM PRs in queue to be integrated
  • Zacharcy has a PR on the CWD fix coming soon

Cori retirement and Perlmutter allocations

  • Per the most recent NERSC weekly email, Cori "will be retired at the end of this allocation year".
  • "AY2023 Allocations will be the topic of the next NUG Monthly Meeting, on August 18".
  • Greg to reach out to Noel to get insight into Perlmutter readiness with e3sm

Walk on topics

  • Marcos: Feedback on CLM-FATES NSF proposal rejection lesson learned
    • It might be wise for use to standardize the way that the we name the model relative to the features available
      • Make it obvious for non-fates modelers that a specific configuration is being used, but is not a special branch or fork of the model.
        • I.e. just say "FATES using the hydro configuration" instead of "FATES-hydro".
  • Jackie
  • Rosie
    • New hire working on NorESM starting up soon
    • ML driven calibration casacade for Schmidt foundation grant
      • A lot of barriers to institution participation
        • Charlie: for foundation work you need approvals up to deputy director level
  • Amsterdam conference
    • 4 people from LBL fates attending

August 08, 2022

Agenda

Upcoming meetings

  • FATES meeting: Thursday August 18, presenter TBD

PR status board (No pre-meeting synthesis today)

The LAI and memory PR has been cleared for take-off. Ryan needs to fix his refactors to the crown damage PR

PR 873 Grassfall discussion

Discussion went into topics of how non-fire disturbance would potentially affect grass heterogeneity. Considered having a pft-specific parameter for disturbance fraction, but group thought maybe it would become a slippery slope that would require more knowledge of disturbance type and grow in scope. Also discussion about refining the patch fusion algorithm to handle grasses better, perhaps by including a fuel score to the comparison metric, or by tweaking the existing agb score.

August 01, 2022

Agenda

Upcoming meetings

  • FATES meeting: Thursday August 04, presenter TBD
    • Cancelling due to people being out on vacation

PR status board (Pre-meeting synthesis)

  • #873 (grassfall) - Jackie provided results and comments detailing her concerns
    • Hold off until Jackie comes back from vacation next week to come to consensus on next steps
  • E3SM#5092 (API24) - testing complete; waiting on HLM approval
  • E3SM#5043 (Harvest Carbon) - develop fates-specific testmod and test this week
  • #788 (Crown damage)
    • Jessica is continuing to analyze the differences
    • Finalizing set of new variable dimensions to add
  • #877 (lai variables) - Marcos to review this week as this overlaps with his work
  • #880 (nutrient dynamics) - Oscillating dynamics addressed. Deconflict forthcoming.
    • Root proportionality timescale adjustment helps to dampen the "wave" dynamic. Currently set to one year long timescale.
  • #769 (cohort memory) - This is on-deck to be brought into To Do
    • Two variables combined are utilizing a lot of memory relative to all other cohort variables

Dimensions "creep" discussion

  • We currently have 23 multiplexed dimensions. Do we want to add anything or is there pruning that could happen?
  • What's the cost of writing out the coordinate?
    • This should only be set during initialization. This may only be an issue with a highly targeted, slimmed-down ensemble of single point cases.
    • Coordinates are written out on everyfile. We could explore only writing out to the first file.
      • Takes up 16kB per file
    • Do user's actually look at the dimension map outputs or do people look at the code to check the mapping?
      • Likely depends on people who develop versus new user's
      • Marcos and Jennifer do look at these, although only as a check; i.e. they don't need it for every file
  • NETCDF offers options for size of integer, if we wanted to save space.

Walk-on topics

  • Report back on E3SM-FATES LBL meeting
  • Greg will be out on vacation from Friday through Wednesday

July 25, 2022

Agenda

Upcoming meetings

  • FATES meeting: Thursday August 04, presenter TBD
    • Jackie is out of town next week
    • Determination of presenter postponed for now

PR status board (Pre-meeting synthesis)

  • #873 (grass fall): regression tests run; diffs look reasonable
    • Jackie tested against her case
      • 100 years, broad scale tests over South America
      • Subtle difference reduces the over area of grasses and changes the fire intensity.
    • This has implications for systems where fire is not going to happen
      • Non-tree items are part of seccesional pathways
      • As we move to model as we below ground processes
      • Recommend testing for implication on patch dynamics where there is no fire
        • We need mechanisms other that mortality to implement for grass disturbance
          • Aside: Charlie noted Zachary's latest work on grass mortality issue affect on fire via litter debris pool
          • This brings up discussion of woody versus non-woody handling in the model
            • ED2: implemented "lignified" fraction pft parameter which shunted fractions to specific debris
    • Jackie will update PR with results and concerns for further discussion
  • E3SM#5092: API24 submitted
    • This should be integrated prior to E3SM#5043 (Harvest Carbon API)
  • #788 (crown damage): Jessica to review Ryan's updates this week

Tree mortality workshop report back

  • Networks of forest plots
  • Reprsentatives from 8 different networks
  • Very open, 4 days of discussion
  • Two main items to report:
    1. Map of best estimates of mortality metrics from ground data is goal that workshop set
    • Any paper from this have first author as consortium of groups (groups broken down by workflow)
    • Training and capacity-building for global south is major focus
      • Jessica will be involved in the analysis/modelling training
    1. Forest ecologist wanted an "accessible" version of fates (GUI, laptop)
    • Site-specific setup (for non-supported sites)
    • Stuart Davies considering applying to https://impactchallenge.withgoogle.com/climate2022
    • Containerized version of ctsm discussion
      • Polly will be using it to run single point simulations
      • Post-processing for visualization is the bulk of cost (not model runs) on AWS
      • Automation subsetting of data has been discussed by Bryan
    • NEON project sounds similar to this
      • Processing of site-specific MET data is the hard part of this and would benefit from a grant
      • Includes a GUI and was geared to users who don't want to/have access to a terminal. Very simple.

July 18, 2022

Agenda

Upcoming meetings

  • FATES meeting: Thursday July 21, no presenter
    • General module development discussion (nuopc, recent feature addition, future plans)
      • Adrianna will update on MCT/NUOPC
      • Summarize the last half-year of updates
        • Utilize slides that were presented to program management team
        • Minimum criteria is a PR has been created
        • Charlie to develop first draft of list to notify authors
    • Solicit future presentations
  • FATES meeting: Thursday August 04, presenter TBD

PR status board (Pre-meeting synthesis)

  • CTSM#1515 (cleanup and minor API fixes): to be integrated today
    • Ryan fixed missing crown damage initialization
    • #854 (fates_fraction) integrated
  • #849 (taper function): numerical precision issue with restart
    • Lesson learned discussion on evaluating intermediate calculations
      • Marcos noted they saw this in ED with fusions that had very large order of magnitude differences. This shouldn't be an issue with high precision fates.
    • Question of concern: is this really a restart issue or a reproducability/stability issue?
      • Ryan noted a perturbation analysis would be interesting for this and help determine one way or the other.
    • Consensus is fix is optimized and correct
  • #873 (grass fall)
    • Jackie ran this and started to review the results. Does not look dramatically different.
    • Charlie noted that youngest patches will likely show the differences, but may be subtle
    • Non-disturbance mortality should be adjusted. Charlie confirmed with Ryan that this is the relevant code.
  • #788 (crown damage): Ryan's update complete; needs review and blessing from Jessica
  • #880 (nutrient dynamics) deconflict forthcoming
  • E3SM#5043 (Harvest Carbon API): deconflict and code cleanup started
    • Future ctsm api update is required to enable this in clm-fates
      • Ryan reported discussion about this. Noted that fates circumvents the clm product subroutine currently and this needs to be taken into account. Non-trivial API update.
  • #824 (IBM deallocation): start investigation later this week

Walk-on topics

  • Marcos report back on Colombia conference trip

July 11, 2022

Agenda

Upcoming meetings

  • FATES meeting: Thursday July 21, presenter TBD
    • Office hours and general module development (nuopc, recent feature addition, future plans)
    • Solicit future presentations
  • Land model working group conference, Oxford
    • Rosie is on organizing committee
    • Many people from FATES community are going

PR status board (Pre-meeting synthesis)

  • #854/CTSM#1515 (history update/satphen cleanup): waiting on cime tag update coming in with future ctsm tag
    • Fixed errh20 initialization error and added lobata to machine configurations
  • #875 (parallel disturbance): patch fusion error on master discovered during long-term testing
    • Greg running another long-term test on an older api to help determine timeframe
    • Implement and test Ryan's suggested fix
  • #873 (treefall): Testing this week. Follow up with Marcos and Charlie email discussion with Paul
    • Jackie to run changes with her case to test this week
    • Trees-only runs shouldn't change answers (this is specific to grasses)
  • #788 (crown damage): Ryan and Jessica working together to reorganize to simplify order of operations
  • #849 (taper function): Email discussion wrapped up. Testing this week.
  • #880 (nutrient dynamics): reviewers reqeuested
    • Ryan to do a conflict resolution and merge

MEGAN discussion update

  • Relevant issues:
  • MEGAN brings in vegetation properties for chemical emmissions
    • MEGAN has 6 broad pfts (VOCs are at species level)
  • Charlie: Does hlm read in MEGAN-index file for information?
    • Yes, there is a MEGAN mapping
  • Beyond fates-sp mode, there needs to be a science discussion of how to represent this in the model
  • New version of MEGAN is coming in the near future
    • Aside: MEGAN is calculating PAR, but should be recieving PAR
  • Short term: Erik's path forward
    • Long term: needs MEGAN-FATES champion

July 04, 2022

No meeting today due to July 4th federal holiday

June 27, 2022

Agenda

FATES crop discussion

  • How should fates handle 78 pft surface dataset?

  • WIP PR: https://github.com/NGEET/fates/pull/817

  • Related issues:

  • [Erik] Should hlm handle crops on different land unit?

    • i.e. using another instance of fates (not on nat veg land unit)
    • In clm, each crop is on its on land unit to avoid water competition
    • Harder to change a land unit since its a set area
  • [Ryan] additional water and mineral source for crop?

    • [Erik] so each crop has it own land unit, which has its own water and mineral source
    • There are irrigated off/on, fertalized/non-fetalized pft types
  • [Charlie] How does this compare to hillslope case? Fates should work with hillslope

    • [Erik] Hillslopes are on the column level, not land unit
  • [Marcos] exchange of seeds?

    • Likely not between crops, but would want on hillslope
  • [Ryan] does elm crop each have there own individual land unit?

    • [Jennifer] Single generic crop own its own unit, no dynamic landunit (i.e. change in area)
      • only 10 crop types
      • Gautam was working on dynamic landunits at one point, but has been de-prioritized
      • There are two perennial bio-energy crops
      • Proposal for Phase 3 is in progress, unsure if dynamic land unit is included
  • [Ryan] does clm have dynamic land units?

    • [Erik] need to check
    • Is a desired feature going forward
  • [Ryan] Do we want fates to handle crops?

    • [Charlie] yes, but simplified
    • [Erik] bgc-crop + fates nat veg is a ctsm goal
      • [Charlie] What is the clm long-term support for bgc-crop?
    • [Jennifer] Maybe we can anologize to bgc/fates nutrients?
      • HLM has existing strength with BGC, Fates with dynamic nutrients; each work together
  • [Ryan] Would like to see document with path forward and options

  • [Jackie] Note that clm side crop model is transitioning to Champagne-Urbana external (?)

  • [Ryan] Will an external fates-crop model be supported on HLM side? We need to have discussion with project management.

    • ELM will be focusing on its own crop model
    • CLM will be transitioning to APSiM
  • [Charlie] What are the immediate practical challenges?

    • [Ryan] We are almost at compatibility for fates-sp/crop runs
    • PFT collapse option?
      • Fates side or clm side?
        • clm collapses to 1 (maybe 2) pfts, passes to fates on nat veg land unit
        • [Erik] param file coordinated for both HLMs?
          • [Charlie] Yes, but calibration cascade may necessitate fates parameter file divergence between HLMs
  • Action: fates-crops pathway document

    • [Ryan] should include representative from all elm, clm, fates
    • [Erik] Volunteered to start work on CLM-fates document
  • [Marcos] Agro-forestry (silviculture) and orchards

    • [Jackie] Tree plantations don't exist in clm yet (someone did a palm-oil one, but not incorporated)
  • fates may be overkill for grass crops

    • [Charlie] counter argument needs to be able to successfully model grass dominance for fates
    • [Jackie] APSIM dictates when actions happen, doesn't 'grow' crops
      • Use fates for growth, APSIM for irrigation/harvest/etc
  • Action: create 78 pft fates parameter file to test

Upcoming meetings

  • FATES meeting: Thursday July 07, presenter TBD
    • Jackie is out of town. Charlie semi-out of town.
    • Adrianna wants to talk about surface roughness, but would like a small-focused meeting
    • Charlie will decide on office hours or cancel (Ryan will be around for office hours)

PR status board (Pre-meeting synthesis)

  • #825 (Carbon Harvest) integrated
    • E3SM#5043 created as WIP, needs some cleanup
  • #854/CTSM#1515 (small history update and cleanup) on deck after next CTSM tag
  • #875 (Parallel disturbance) testing this week
  • #788 (Crown damage) Ryan to deconflict and review
  • #873 (Grass treefall) on deck for testing
    • Marcos had a followup with Paul
      • Violates the idea that patches are not geographically represented
      • Marcos to start up email thread to facilitate discussion
      • Treefall rate should have a preferential location to occur
      • [Jackie] size of plant dictates size of disturbance (i.e. tiny trees)
  • #840 (taper function) Ryan and Chongang discussing via email
  • #880 (nutrient dynamics) massive pr; needs review

June 20, 2022

No meeting today in observation of Juneteenth federal holiday

June 13, 2022

Agenda

Upcoming meetings

  • FATES meeting: Thursday June 23 1100 PST, Marcos presenting
  • CESM Workshop this week: https://www.cesm.ucar.edu/events/workshops/2022/
    • No CTSM-FATES standup, CTSM software, or CLM meeting this week
  • E3SM Land/Energy NGD Monthly meeting canceled
    • Rosie asked where the notes meetings are held
  • No FATES software meeting next week (Monday June 20th): Juneteenth holiday
    • Next meeting: Monday June 27, 2022

PR status board (Pre-meeting synthesis)

  • #862/CTSM#1766 (patch count api update): Review in progress
    • Dry deposition discussion to be held between Erik and Ryan this week
      • Recommendation is to leave the dry deposition update for a future PR
    • Dynamic stoichiometry question [Ryan]: leaf-only or other organs in the future?
  • #863 (carbon allocation bug fix): Review complete, testing in progress
  • #856 (pft dimension mortality carbon fluxes): on deck

Walk-on topics

  • [Greg] E3SM software-centric meeting?
    • This came up in the context of Greg forgetting E3SM PR protocols and standards
    • Would be good to get heads up on upcoming infrastructure changes or impacts to elm
    • [Jennifer] agrees and recommends reaching out to Rob if there is a regular meeting
  • [Ryan] Next meeting main discussion will be on fates and crops
    • Ryan reported that Charlie would like to be able to collapse the crops into either C3 or C4
      • Some crop pfts are associated with irrigated or not irrigated
    • [Rosie] has anyone tried to have fates running with crops on a different landunit? Ryan is not sure.
    • [Greg] Erik has an action item to develop a matrix of the options for running fates/crops. This will be presented at the June 27 fates software meeting.
    • [Ryan] elm-fates crop coordination
      • [Jennifer] ELM crop model is based on CLM4.5 and being conducted by Beth Drewniak
        • Eva Lieungh is working on crop in elm as well using the FAN model
        • Phase 3 proposals being worked on currently, crops will be a part of this.
        • Jennifer will follow up on this to provide status update for June 27 meeting

June 06, 2022

Agenda

Upcoming meetings

  • FATES meeting: Thursday June 9 1100 PST, Xiangtao Xu presenting
    • Marcos to moderate as Charlie will be out due to vacation
    • Next meeting: Marcos offering to discuss defining pfts based on trait values
      • A number of people working on similar things, so maybe a good topic to get updates
  • DS4E meeting: Monday June 13 1200 PST
    • Alternate time for FATES software meeting?
      • Marcos to attend the DS4E meeting and report back at next fates software meeting. As such, regular fates software meeting time will be held
      • Adrianna, Jackie, and Charlie will be absent as well due to CESM workshop and vacation, respectively

PR status board (Pre-meeting synthesis)

  • Completed
    • #800 b4b PRs all integrated into single fates tag
      • #863 Carbon allocation bug to be integrated in near future
  • In progress
    • #862 parameter file under review
      • Ryan still trying to find non-b4b cause
    • #854/CTSM#1515 to be integrated after #862 and CTSM#1771
    • #788 crown damage module deconflicted but not yet merge back to PR branch
  • Up next (this week)
    • Carbon flux fix (#863) and mortality carbon flux pft update (#856)
      • Charlie notes that we should check that the length of the new history variables

Grass mortality disturbance discussion

  • Charlie confirmed this is a problem and has a PR fix in
    • This came up in the context of the California fire work
    • Aside: parallel disturbance #875 was not investigated in this context yet
  • The PR fix results in grasses can not cause canopy opening disturbances
    • fates_mortality_disturbance_fraction only applies to shrubs and trees now
    • Only disturbance with regards to grasses is due to fire (or future grass specific disturbances: e.g. grazing)
  • Related: some discussion about size of tree in creating a gap
    • [Marcos] degree of disturbance versus binary gap/no gap
    • currently fates only considers things in the canopy contributing to treefall mortality
    • Hank Shugart lab mentioned by Jennifer and Jackie with regard to potential future project
  • Decision: Move forward or further specific case tests?
    • Marcos to discuss

Walk-on topics

  • Tom Powell email discussion: respiration parameter is off by an order of magnitude when tuning for Panama
    • Marcos thinks we should revisit the fates_base_mr_20 to make sure units are correct (gC/gN/s)
    • Ryan seeing similar issue with root respiration
      • Charlie and Jackie noted that lack of root data is likely causing a large amount of uncertainty
        • Thus it is a scientific result worth highlighting
      • FRED data set has good respiration data and has been workign to get BCI data included
        • Perhaps it has root respiration data now?
      • Also check out https://daac.ornl.gov/SOILS/guides/SRDB_V5.html
        • Ben Bond-Lamberty is associated with this
  • CESM Workshop
    • Marcos to supply a summary slide
    • Charlie to supply reduced complexity slide
    • Jackie will run the compiled slide deck past everyone that contributed
  • [Ryan] FATES hydraulics V2 coming soon
    • Includes fine roots biomass as prognostic variable (see ESS PI meeting slides for details)

May 30, 2022

No meeting due to federal holiday.

May 23, 2022

Agenda

Upcoming meetings

  • No FATES meeting this week due to ESS PI meeting
    • Next FATES meeting: Thursday June 9 1100 PST, Xiangtao Xu presenting
  • Aside: Charlie will be on vacation June 9-21

PR status board (Pre-meeting synthesis)

  • canopy_layer_tlai fix (828): testing complete, but looking into why some 1x1_brazil tests are b4b and others are not (on the first output step)
  • crown damage (788): Greg to review this week
  • parameter file update (862): ctsm-side pr (1766) created

Gridcell-to-gridcell seed dispersal meeting recap

  • Yanlan presented an overview of the work to date
  • Greg to develop improvements on the grid2grid code (current code based on Sam's prognostic beetles code)
    • Ryan asked if the changes would be applicable to other grid2grid spread disturbances
      • Yes, but performance will likely be an issue for the current land model pe layout scheme for fast disturbances (e.g. fire)
  • Charlie, Greg, Jennifer and Yanlan to start meeting every other week to track progress and discuss

fates patch count controls (pr 1766) feedback request

  • Companion to https://github.com/NGEET/fates/pull/862
  • Allows fates to dictate number of natveg patches. Also incorporates updatest to handle cfts.
  • Erik, Greg and Ryan to have meeting to discuss tomorrow
  • Related: what other dynamic fates + fates-sp mode combinations might be of interest? Moving from fates global flags to per column flag.
    • Ryan suggested running dynamic fates for non-crop pfts, but fates-sp for crops
    • Charlie also suggested regional dynamic fates, with everywhere else being fates-sp

Issue triage discussion items

  • Charlie, Greg, and Ryan walked through the issue board last week to triage and prioritize
    • We will conduct this as a regular every-other month meeting
  • Items to discuss
    • 303 - postponed for future discussion with Rosie
    • 311 - closed and transferred to fates-users-guide issue
    • 312 - still valid future goal, left open
    • 342 - postponed for future discussion with Rosie
    • 861 - move forward with proposed fix

Walk-on topics

  • No woody pft check on treefall disturbance?
    • Charlie ran into this when assessing California fire results. This means grasses will have impact on treefall disturbance rates. Should this be the case?
    • Marcos noted that there had been similar discussions about this in the past. Somewhat of a philosophical debate.

May 16, 2022

Agenda

Upcoming meetings

  • Next FATES meeting: Thursday May 26 1100 PST
    • Cancelled due to ESS meeting and CTSM training
    • June 9 presenter: Xiangtao Xu
  • Gridcell-to-gridcell seed dispersal discussion: Friday May 20 1530 PST
  • CLM meeting: Thursday May 19 1200 PST
  • CA FATES meeting: Thursday May 19 1300 PST
  • Note: E3SM meeting last week was canceled.

PR status board (Pre-meeting synthesis)

  • Cheyenne down last week so no regression testing conducted
  • Crown damage: added a new parameter and noted in the PR
    • PR needs deconflict and review, but should be good for regression testing
  • Crown fire (857): lots of discussion and testing
    • Jackie: Functionality testing shows it is working, but still needs to be assessed for correctness
  • Parameter file (862) updates
  • Carbon harvest (825): post-LULCC impact?
    • We decided that the harvesting rates could be decided by the fates side just needs review.
    • Check that we need a ctsm side PR

FATES LULCC recap and new board

  • Post-meeting recap:
    • FATES LULCC V1 discussion and presentation by Shijie
    • One action item prioritized from meeting: CTSM issue 1046
    • Jennifer to publish her meeting notes and will send out google doc with initial notes so that others may contribute
  • New project board started to transfer from fates board to organization beta project
    • Board is public, but currently only Jennifer and Greg have write access. Contact either for access.
      • Aside: organization-level beta projects provide more control over board access
    • Greg would like to use this board to help organize the work into potential milestones and set issue priority
      • Jennifer suggested using the notes as starting point to update project board
  • Jackie to talk to the Lawrences to conduct a set of complimentary cases to Shijie's presentation

HLM-FATES meeting recap

  • Outcome: it would be worthwhile to talk about surface roughness
    • Until there is a person identified it is on the backburner
    • Adrianna states she will do some review into this issue and coordinate with Ryan on his work from ED2
    • Specific topics to be discussed in clm or fates meetings (i.e. not a regular recurring meeting)
  • Related: FATES topics for CESM workshop
    • Will sent out email this morning encouraging participation
    • Jackie suggesting that Shijie present his LU work
    • Charlie will be out for the workshop
    • Rosie: Calibration cascade presentation
    • Highlights presentation idea
      • Adrianna, Jackie, Jessica, Marcos
        • Charlie will query CA fates attendees
    • Format will be the standard setup. Agenda is very preliminary and flexible.
      • Adrianna: FATES single point tutorial?

Walk-on topics

  • Cori issue: PIO errors
    • Jessica talked to Noel who is looking into it

May 9, 2022

Agenda

FATES meeting topics and presenters

  • Shawn Serbin to present this week
  • Marcos to follow up with Xuanlie about doing June fates meeting

PR status board (Pre-meeting synthesis)

  • 800 Carbon allocation bug fix: Marcos has closed and transfered to separate PRs
  • 862 Param file update: we will discuss this in later agenda
  • Erik merged ctsm pr to run fates-sp without soilbgc

Upcoming meetings

  • HLM-FATES meeting: Thursday 1PM PST/2PM MST
    • Agenda preview
      • Plan for long term coupling HLM-FATES
        • Process duplication (radiative transfer/albedo, roughness length and displacement height, photosynthesis, hydraulics)
        • What is being passed from FATES to HLM and how is HLM using this?
        • Case study: Roughness length and displacement height
      • Additional features: BVOC and dry dep, CLM-FATES nutrients, LCLUC
      • Broader context:
        • Where processes are duplicated (radiative transfer/albedo, roughness length, displacement height, photosynthesis, stomatal conductance, hydraulics)
        • What is being passed from FATES to HLM and how is HLM using this? (e.g. albedo)
  • LULCC meeting: Friday 9AM PST/10AM MST
    • Agenda preview
      • Cover Ryan's email thread on harvesting rates
      • Discussion of Shijie's mass based harvesting: https://github.com/NGEET/fates/issues/869
      • Ryan suggesting assessment of where we are and what needs to happen
        • Asked Marcos if he could overview ed2 development history
        • Charlie: What needs to be done to facilitate full global landuse CMIPs-like scenario? Charlie believes Peter is key for this.
      • For future meeting:
        • Bill Sacks: linking dynamic land use

NGEET github projects

  • FATES performance project: https://github.com/orgs/NGEET/projects/2
    • Purpose: Organize related issues and PRs into common view across organizations and repositories
      • performance issue came out of ctsm training effort that spawned a discussion culiminating in two main tasks (so far): hifrq history update and fates-sp mode not in soilbgc
  • FATES issue board: https://github.com/orgs/NGEET/projects/1
    • Purpose: Provide location to triage all FATES-related issues
      • Segragation of issues that are mainly discussion topics, issues that are requests for features, etc
      • Impetus was to setup a board similar to PR status for high-priority issues that we could review in this meeting
        • Perhaps review only monthly?
  • Questions:
    • Charlie: should we convert the existing repo-level?
      • Greg:

Parameter file change tool demo

  • Ryan has developed a tool to facilitate updates to be included in PR 862.
    • Uses xml to categorize type of change (e.g. variable add, variable change, etc)
    • Includes update to
  • Jennifer: do we want to have a prefix for each parameter?
    • Ryan: some things aren't directly associated with a specific module

Walk-on topics

  • LBL IT looking into ESMF installation and support

May 2, 2022

Agenda

FATES meeting topics and presenters

  • List update status
    • Shawn Serbin will present on May 12

Project Board (Pre-meeting synthesis)

  • Quick updates:
    • Recent integrations: Logging clean up (729) integrated
    • 1643: mimics fates testing ongoing
    • Smaller PRs prioritize for this week: 828, 849, 854, 856
    • 788: Crown damage module moved to review
      • Jessica created ctsm branch
    • 857: Sam likely getting back to working on active crownfire this week
  • 851: Discussions with Chongang
    • Greg to reach out to Chongang via the email for the task list that I missed
  • 825: LULC discussion set for Friday May 13
    • Email discussion resulted in consensus
    • API branch needs PR (or incorporate into 862, see below)
  • 862: Parameter file updates
    • Some discussion on what will go in, but Ryan getting feedback
    • Ryan's questions:
      1. Do we want max cohorts per patch or use as hardcoded constant (https://github.com/NGEET/fates/issues/853)
        • clm needs a max some we need to have something to pass it
        • Rosie: Instead of max cohorts per patch do max cohorts per pft?
          • I.e. pft specific parameter. Charlie remembering that we were worried it was too complex a change.
          • Rosie: Just do a check instead of putting in parameter file
      2. Should we do a deep dive on organizing how we name things (i.e. ground lables)?
        • E.g. fates_fire_crowndepth -> fates_allom_fire_crowndepth
          • Also discussed what things should be switched
        • Make sure to include a transformation list
        • Jackie: Should we have a larger, long-term discussion about what to remove?
          • I.e. what was added for development, but now given tuning and hypothesis outcomes could be removed?
          • Greg: Should we start an issue to discuss this?
          • Ryan: Noted that this relates to determining what things we allow in a parameter file in the first place
        • Marcos: semi-deciduous parameter dependency on hypothesis
        • Jessie: Number of damage bin to default to be changed to 2
          • Charlie: what about adding a meta-data note stating what this is used for?
        • Add another attribute for meta-data
          • Naming: Mode or use case
          • Adrianna: What about adding an attribute to the variable to identify the grouping (in lieu of in the name)?
            • Consensus is that we should do both meta-data and namespacing
          • Jackie/Adrianna/Greg: have a stable/development attribute?
          • Perhap we could also include other information like hlm
          • Add an entry into the user's guide in coordination with run modes
      3. Should we add Shijie and Jessie's API changes?
        • 862 and eventual ctsm api needs to come in before 788 and 825
        • Adrianna: Offered to standardizing unit descriptions. Ryan to let her know when ready and then she will update.
        • Consensus is that its probably ok as long as the amount of work for each API isn't too large
  • Related:

NUOPC/ESMF issues

Walk-on Topics

  • CLM meeting: CTSM tutorial discussion
    • It may also have a discussion of surface roughness length

April 25, 2022

Agenda

FATES meeting topics and presenters

  • Hui Tang still on docket for this week
  • Charlie started list for future meeting planning

Project Board (Pre-meeting synthesis)

  • MIMICS coupling completed with ctsm5.1.dev091
  • #792 Logging is ready for testing
  • Ryan to start working on PR to wrangle the next paramfile update: https://github.com/NGEET/fates/issues/799
  • Harvest carbon status request from Jennifer
    • Jackie: we were talking about discussing this within the broader LULC discussion
      • Jackie will send out email this week to get the discussion going
      • Charlie doesn't want use regular fates meeting (so find different date than May 12 idea)

Performance and timing discussion

  • Discussion of fates timing in general and fates-sp mode specifically. Based on notes from Brian and calculation
  • Charlie: In fates-sp mode should we not run soil bgc?
    • Erik: I think we can turn this off?
    • Charlie: is dyn_subgrid supposed to be on as well?
      • David: yes, it can be on regardless of soil bgc
  • Charlie: hifrq_hist update
    • cost is remapping from cohort to ecosystem level every time step
    • Ryan: split into two groups site level and cohort level; add logic to select one or the other
    • Charlie: Break up associate statement and add if use_fates_sp logic
    • David: Why is this expensive for fates-sp if there is only one cohort per patch?
  • David: is this the bulk of fates cost? What about timing regular fates?
    • Adrianna: can we add timing to fates?
      • Ryan/Greg: yes we should be able to do this (perf_mod)
        • Erik: this should be the same from elm and clm
    • Dave will run f45 for non-sp mode comparison
  • Next steps
    • Jackie: Make issue and make sub issues
      • Ryan to file issue for fates wrap hifrq refactor
    • BGS shutoff
      • Ryan: edbgc wrappers need to be dealt with in clm_drv

Walk-on topics

  • Nitrogen coupling of ctsm-fates
    • David: identified as main deliverable for CESM3. What resources do we have for this?
    • Ryan: Will need pair programming; estimating half-year of implementation
      • Very difficult to estimate
      • Rosie: Mechanics versus tuning
        • David: Just asking about mechanics
      • Ryan: elm required a bunch of cleanup to facilitate this
        • EDBGC dyn file doesn't exist in elm

April 18, 2022

No notes recorded

April 11, 2022

Agenda

FATES meeting topics and presenters

Adrianna will present this week Jackie feedback:

  • Heard back from Sean Serbin
    • Wants to give a talk in the Fall about SW site
  • Gordon agreed to give talk on multi-layer canopy work
    • Will discuss with Ryan; Marcos will participate in discussion as well
    • Not available until mid-May at the earliest
  • Jennifer asking after focused land cover, land use model discussion.
    • Planning still in process via email.
      • Jackie will check with folks on NCAR side
      • From E3SM: Alan, Shijie, maybe Ben
    • Tentatively schedule for Thursday May 12
  • Rosie sent out email on Mar 28 for Hui to present 4 weeks (i.e Apr 28)

Action item: Create a calendar or spreadsheet to populate the list of presenters

Project Board (Pre-meeting synthesis)

  • #800 - ID of answer changing parts of the code are still in progress
    • Aside: Charlie asking if we've ever done the check to see what if any impact small noise in initial conditions has on output
    • Ryan and Marcos recommending splitting this PR into answer chaning and non-answer changing sections
      • Will need additional reviewers
  • #850 (CTSM #1643) - CTSM side is under active review.
  • #849 - Ryan to check in with author about scope of tests if any.
    • Ask Brad to review
    • Potentially, conduct a unit test "sanity check" against the previous range of values.
  • #851 - Feedback provided regarding transpiration error
    • Team would like to have further discussion with Chongang
  • #584 - Active crownfire starting back up. Greg, Jackie and Sam to discuss tomorrow
    • Feedback portion is not yet complete
  • #769 - Ryan looking towards working on this in near future
  • #788 - Still working on canopy too full issues. Greg to help debug.
    • Things taking too long, seems like its in the promotion/demotion scheme
      • Using the default scheme from the BCI parameter files.
        • Charlie recommends trying to use the opposite issue

Parameter file update

  • 12 new parameters not inluding another minimum of 4 parameters forthcoming from Jennifer
  • Ryan to create PR list
  • Discussion on when to add parameter
    • At minimum, new parameter shouldn't change answers without out some sort of discussion
    • Add section to develper's guide to for style guide on parameter file
    • Add user's guide section that discusses good first steps
      • Add to reference section additional meta-data for parameter file. Will require curation
  • How can we buffer a new user with what are relevant parameters

SIF discussion

  • Relevant to Gordron/Ryan/Marcos discussion above
  • We need to have a higher-level discussion about radiation prior to this. Tabled for later.

Walk-on topics

  • No topics

April 4, 2022

Agenda

FATES meeting topics and presenters

  • Team brainstormed a number of individuals. Ryan will email out potential list to include Charlie and Rosie to get there feedback.
  • Discussed using the dedicated fates Thursday time to access as focus conversation topics that might not be of interest to everyone

Project Board (Pre-meeting synthesis)

  • #847: Integrated and tagged
  • #800: Marcos to remove out-of-scope phenology-code from the PR
  • #825: Greg deconflicted and Shijie merged. Need to continue discussions; Ryan sent email to coordinate time.
  • #CTSM 1643: Added to board. This needs a FATES-side PR.
  • #788: Jessica is working through canopy too full errors.

CESM3 meeting preview discussion

  • Team asked Adrianna and Jackie for throughts on FATES within the context of CESM3
    • This conversation segued into the FATES updates question from Erik (see below)

CTSM software meeting FATES reporting

  • Related: Greg floated the idea of a broad quarterly FATES newsletter. Received
  • Related to CESM3 discussion
  • Related: E3SM methodology for tracking tasks to larger goals? Using Atlassian tools.
  • Adrianna and Jackie: User's stories, Agile framework, Epochs & Initiatives, clickup
  • Use this meeting to frame the report backs for the upcoming ctsm

Walk-on topics

  • CTSM training (May 25 and 26, 9am to noon)
    • Some discussion on how we can support this for FATES training
    • Overlaps with DOE ESS PI meeting

March 28, 2022

Agenda

FATES meeting topics and presenters

  • Adam is interested, but declined due to schedule conflicts
  • Current plan is to hold office hours this week
  • Potential future talks ideas:
    • PPE plans from Adrianna
    • Norway group: Rosie to follow-up.
    • ED2 improvements for comparison: Marcos to follow-up.
    • Ask Peter to give LULC status update: Jackie to follow-up.

Project Board (Pre-meeting synthesis)

  • #847: Greg to integrate (and update baselines)
  • #800: Ryan running some additional test cases by increasing LAI capacity allowable
  • #825: Ryan to double check units and Greg to deconflict
    • Discussion of the harvesting logic
      • Job of the host to tell FATES to harvest
      • ELM wants to read in smoothed
        • We asked Shijie to make it agnostic
      • Eventually FATES will allow HLM to dictate harvesting everyday
        • We need an update to the disturbance code to allow all types in a given day to
      • Next steps: Continue discussion on the github thread
        • Why does ELM want smoothing?
        • Should we give a time bounds for the harvesting interval?
          • CTSM crop model does something similar
  • #828: Merge in Ryan's branch commits to remove unnecessary LAI and continue discussions about LAI calculation method
    • Methodolgy of assessing what is in canopy 'above' for future PR
      • Currently looking at instantaneous LAI
      • Ryan suggesting a time-averaging idea
  • #824: Greg has access to Summit now. Will attempt to replicate issue this week.
  • #801: Status update on work to date
    • What are the user controllable parameters that the community would like to see?
      • Related: Jessica and Marcos have updates as well (see issue #799)
    • This PR will be superseded by a rebased version of the PR branch

Unit check for soil metric potential

March 21, 2022

Canceled: No pressing topics to discuss and some members out on vacation.

March 14, 2022

Agenda

FATES meeting topics and presenters

  • Charlie to reach out to Adam about FATES meeting on March 31

Project Board (Pre-meeting synthesis)

  • CTSM #1681 going through re-test
  • E3SM #4777 under review
  • #825 next steps being discussed via email
  • #800 discussion
    • This PR is tripping the LAI explosion issue (#844) at year 16
      • Master branch does not show the issue
    • Should we proceed as normal and integrate (PR is passing regression tests)?
    • Charlie: How do we explore the hypothesis that the PR is just unlucky?
      • Increase number of leaf layers until it passes the long case test
      • Ryan: Has a test that is running to cap the LAI
    • slamax provenance
      • Elise
    • Nominally tree should trim to reduce lai
    • Charlie: This is a long standing problem
      • CLM4 creataceous runs was getting huge lai as well
      • Allowing leaves thiness to approac infinty is problematic and allows much higher LAI
    • Charlie: Sergei and Elena use a memory term similar to what Ryan is proposing
    • Ryan to run sensitivty analysis to assess how this is performing
      • Code update is based on Greg's PR #828. We will likely expand the scope of that PR to include this.
  • #788: Jessie is having issues getting Lawrencium and the nuopc update
  • #768: Jennifer and Ryan to meet after this meeting to discuss next steps

Walk-on topics

  • #843: Adrianna to make change and test

March 7, 2022

Agenda

FATES meeting topics and presenters

  • Adrianna: volunteered to present allometry work for next meeting
  • Marcos: maybe Adam could present reproduction work in two weeks. Charlie to reach out.

Project Board (Pre-meeting synthesis)

  • #831: coming in next with a minor api change ctsm PR
    • Last time we update the parameter file was API 16.1
    • Jessica asked how to incorporate parameter file update for her #788
      • Ryan pointed to issue #799 that will get its own PR soon
  • #824: Summit request denied. Emailed ORNL for explaination.
  • E3SM #4777 needs to be brought upto API 22 now. Ryan to work on it soon.
  • #800: Review and testing this week. Ryan to handle.
  • #825: Email sent to Shijie to confirm the plan from last comment.
    • Will need porting of e3sm changes to ctsm side change to provide testing.
  • #788: Jessica is implementing a reviewer request which is resulting in the code breaking due to a weird bug.
    • Failure at ~70 years. Restarting from that point allows run to continue for another decade or so before another repeat failure.
      • Maybe memory leak or un-initialized variables?
  • #801: Drought deciduous needs to be brought up to date
  • #768: Jennifer has a few other parameters to add to issue #799
    • Ryan and Jennifer to connect offline to scope out the necessary refactor

Diffusion radiation angle granularity

  • Issue #841 to be included in future PR. Discussion delayed until later date.

Re-visit respiration issues

  • Ryan suggested making this a topic for a future FATES meeting.
    • Review Xinyuan's view on the topic

Readthedocs.org membership payment

  • Should we reinstate paying for the readthedocs membership to remove community ads?
    • Consensus is yes, pay to remove the ads.

Walk-on topics

  • Adrianna: Scientific unit tests
    • Would they help illuminate specific issues, e.g. radiation balance issues?
    • How do we know to set for the module level inputs? E.g. Dummy cohorts/pacthes?
      • Charlie: MAAT (Anthony) test bed (R scripts) useful here?
    • Ryan has some stuff built up, e.g. for allometry.
    • Greg: For infrastructure code might be worth it to keep an eye on interactive fortran project: https://lfortran.org/
  • Ryan: Should we spend time moving to a different radiation model instead of Norman?
    • Would be a significant effort.
    • Jackie: Is there benefit from adopting some of two-stream (clm) model concepts?

February 28, 2022

Agenda

Project Board (Pre-meeting synthesis)

  • #831 recap
    • Rosie doesn't believe that the albedo bug she is working on is related to the parameters
    • Jackie noted that there are differences discussed in the paper between the supplement (which is used by clm) and the averages presented in some of the main tables. It might be worth reviewing how this affects the ILAMB results.
      • Adrianna offered to handle this
    • Jackie to ask Shawn Serbin to provide his insight into the source paper based on his work
  • #838 update
    • This generated a discussion on how to mitigate technical debt
  • #788 status update: Jessica noted that she's ready to start testing

Walk-on Topics

  • #837 issue status update: Albedo bias (Rosie)
    • Still seeing high bias with the test case (1x1_brazil)
    • Comparison to CLM-bigleaf isn't entirely informative given the difference between the way CLM and fates handles this
    • If stems are removed from fates there is still a high albedo bias
    • Radiation balance check might be relevant to the testing
    • Not related to multiple canopy layers as Rosie is testing in SP mode (single canopy layer per patch)
    • Aside: To enable Issues in github for your personal fork of fates, click the checkbox under "General" section in "Settings" tab.
  • FATES_FRACTION rename (Adrianna)
    • Discussion about rolling this into an upcoming PR
  • Charlie has a PR bug fix forthcoming
  • History variable indexing and IO discussion
    • How much time is taken up by the output calculation and IO
    • How much time is the hi-freq output taking up in calculation and IO operations? What about instantiating a flag for setting different levels of output
      • Charlie to look to see if we have an issue already and bump it up the list. Otherwise he will
    • Aside: Is there a better way to setup and allocate fates variables? Discussion about Adrianna's hashmap idea.
  • Charlie proposed discussing the Thursday fates meeting topic setting/speaker recruiting in this meeting
    • We gave it a try for this coming Thursday
      • One topic will be to do a walk through of the User's Guide to advertise it and solict feedback in the form of github issues
  • TSS meeting report back

February 21, 2022

No meeting due to President's Day holiday

February 14, 2022

Agenda

Project Board (Pre-meeting synthesis)

  • #766 Hydro picard ready and waiting on next ctsm tag
  • #833 NoComp + disturbance testing passed as expected
  • #831 optical properties double check in progress
    • Adrianna presented ILAMB comparison between FATES spmode master param file and the new optical parameters
      • GPP and Upwelling is worse
      • Adrianna will posted the results link and context for further discussion
        • This spurred a general discussion on how we want to benchmark with ILAMB going forward
  • #824 Summit account requested
  • #836 frost mortality scheme drafted
    • Check in with Marius

Updates to FATES documentation

  • Namelist options clarification
    • The links to the different section of the wiki and tech-notes are good. Just need to expand these to modes that don't have coverage and flesh out some of the ones that currently do as well.
  • Autobuilding of PRs through Read The Docs server enabled on fates-docs and fates-users-guide
    • Integrated with github. Will block merge button if failing. Notifies defined email list of failure.
  • FATES User's guide
    • Good initial feedback and suggestions. Greg will transfer this to the NGEET repo once the wiki pages have been converted.
    • Started up a new page titled "Notable Differences" for comparison against big leaf (and other) models.
      • Greg asked for and encouraged contribution to this page. The focus should be on broad differences, with links to specific documentation for details (i.e. tech-note, clm user's guide, etc.)
    • Greg to update user's guide configuration to utilize markdown to speed up wiki conversion process

Walk-on topics

  • Discussion of ILAMB specific flux variables for grid cell averaging
    • Can we conducting the necessary averaging calculation in ILAMB (i.e. post-processing) rather than in the model code?
      • Per Adrianna, yes this is possible.
        • ILAMB can conduct a confrontation using a ‘derived’ variable
        • There are still some calculations that FATES does not currently conduct that might not be derivable.
      • Charlie noted that this would help avoid divergence between HLMs.
        • Jackie noted, in support, that ILAMB isn't HLM specific either
      • Aside: Is it possible to output time-invariant data in just the first timestep? Charlie thinks CLM can do this.
        • Adrianna believes so. Will ask Erik. Uncertain if this is pulled straight from dataset or passed through model.
        • Doing this for fates would help cut down on storage for certain things.
      • Short discussion of using ILAMB as development tool for scientific comparsion between tags
  • Drought decidous flickering issue question wrt #801
    • Charlie was reminded of the issue working on #833. Does Marcos' #801 fix this?
      • Marcos: Unfortunately, no.

February 7, 2022

Agenda

Project Board (Pre-meeting synthesis)

  • #766/CTSM#1592 - waiting on next ctsm devtag
    • Ryan will incorporate this into the API 21 FATES-ELM update
  • #831 - do we want to conduct additional science tests for the update optical parameters?
    • Conduct an assessment using ILAMB
      • This spurred discussion about incorporating an ILAMB test for each answer changing PR. The team agreed that this would be a good practice to adopt.
      • Adrianna to create an issue to track this and take point on this
  • #800 - Ryan brought up idea to adapt this PR to coordinate the simplified model using parteh
    • We agreed to push forward with this PR as is and conduct the leaf organ priority
  • #824 - IBM compiler issues
    • Greg to attempt to replicate in community, local version of IBM compiler. Will attempt to get summit account in parallel.
    • Marcos suggests adding test to check if the pointer is actually allocated/associated prior to the deallocation
  • #825 - Charlie will review
  • #833 - Charlie just submitted this to get nocomp and disturbance working together
    • Should be ready for review and testing
  • #788 - Still in progress
  • #768 - Jennifer will merge this up to match Ryan's nutrient branch work
    • Anthony has a paper that is coming out soon that we'd like to use for this hypothesis

FATES wiki updates

  • Tabled for next week

FATES User's guide

  • Greg will attempt to consolidate the wiki into read-the-docs as a starting point
  • Rosie noted that CICE consortium README page is nice and might be good structure to adapt

Subset single point function

  • Team agreed that zeroing out the lake, urban, glacier and wetland percentages is correct. Ryan noted that tower sites usually don't consider these things as part of their inventory.

Single pft SP mode issue

  • Push on with PR to address crops or split out the code relevant to the order of operations fix?
    • Greg will test out the latter option
    • Or we could remove the crop related param file changes and just repurpose the PR as the fix to #728

FATES AD Spinup report

  • Ryan noted that investigation on why accelerated decomp is not working with clm-fates is ongoing but slow due to other commitments
    • We're not broadcasting the spinup flag when fates is on
  • Charlie volunteered to take a look at this as well

January 24, 2022

Cancelled due to NCAR LMWG meeting

January 24, 2022

Agenda

  • Weekly synthesis of the PR Status Board: https://github.com/NGEET/fates/projects/7
  • E3SM FATES API Updates
  • Planning CLM-FATES surface roughness science discussion
  • Discuss ways to facilitate new user simulations and generate more scientifically supported configurations
  • Walk-on topics

Project Board (Pre-meeting synthesis)

  • SP mode is now in ELM. (are the ERS fixes in?)
  • Ryan will submit new PR now to E3SM for FATES API 21 (merge in Greg’s Changes)
  • Picard Hydraulics PR: (needs minor changes: SRL and loop cosmetic fixes in the new Solver) - Greg and I are going to team up on this. No more review necessary, just implementing review comments
  • Marcos’ allocation algorithm - this is a bug fix PR, so we are prioritizing this now, along with 829 and 828 (Josh’s bug and minor update to calculating TLAI)
    • Marcos' new capability will eventually need a regression test.
    • Charlie: Maybe we should add a conservation check on individuals to help catch errors like Josh's mortality crisis?

E3SM FATES API Updates

  • Ryan and Greg are working with Teklu to add an E3SM-FATES-TOPO test on the 4x5
    • Jennifer: Doing this will also exercise precipitation downscaling

CLM-FATES surface roughness

  • Meier is using Monin-Obukhov
  • Ryan: roughness calculation is an 'emergent' assemblage in fates at the canopy-level (patch) and not tied into pft
    • Charlie: counter-point that roughness can be based on pft shape (shape of indiv crown)
    • Ryan: we should do a review of the literature to see if there is a way to do better to use the structural information
  • Jackie was suggesting we have a discussion in a fates science call
    • The consensus seems to be this would need a dedicated person to explore this issue

Facilitating and engaging new users

  • FAQ documentation
    • Rosie: Perhaps we could create a survey for CLM-LMWG to help populate/refine a FATES FAQ
    • Greg: In the FAQ it might be helpful to include a "Noteworthy Differences" section to compare the general model type (i.e. versus gap models) and the specific implementation (e.g. versus big-leaf). See this example from Julia language docs.
    • Marcos: We should highlight which hypothesis we can analyze with fates that are difficult in others (e.g. selective logging)
  • Workflows
    • Jackie: Getting things into ILAMB framework will help
    • Rosie/Marcos: May have more opportunity to conduct benchmarking using additional workflows
    • Adrianna: BCI usermods and fates for NEON
  • Training
    • Jackie: TSS FATES specific tutorial would be helpful, but will need dedicated support

January 10, 2022

Agenda

E3SM FATES API Updates

  • Junyan is going to be doing simulations in the Chesapeake bay region for fates hydro and will want to use sp mode as well
  • Updating to API 20:
    • Ryan adopted a method for how elm calculates snow fraction for the bc_in%fcansno
    • Charlie: ctsm folks wrote a paper when they updated the canopy snow cover.
      • Ryan to write email asking about this.

API changes inbound with updates to hydraulics

  • Ryan updated the group on hydraulics update. CTSM side update has been approved by Greg

PR status board

Walk on topics

  • FATES ILAMB support discussion
    • We discussed having an audit of what we currently pass to the hlm (prognostic versus diagnostic)
      • What additional values do we need/want to send to be included for ILAMB?
    • Adrianna and Ryan to team up after LMWG meeting (Jan 31 to Feb 4) to work on this