Update benchmark table output to better show which diagnostics have Ref = Dev #200
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gcpy/benchmark.py - Now use f-strings to simplify print statements - Use numpy.array_equal to test if the Ref and Dev numpy arrays (after masking, if necessary) are equal. - Print a column for zero-diff results Signed-off-by: Bob Yantosca <[email protected]>
gcpy/benchmark.py - Update try/catch statement for file open - Use xr.Dataset.equals to test if the Ref and Dev datasets are identical to each other or not. (NOTE: This will likely fail when testing GCC vs GCHP, since the coordinates and data variables are different). This was the best way to proceed because in routine print_totals, the test for zero-diff is done per variable. - Now print out identicality or difference of Dev vs Ref in the table header gcpy/util.py - Restored the display name to a width of 19, as this is also used by the emissions tables. Signed-off-by: Bob Yantosca <[email protected]>
gcpy/benchmark.py
- In create_benchmark_emission_tables:
- Make sure columns and headers line up
- Now compute if each diagnostic for an emissions species or
inventory is identical, and display that information in the header
- Improve error trapping when opening the file for output
- In create_benchmark_mass_tables:
- Make sure columns and headers line up
gcpy/util.py
- In routine print_totals, make sure the header line with Ref & Dev
matches up with the output of create_benchmark_emissions_tables
and create_benchmark_mass_tables
- Removed is_zero_diff function, we will add another function like it
later.
Signed-off-by: Bob Yantosca <[email protected]>
yantosca
added
category: Feature Request
|
In commits 7e05db3,d1aa8bf28d70d1e94434797578ae23dfa00d1e5a, andd3d5568b17d4cd4f975f93e27144dbcaa6e6204b, we have added the following output:
Examples are shown here: Emisssions table output (the Inventory table output is similar): ###########################################################################################
### Emissions totals for species ACET [Tg] ###
### Ref = GCC_ref; Dev = GCC_dev ###
### ###
### Dev differs from Ref for species ACET ###
###########################################################################################
Ref Dev Dev - Ref % diff no-diff
ACET Anthro : 0.000000 0.000000 0.000000 nan True
ACET BioBurn : 0.342551 0.342551 0.000000 0.000 True
ACET Biogenic : 4.967504 4.364605 -0.602899 -12.137 False
ACET Ocean : 3.805566 3.805566 0.000000 0.000 True
------------------------------------------------------------------------------------------
ACET Total : 9.115759 8.512860 -0.602898 -6.614 False
... etc ...
###########################################################################################
### Emissions totals for species BCPI [Tg] ###
### Ref = GCC_ref; Dev = GCC_dev ###
### ###
### Dev is identical to Ref for species BCPI ###
###########################################################################################
Ref Dev Dev - Ref % diff no-diff
BCPI Aircraft : 0.000486 0.000486 0.000000 0.000 True
BCPI Anthro : 0.093466 0.093466 0.000000 0.000 True
BCPI BioBurn : 0.066152 0.066152 0.000000 0.000 True
BCPI Ship : 0.001399 0.001399 0.000000 0.000 True
------------------------------------------------------------------------------------------
BCPI Total : 0.161503 0.161503 0.000000 0.000 TrueMass table output: ###########################################################################################
### Global mass (Gg) at end of simulation (Trop + Strat) ###
### Ref = GCC_ref; Dev = GCC_dev ###
### ###
### Dev differs from Ref ###
###########################################################################################
Ref Dev Dev - Ref % diff no-diff
A3O2 : 0.076001 0.099527 0.023526 30.956 False
ACET : 6386.370605 6221.706055 -164.664551 -2.578 False
ACTA : 387.734192 372.065002 -15.669189 -4.041 False
AERI : 5.033212 5.486062 0.452850 8.997 False
ALD2 : 311.897247 299.977386 -11.919861 -3.822 False
ALK4 : 553.078247 505.029388 -48.048859 -8.688 False
AONITA : nan 39.504707 nan nan False
AROMP4 : nan 0.149606 nan nan False
AROMP5 : nan 0.090108 nan nan False
AROMRO2 : nan 0.113271 nan nan False
... etc ...
H2 : 178186.625000 178186.625000 0.000000 0.000 True |
|
@yantosca Is it possible to include a list of species that differ at the top of the global mass tables? That way we wouldn't have to comb through the list for all of the True values. For example: |
Hi @msulprizio. I can try to implement that. The only thing is that the differences are computed when the totals are printed so we'd need to do a separate computation for the top of the screen. I'll see what I can do. Also I've been tinkering with flagging the differences with an asterisk instead of true/false: Ref Dev Dev - Ref % diff diffs
ACET Anthro : 0.000000 0.000000 0.000000 nan
ACET BioBurn : 0.342551 0.342550 -0.000001 -0.000 *
ACET Biogenic : 4.364605 4.862582 0.497977 11.409 *
ACET Ocean : 3.805566 4.245402 0.439836 11.558 *
------------------------------------------------------------------------------------------
ACET Total : 8.512860 9.450675 0.937815 11.016 * Let me know which format you like better. |
|
@msulprizio: Or maybe what. I can do is to keep a running list of species differences and then append that to the header. |
Thanks @yantosca. I do like the asterisk better. |
|
@msulprizio: I was able to get this output: Species that differ between GCC_ref and GCC_dev
ACET ALD2 EOH HNO3 ISOP LIMO MTPA MTPO NO NO2 PRPE SO2 SOAP SOAS
###########################################################################################
### Emissions totals for species ACET [Tg] ###
### Ref = GCC_ref; Dev = GCC_dev ###
###########################################################################################
Ref Dev Dev - Ref % diff diffs
ACET Anthro : 0.000000 0.000000 0.000000 nan
ACET BioBurn : 0.342551 0.342551 0.000000 0.000
ACET Biogenic : 4.967504 4.364605 -0.602899 -12.137 *
ACET Ocean : 3.805566 3.805566 0.000000 0.000
------------------------------------------------------------------------------------------
ACET Total : 9.115759 8.512860 -0.602898 -6.614 * |
|
Also we now get this output for the mass tables: #########################################################################################
### Global mass (Gg) at end of simulation (Trop + Strat) ###
### Ref = GCC_ref; Dev = GCC_dev ###
#########################################################################################
Species that differ between GCC_ref and GCC_dev
A3O2 ACET ACTA AERI ALD2 ALK4 ASOA1 ASOA2 ASOA3 ASOAN ASOG1 ASOG2 ASOG3 ATO2
ATOOH B3O2 BCPI BCPO BENZ BRO2 Br Br2 BrCl BrNO2 BrNO3 BrO BrSALA BrSALC C2H6
C3H8 C4HVP1 C4HVP2 CCl4 CFC11 CFC113 CFC114 CFC115 CFC12 CH2Br2 CH2Cl2 CH2I2
CH2IBr CH2ICl CH2O CH2OO CH3Br CH3CCl3 CH3CHOO CH3Cl CH3I CH4 CHBr3 CHCl3 CLOCK
CO CO2 Cl Cl2 Cl2O2 ClNO2 ClNO3 ClO ClOO DMS DST1 DST2 DST3 DST4 EOH ETHLN ETNO3
ETO2 ETP GLYC GLYX H H1211 H1301 H2402 H2O H2O2 HAC HBr HC5A HCFC123 HCFC141b
HCFC142b HCFC22 HCOOH HCl HI HMHP HMML HNO2 HNO3 HNO4 HO2 HOBr HOCl HOI HONIT
HPALD1 HPALD1OO HPALD2 HPALD2OO HPALD3 HPALD4 HPETHNL I I2 I2O2 I2O3 I2O4 IBr
ICHE ICHOO ICN ICNOO ICPDH ICl IDC IDCHP IDHDP IDHNBOO IDHNDOO1 IDHNDOO2 IDHPE
IDN IDNOO IEPOXA IEPOXAOO IEPOXB IEPOXBOO IEPOXD IHN1 IHN2 IHN3 IHN4 IHOO1 IHOO4
IHPNBOO IHPNDOO IHPOO1 IHPOO2 IHPOO3 INA INDIOL INO INO2B INO2D INPB INPD IO
IONITA IONO IONO2 IPRNO3 ISALA ISALC ISOP ISOPNOO1 ISOPNOO2 ITCN ITHN KO2 LBRO2H
LBRO2N LCH4 LCO LIMO LIMO2 LISOPNO3 LISOPOH LOx LTRO2H LTRO2N LVOC LVOCOA LXRO2H
LXRO2N MACR MACR1OO MACR1OOH MACRNO2 MAP MCO3 MCRDH MCRENOL MCRHN MCRHNB MCRHP
MCROHOO MEK MENO3 MGLY MO2 MOH MONITA MONITS MONITU MP MPAN MPN MSA MTPA MTPO
MVK MVKDH MVKHC MVKHCB MVKHP MVKN MVKOHOO MVKPC N N2O N2O5 NH3 NH4 NIT NITs NO
NO2 NO3 NPRNO3 O O1D O3 OCPI OCPO OCS OClO OH OIO OLND OLNN OTHRO2 PAN PCO PH2O2
PIO2 PIP PO2 POx PP PPN PRN1 PROPNN PRPE PRPN PSO4 PYAC R4N1 R4N2 R4O2 R4P RA3P
RB3P RCHO RCO3 RIPA RIPB RIPC RIPD ROH RP SALA SALAAL SALACL SALC SALCAL SALCCL
SO2 SO4 SO4s SOAGX SOAIE SOAP SOAS TOLU TRO2 TSOA0 TSOA1 TSOA2 TSOA3 TSOG0 TSOG1
TSOG2 TSOG3 XRO2 XYLE pFe
### Ref = GCC_ref; Dev = GCC_dev
Ref Dev Dev - Ref % diff diffs
A3O2 : 0.076001 0.099527 0.023526 30.956 *
ACET : 6386.371096 6221.706425 -164.664671 -2.578 *
ACTA : 387.734197 372.064986 -15.669211 -4.041 *
AERI : 5.033211 5.486063 0.452852 8.997 *
etc |
gcpy/benchmark.py
- In routines create_benchmark_mass_tables and
create_benchmark_emission_tables
- Add placeholder text for the list of species diffs
- Define "diff_list" list to hold the list of species names
for species w/ nonzero diffs
- Remove references to np.array_equal and xr.dataset.equal
- Header lines are now 89 chars wide
- Now pass diff_list to util.print_totals
- Trim trailing whitespace
- Comment out code in create_benchmark_summary_table for now,
we will add this functionality in soon.
gcpy/util.py
- Now import textwrap.wrap function
- In function "print_totals"
- Now accepts and returns "diff_list"
- Add more error checks
- Compute total_ref and total_dev in 64-bit floating point
- Added function "unique_values"
- Added function "insert_text_into_file"
Signed-off-by: Bob Yantosca <[email protected]>
|
We now print out a summary table as well: ################################################################################
### Benchmark summary table ###
### Ref = GCC_ref; Dev = GCC_dev ###
################################################################################
-------------------------------------------------------------------------------
AerosolMass: GCC_dev differs from GCC_ref
Diagnostics that differ
AerMassHMS
AerMassOPOA
-------------------------------------------------------------------------------
Aerosols: GCC_dev differs from GCC_ref
Diagnostics that differ
AerNumDensityStratParticulate
Chem_AeroRadiMDUST1
Chem_AeroRadiMDUST2
Chem_AeroRadiMDUST3
Chem_AeroRadiMDUST4
Chem_AeroRadiMDUST5
Chem_AeroRadiMDUST6
Chem_AeroRadiMDUST7
Chem_KhetiSLAClNO3HBr
Chem_KhetiSLAHOBrHBr
Chem_KhetiSLAHOClHBr
Chem_KhetiSLAN2O5HCl
... and 9 others
-------------------------------------------------------------------------------
Emissions: GCC_dev differs from GCC_ref
Diagnostics that differ
EmisACET_Anthro
EmisALD2_Anthro
EmisALK4_Aircraft
EmisALK4_Anthro
EmisALK4_Total
EmisBCPI_Aircraft
EmisBCPI_Anthro
EmisBCPI_Total
EmisBCPO_Anthro
EmisBCPO_Total
EmisBENZ_Anthro
EmisBENZ_Total
... and 181 others
-------------------------------------------------------------------------------
JValues: GCC_dev differs from GCC_ref
Diagnostics that differ
Jval_BALD
Jval_BENZP
Jval_BZCO3H
Jval_CHCl3
Jval_ETHP
Jval_HC5A
Jval_MPAN
Jval_NIT
Jval_NITs
Jval_NPHEN
-------------------------------------------------------------------------------
Metrics: GCC_dev is identical to GCC_ref
-------------------------------------------------------------------------------
SpeciesConc: GCC_dev differs from GCC_ref
Diagnostics that differ
SpeciesConc_AONITA
SpeciesConc_AROMP4
SpeciesConc_AROMP5
SpeciesConc_AROMRO2
SpeciesConc_BALD
SpeciesConc_BENZO
SpeciesConc_BENZO2
SpeciesConc_BENZP
SpeciesConc_BZCO3
SpeciesConc_BZCO3H
SpeciesConc_BZPAN
SpeciesConc_C2H2
... and 24 others
-------------------------------------------------------------------------------
StateMet: GCC_dev differs from GCC_ref
Diagnostics that differ
FracOfTimeInTrop
Met_AD
Met_CLDF
Met_DELP
Met_DELPDRY
Met_DQRCU
Met_DQRLSAN
Met_DTRAIN
Met_OMEGA
Met_OPTD
Met_PMID
Met_QI
... and 15 others@msulprizio @lizziel let me know if you like this format |
|
Also fixed the output so that it prints "too many diffs" at the top of the header if the string with the names of the species that are different gets beyond ~80 chars. #########################################################################################
### Global mass (Gg) at end of simulation (Trop + Strat) ###
### Ref = GCC_ref; Dev = GCC_dev ###
### ###
### Species that differ btw GCC_ref and GCC_dev: ###
### ... Too many diffs to print (see below for details) ###
#########################################################################################
Ref Dev Dev - Ref % diff diffs
A3O2 : 0.076001 0.099527 0.023526 30.956 *
ACET : 6386.371096 6221.706425 -164.664671 -2.578 * |
|
(I'm using the GCPy test data so in the actual benchmarks there will be the version string instead of "GCC_ref" and "GCC_dev", etc |
benchmark/run_benchmark.py
- Now call "make_benchmark_summary_table" for GCC vs GCC, GCHP vs GCC,
and GCHP vs GCHP benchmarks.
benchmark/1mo_benchmark.yml
- Added "summary_table" switch under "plot_options"
gcpy/benchmark.py
- Removed typo at top of script
- In create_benchmark_emissions_table and create_global_mass_table:
- Write an alternate message if the string with species differences
is longer than ~80 chars
- Added new function "diff_list_to_text"
- Added new function "make_directory"
- Updated "create_benchmrk_summary_table" to call get_filepaths for
either GCHP or GCC Ref and/or Dev data
gcpy/util.py
- In routine print_totals:
- Now keep track of which species have differences btw Dev & Ref
in "diff_list", which is passed to the calling routine
- Added new function "wrap_text"
- Added new function "array_equals"
CHANGELOG.md
- Updated accordingly
Signed-off-by: Bob Yantosca <[email protected]>
msulprizio
approved these changes
Feb 9, 2023
|
Thanks @yantosca! I like the use of asterisk to denote diffs next to species. This solves the issue of not having enough precision printed to see if there are very small diffs. I like the summary file too. Since whether there are global mass diffs, and what the species are, is in the summary time I wonder if we need the summary of diffs in the global mass header. Removing it would make the file less cluttered. Sorry, I know you put work into putting it in there. @msulprizio, what do you think? |
lizziel
requested changes
Feb 9, 2023
|
@lizziel: I now restrict the summary of diffs to about 80 chars, or else it'll be "Too many diffs". Is that OK? |
@lizziel That's a good point. What if at the top of the global mass tables we instead include either "Zero differences between Ref and Dev" or "Ref and Def differ for [number] species". The idea of including a line at the top was to avoid having to scroll through the list to see if there were differences or not. |
|
@llzziel, I can implement that. Thanks. @msulprizio, @lizziel: Am also working on a fix in compare_varnames. As it happens, the Metrics was not identical even if in the summary page it was. |
gcpy.util.py
- In function compare_varnames
- Add variable names to commonvarsData if they have lon/Xdim or
lat/Ydim coordinates. Plottable data variables need at least
these dimensions.
- commonvarsOther are variables that are not in commonvarsData
- commonvars2D and commonvars3D are plottable data variables
that either do not have lev/ilev dimensions, or do have them.
This fixes an issue where 2D plottable data variables were getting
lumped together with the index variables.
Signed-off-by: Bob Yantosca <[email protected]>
gcpy/util.py
- In routine array_equals:
- Add dtype parameter to select numeric type for the comparison
- Return the inverse of ABS(devsum-refsum) > 0, since this is the
condition for not equals.
gcpy/benchmark.py
- Skip reading "AREA" when computing the summary table
- Pass dtype=np.float32 to util.array_equals
Signed-off-by: Bob Yantosca <[email protected]>
I like this idea! |
gcpy/benchmark.py: - Updated routine diff_list_to_text so that it also accepts refstr & devstr args. It now returns either line saying "dev & ref are identical" or "dev & ref show X differences" - Now pass devstr and refstr in calls to diff_list_to_text - Trimmed whitespace Signed-off-by: Bob Yantosca <[email protected]>
|
@lizziel @msulprizio, we now show: GCC_dev and GCC_ref show 14 differences
#########################################################################################
### Emissions totals for species ACET [Tg] ###
### Ref = GCC_ref; Dev = GCC_dev ###
#########################################################################################
Ref Dev Dev - Ref % diff diffs
ACET Anthro : 0.000000 0.000000 0.000000 nan
ACET BioBurn : 0.342551 0.342551 0.000000 0.000
ACET Biogenic : 4.967504 4.364605 -0.602899 -12.137 * or #########################################################################################
### Global mass (Gg) at end of simulation (Trop + Strat) ###
### Ref = GCC_ref; Dev = GCC_dev ###
### ###
### GCC_dev and GCC_ref show 292 differences ###
#########################################################################################
Ref Dev Dev - Ref % diff diffs
A3O2 : 0.076001 0.099527 0.023526 30.956 *
ACET : 6386.371096 6221.706425 -164.664671 -2.578 * |
|
Or it says "dev and ref are identical", with the string for dev & ref versions |
|
Thanks @lizziel @msulprizio! |
yantosca
added a commit
that referenced
this pull request
Feb 9, 2023
This brings some later fixes for issues with the creation of text saying whether or not 2 version have identical results in the table output. Signed-off-by: Bob Yantosca <[email protected]>
hannahnesser
pushed a commit
to hannahnesser/gcpy
that referenced
this pull request
Jan 22, 2024
…eoschem#204 This brings some later fixes for issues with the creation of text saying whether or not 2 version have identical results in the table output. Signed-off-by: Bob Yantosca <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the companion PR to #199. We propose to add additional text in the benchmark plots & tables to show which diagnostics are identical between benchmark Ref & Dev versions.
Tagging @msulprizio @lizziel