[DISCUSSION] Profiling GCHP Performance using external utilities

[lizziel]

Discussed in https://github.com/geoschem/GCHP/discussions/142

^{Originally posted by WilliamDowns May 11, 2021}
This issue is intended for discussing profiling GCHP performance using various external utilities such as TAU, IPM, Intel VTune, etc.

In my testing I found that using TAU causes GCHP to hang during ExtData initialization when using the -ebs -ebs_resolution=function option with tau_exec. I've successfully completed runs using mpirun -np ${SLURM_NPROCS} tau_exec ./gchp >> ${log}, which outputs profiling data at the MPI call level.

I'm unable to compile GCHP with TAU, as CMake hangs when I set my Fortran compiler variables to TAU's requested compiler script (whether through setting FC=$(spack location -i tau)/bin/tau_f90.sh or using DCMAKE_Fortran_COMPILER=$(spack location -i tau)/bin/tau_f90.sh).

My system + run config:

• AWS ParallelCluster 2.10.1, centos7
• GNU Compilers 9.3.0
• Intel MPI 2019U10
• TAU 2.30.1
• GCHP 13.0.2
• c90
• 9 nodes / 324 cores (8 model nodes and 1 output node)
• Emissions, Metrics, SpeciesConc diagnostics
• 1 week simulation

I also found that IPM does not compile with GCC 9.3.0, but does compile when using ifort 2021.2.

[lizziel]

We are switching from GitHub Discussions to GitHub Issues. I am going to delete the Discussion now. The history of the thread is as follows:

@lizziel:

@mathomp4, @tclune: could you comment on what tools you currently use to profile GEOSgcm / GEOSctm performance?

@mathomp4:

I'm going to ping @aoloso here as well as he's done more profiling recently compared to me. I think the last few times I did so was with ARM MAP and Intel Vtune. TAU has always been my archenemy. You get it working and then the next day it doesn't work.

@lizziel:

@mathomp4, did you hear back form anyone about tools recently used for profiling GEOS?

@mathomp4:

@lizziel As I said, MAP, VTune and TAU are the three tools I and @aoloso have used for GEOS. I tend to use MAP first as it is usually the most user friendly, though it can't get the detail that TAU does.