O1.1.1 GPU performance improvements

[sriharshakandala]

Purpose

Achieve a computational throughput of >8 SYPD (simulated years per day) on 4 (A100) GPUs for the dynamical core at 100 km horizontal resolution, with strong scaling efficiency exceeding 85% when using 16 GPUs.

Cost/benefits/risks

It is expected to take about a month and a half to implement these tasks.

Producers

@simonbyrne @sriharshakandala @charleskawczynski

Components

Inputs

Results and deliverables

A description of the key results, deliverables, quality expectations, and performance metrics.
...

Reviewers

The names of CliMA personnel who will review the Proposal and subsequent PRs.

Horizontal kernel improvements

Preview Give feedback

1. 3D spectral element curl Add 3D spectral element curl #1425
2. Better use of threading #1431
   +0
   
3. Implement reference kernels, use them for tuning

Vertical kernel improvements

Preview Give feedback

1. Better use of threading Thread-per-node stencil operator kernels #1398
2. QA examples
3. Jacobian

CliMATimeSteppers

Preview Give feedback

1. Only call set_precomputed_quantities! once in the implict stage ClimaAtmos.jl#2015
   +0
   
2. Add a callback after each stage ClimaTimeSteppers.jl#193
   +0
   

Miscellaneous

Preview Give feedback

1. DSS doesn't work for Covariant123Vectors #1443
   bug +1
   
2. Identify key GPU bottlenecks, and replicate in sample script #1361
   +0
   
3. Inline all the things #1455
   enhancement +1
4. Add multi-GPU performance case to pipeline ClimaAtmos.jl#2154
   +0
   

CC

@simonbyrne @cmbengue @tapios