Regression Test Policy for UFS Platforms and Compilers

Overview

The goal of the policy is to help define and communicate a strategy for maintaining and supporting UFS applications using a variety of platforms and compilers. The goals and current status are application-specific, i.e., different applications can have different goals. The portability of the UFS code base as a whole is defined by the portability of its applications. Similarly, the portability of each UFS application is defined by the portability of its components.

This policy pertains to the critical branches of the UFS umbrella repositories for each application. In other words, before code is brought into branches/repositories that are (1) in use by the UFS applications or (2) are authoritative branches from which critical branches pull, it must conform to this policy.

It is understood that the UFS is a community modeling system with components that have their own governance, code management, and portability goals. While the UFS community cannot impose requirements to the component repositories, it can make requests and suggestions.

It is recognized that the desirable portability goals differ from the current status.

The UFS portability is organized around a three-tier system, as described in the next section.

Tiered System for Platform and Compiler Support

Where possible, generic systems are used instead of actual computer names. For both generic systems and specific computers, specifics of the operating system, compiler, MPI library, and the remaining software stack should be provided because regression testing and continuous integration depend heavily on vetted combinations. The list of systems in each tier should be revisited regularly.

Tier-1: Computing systems used for operations and significant development efforts. All regression tests must pass on all Tier-1 systems before code changes are merged into the critical branches. For these systems, build configurations are provided. The compilers used in the Tier-1 platforms must conform to the following:

• compiler option template provided
• utilized for continuous integration
• guaranteed to pass full regression test suite
• compilation and execution guaranteed

Tests are run for every commit a priori.

Tier-2: Computing systems with evidence of utilization within the community. Regression tests may be conducted after the commits. The compilers used in the Tier-2 platforms must at a minimum conform to the following:

• compiler option template provided
• periodic runs of regression test suite
• no guarantee of correctness or reproducibility of results
• compilation and execution guaranteed
• if compilation/execution are broken because regression tests are run after commits, these must be fixed within a certain timeframe. Ideally, broken code is fixed with the following commit, but at the latest four weeks after the offending commit was made (to reduce the number of merge conflicts)

It is envisioned that compilers in this tier include those transitioning up or down from the top tier. For example, Intel v17 has been downgraded in favor of Intel v18. Likewise, Intel v19 has been released and is under evaluation for promotion to the top tier.

Tests are not run for every commit, but should run at least once a month (ideally once a week).

Tier-3: Other computing system types. The compilers used in the Tier 3 platforms must at a minimum conform to the following:

• compiler option template provided

There is no guarantee of compilation or correct execution. Compilers in this tier have a small user base or are known to have issues that make it unsuitable for consideration of support.

Roles and Responsibilities

For each of the platforms and compilers in Tier-1 and Tier-2, a team must be designated to ensure compliance with the portability requirements. For Tier-1 systems, the team must run the regression tests prior to any commit that updates the authoritative repositories for the particular application and for any commit that updates the upstream repositories of these. The team is also responsible for maintaining the official regression test baseline on this system and serves as a primary point of contact for user requests on this system. Broken compilation or execution force a halt on planned commits.

For Tier-2 systems, the team must run the regression tests either on a regular schedule (i.e., weekly) or following commits that update the authoritative repositories of the application or their upstream repositories. The regression test baseline must be kept up-to-date within a certain timeframe, to be defined by each application. Broken compilation or execution needs to be addressed by the team in collaboration with the developers and the code managers. The team also serves as primary point of contact for user requests and issues on this system.

Note: Primary point of contact does not mean that the respective team has to solve the problem; they can forward the issue as necessary (e.g., to HPC sysadmins or to developers).

Policy Implementation

It is up to each UFS application code management team to define what platforms should be considered Tiers 1, 2, 3. Please refer to the GitHub wiki for specific applications for more information. For example, for the UFS Short-Range Weather (SRW) Application, visit https://github.com/ufs-community/ufs-srweather-app/wiki.