Build and Run Recipes

All recipes assume the code was downloaded recursively and you are in a clean directory one level into the build. E.g.,

git clone --recursive [email protected]:lanl/phoebus.git
mkdir -p phoebus/build
cd phoebus/build

Specific Problems

Unit tests

cmake -DPHOEBUS_ENABLE_UNIT_TESTS=ON ..
make -j
make test

Blast wave 3D

Many small meshblocks. Lots of mesh refinement.

cmake ..
make -j
./src/phoebus -i ../inputs/blast_wave_3d

Linear modes 3D

Uniform grid problem. Not very stressful on the infrastructure.

cmake ..
make -j
./src/phoebus -i ../inputs/blast_wave_3d

Torus

Realistic production problem

cmake -DPHOEBUS_GEOMETRY=FMKS -DPHOEBUS_CACHE_GEOMETRY=ON .. 
make -j
./src/phoebus -i ../inputs/torus.pin

Disabling HDF5

If outputs are not desired (which is possible) compile with the additional cmake option

-DPHOEBUS_ENABLE_HDF5=OFF

Building HDF5 manually

The following is the appropriate recipe for building HDF5 parallel.

• Download here

tar -xvzf hdf5-1.14.0.tar.gz # for example. May need to change version
cd hdf5-1.14.0.tar.gz
mkdir -p ${HOME}/local # or similar
# this assumes mpicc for the relevant MPI is in your path
CC=mpicc ./configure --prefix=${HOME}/local --enable-build-mode=production --enable-hl --enable-symbols=yes --enable-parallel

In the cmake configuration for Phoebus, you will then need to populate the prefix path to point to hdf5:

-DCMAKE_PREFIX_PATH=${HOME}/local

Host-pinned memory

While building cuda-aware MPI is beyond the scope here, if cuda-aware MPI is not available, you can enable host-pinned memory in the configure stage of CMAKe with

-DPARTHENON_ENABLE_HOST_COMM_BUFFERS=ON

Disabling MPI

If MPI is not desired, configure with -DPHOEBUS_ENABLE_HDF5=OFF

Enable OpenMP

To run with OpenMP looping, use -DPHOEBUS_ENABLE_OPENMP=ON

Disabling HDF5 Compression

You may need to configure with -DPARTHENON_DISABLE_HDF5_COMPRESSION=ON in some circumstances. The cmake error message will tell you.

MPI

Phoebus is inherently MPI-capable. If you built the code with MPI (the default) you can simply prefix the run command with mpirun -n N where N is the number of ranks.

Cmake package cache

Phoebus relies on submodules to build dependencies in-tree. cmake however, can interfere with this by hiding installation information for built packages in a local user path. If your build fails you may try deleting this path by deleting the .cmake directory in your home directory.

Cuda

You can enable cuda offloading with -DPHOEBUS_ENABLE_CUDA=ON

Changing compilers

On an HPC system, it should be enough to change the modules loaded. But if it isn't, use

-DCMAKE_CXX_COMPILER=<path/to/compiler>

You can also tell kokkos to optimize for a particular architecture with -DKokkos_ENABLE_<architecture_name>=ON.

Changing runtime parameters

You can change input parameters on the command line, rather than the input deck. For example, tro run the blast wave 3D for only 10 cycles, since the simulation is likely expensive, use the following run command:

./src/phoebus -i ../inputs/blast_wave_3d parthenon/time/nlim=10

where parthenon/time/nlim is the maximum number of cycles before the simulation ends.

Some other parameters that may be worth modifying are:

• parthenon/mesh/nx1, parthenon/mesh/nx2, and parthenon/mesh/nx3. Integers. These control the total problem size and specify the resolution of the base grid
• parthenon/meshblock/nx1, parthenon/meshblock/nx2, and parthenon/meshblock/nx3. Integers. These control the size of meshblocks, which are the pieces out of which the AMR mesh is constructed. parthenon/mesh/nx* must be evenly divisible by partheon/meshblock/nx*.
• parthenon/mesh/pack_size. Integer. Controls how meshblocks are coalesced into loops. -1 is the default and means all meshblocks on an MPI rank are coalesced. 1 means loops are per meshblock. N means loops contain N meshblocks.

Hardware specific recipes

Darwin volta-x86 partition without nvhpc

module load gcc/9.4.0 openmpi/3.1.6-gcc_9.4.0 cuda/11.4.2 cmake/3.22.2

• Build HDF5 as described above

cmake -DPHOEBUS_ENABLE_CUDA=ON -DPARTHENON_ENABLE_HOST_COMM_BUFFERS=ON -DCMAKE_PREFIX_PATH=${HOME}/local/x86-volta -DPARTHENON_DISABLE_HDF5_COMPRESSION=ON ..
make -j