Parameters

KHARMA has a large number of runtime parameters, which configure everything about its operation from physical theory to problem setup to debugging output.

Parameters are divided into blocks of related options. An individual parameter on the command line is denoted block/parameter_name=value e.g. to change the simulation end time, parthenon/time/tlim=100000 or whatever. Nearly all blocks are optional, with the exception of the <parthenon/x> blocks, <GRMHD>, and <coordinates>. Defaults are listed only when they seem relevant. Most booleans default to false.

<parthenon/job>

Any block beginning parthenon/ consists of global options, generally implemented by Parthenon and not KHARMA. The full list of Parthenon parameters is in the Parthenon documentation.

• problem_id is the name of the physical problem KHARMA will set up. It is also prefixed to all output files.
• archive_parameters creates a parameter file with all known runtime parameters, for reproducibility -- this way, defaults, runtime-changed parameters, or additional command line parameters are all captured in one file, the idea being that with this parameter file, and using the same commit of KHARMA, one can reproduce the same simulation. Note that regardless of this setting, a copy of this file is also included in every dump file, as the File attribute of the Input group.

<parthenon/mesh>

• refinement: none, static, or adaptive. Details of static or adaptive mesh refinement are specified in later blocks.
• numlevel: maximum total number of refinement levels to use, with 1 being no refinement.
• nx1, nx2, nx3: Size of the (base-level, Eulerian, evenly-spaced) mesh. KHARMA uses transformed coordinate systems to compress these where they are useful, see the <coordinates> block.

Non-spherical systems

In spherical geometry the following parameters are unnecessary: KHARMA automatically sets these variables, using coordinates/r_out, coordinates/r_in if specified, and assuming a full sphere. Note that Parthenon examples and old parameter files sometimes set, e.g., ix1_bc in this section, but these should never be set in KHARMA parameter files. Use <boundaries> instead.

• x1min, x1max, x2min, x2max, x3min, x3max: boundaries of the mesh in the $X^1$, $X^2$, and $X^3$ coordinates. Note that if you are applying a coordinate transformation, these are specified in the "native" coordinates of the code, not the "base" coordinates $r$, $\theta$, $\phi$.

<parthenon/meshblock>

• nx1, nx2, nx3: size of a meshblock. Must evenly divide into the mesh size.

<parthenon/static_refinementX>

• x1min, x1max, x2min, x2max, x3min, x3max: Region to refine. Any block overlapping this region will be refined to the specified level, then adjacent blocks will be refined to one level fewer, and so on to keep proper nesting (no block bordering a block more than one refinement level higher or lower). In KHARMA (but not Parthenon generally) these are specified as a proportion of the total lengths, i.e., coordinates/x1min = 0, coordinates/x1max = 1. This makes the simple refinement geometries usually used in KHARMA (e.g., "refine the midplane close to EH" or "refine the funnel wall") a bit easier and more robust to tweaks in the coordinate transformation, domain size, or other parameters.
• level: Level to refine all overlapping blocks. 0 indicates no refinement, 1 refines by a level over the base grid, etc. Note that a grid with 1 level of refinement in addition to the base has parthenon/mesh/numlevel = 2, but parthenon/static_refinement0/level = 1.

<coordinates>

These parameters detail the coordinate transformation between "native" coordinates (where the code evenly spaces zones and generally does its calculations) and the "embedding" coordinates (whatever simpler coordinate system covers the desired spacetime, e.g. spherical Kerr-Schild coordinates or Cartesian coordinates in Minkowski space). Note that extra parameters for a different

• base: spherical_ks aka ks, spherical_bl aka bl, spherical_minkowski, cartesian_minkowski aka minkowski. Base or "embedding" coordinates describing the desired space-time. The ks and bl systems describe a Kerr metric -- set a to zero for a Schwarzchild black hole. Several KHARMA forks implement more exotic metrics with various coordinate systems.
• transform: null aka none, exponential aka exp aka eks, superexponential aka superexp, modified aka mks, funky aka fmks, widepole aka wks. Coordinate transformations. Most transformations take the radial coordinate $r \rightarrow \log(r)$, with different strengths and modes of compressing zones in $\theta$. See the wiki page or Wong et al. for details of "modified" and "funky" systems. "Superexponential" is as described in Tchekhovskoy et al. and implemented in HARMPI. "Wide-pole" coordinates widen only the last zone next to the pole, and are described in full in Cho et al. Appendix C.
• r_in, r_out: In spherical coordinates, these set the domain size. Only r_out must be set -- if r_in is not specified, KHARMA will place 5 zones inside the event horizon to ensure that boundary conditions do not causally affect material outside the EH by lying within the same reconstruction stencil.
• ext_g: Whether to include an external gravity term, for certain very large-scale simulations (see Cho et al. eqn. 12)
• a: black hole spin, in the range $(-1, 1)$ for Kerr spacetimes
• r_br, npow, cpow: parameters for superexp; these and following parameters are coefficients in transformation formulae; consult the documentation above for specific meanings.
• hslope: parameter for mks. Defined negatively, so hslope = 1 corresponds to uniform spacing in $\theta$.
• mks_smooth, poly_xt, poly_alpha, fmks_zero_point: parametes for fmks. fmks_zero_point corresponds to startx1 on the wiki page -- it is used to make FMKS systems at different spin and radius more consistent, and avoid weird coordinate crossover in ghost zones.
• lin_frac, smoothness: parameters for wks

<parthenon/time>

Time steps and limits. Note KHARMA does not support all of the time-stepping options listed in Parthenon's documentation, as most were added very recently. Instead, it supports:

• tlim: simulation time limit
• nlim: maximum number of steps. 0 is valid, -1 sets an unlimited number of steps.
• use_dt_light: Use the light-crossing time as the timestep for safety, rather than computing it from signal speeds. Leads to much shorter timesteps early in a simulation, slightly shorter timesteps throughout.
• start_dt_light: Use the light-crossing time as the first timestep, but the signal speed thereafter.
• use_dt_light_phase_speed: Use the phase speed, rather than $c$, when calculating the light-crossing time
• dt: default/first timestep. Default: dt_min
• dt_min: minimum timestep. Default: 1e-5
• max_dt_increase: maximum proportional increase in dt from a step to the subsequent step. Default: 2.0

<GRMHD>

• cfl: Portion of the Courant-Friedrichs-Lewy stability maximum to use when setting the GRMHD timestep.
• gamma: Fluid adiabatic index. KHARMA supports only ideal equations of state with constant $\gamma$
• implicit: Evolve GRMHD variables semi-implicitly, solving for the next state including any time-dependent source terms. Turned on automatically when necessary, e.g. for Extended GRMHD simulations
• refine_tol, derefine_tol: minimum difference between the extrema of prims.rho before triggering refinement on an AMR grid. Maximum difference between extrema before derefining.
• sync_utop_seed: synchronize primitive variables, even when treating conserved variables as fundamental. This uses some MPI bandwidth but no latency, in order to ensure that primitive variable recovery happens identically in physical and ghost zones. Default: true

<flux>

• type: llf or hlle. Use the Local Lax-Friedrichs (LLF) flux common to HARM codes, or the more accurate but less stable Harten, Lax, van Leer and Einfeldt (HLLE) flux. KHARMA lacks a check for whether the latter is imaginary, which is not uncommon, so llf is default & recommended.
• reconstruction: donor_cell, donor_cell_c, linear_vl, linear_mc, weno5, weno5_linear, ppm, ppmx, or mp5. Reconstruction scheme used to find the values of primitive variables at faces, given values at adjacent cell centers. linear_mc and linear_vl use the monotonized central and Van Leer slope limiters, respectively. donor_cell_c is just an alternate implementation of donor_cell for testing and behaves identically. weno5_linear is a WENO5Z implementation which falls back to linear reconstruction according to smoothness indicators (Similar to WENO5-AO but simpler. "WENO5-AO at home," if you will). ppmx is extremum-preserving PPM. If you suspect the reconstruction of causing instabilities, fall back to linear_mc. Default: weno5
• low_order_edges, low_order_poles: use weno5 reconstruction over the interior domain, but linear_mc reconstruction on the x1 and x2 boundaries of the domain, respectively.
• reconstruction_floors: apply geometric floors to the reconstructed values at faces. Set by default for non-TVD reconstructions.
• reconstruction_fallback: use results of a TVD reconstruction to replace any values which would fall below geometric floors. False by default, overrides reconstruction_floors if both are set.
• consistent_face_b: When using Face-CT magnetic transport, use the existing values of corresponding magnetic field components already defined at faces, rather than reconstructed values. This is how constrained transport schemes generally work, and it is a bad idea to turn this option off. Default: true

<driver>

• type: kharma, imex, or simple. Details of the driver modes are provided on the Driver page.
• two_sync: whether to perform two MPI syncs per step, updating ghost zones again after primitive variables have been solved. On by default but likely safe to disable for simple situations using the default Kastaun primitive variable recovery and Face-CT magnetic field transport.

<boundaries>

All parameters with * can be enabled for any boundary by adding the name, e.g. check_inflow_inner_x1. Most such options are implemented for most boundaries, but generally they are supposed to be set by KHARMA according to the first few options. Overriding them is allowed for testing.

• check_inflow: test for inward-directed velocity and reset it to zero. Enabled by default for outflow boundaries.
• zero_polar_flux: disallow any flux over the pole itself, as the size of polar faces is zero in a full grid. Default: true
• excise_polar_flux: allow fluxes over the pole as if the last half-zone were excised, but otherwise treat the grid as full. Useful for making transmitting boundary conditions less disruptive. Overrides zero_polar_flux. Default: false
• domain_bounds_on_conserved: apply the physical boundaries to conserved variables, rather than primitive variables. Experimental for future performance improvements. Default: false
• fix_corner: re-apply inner X1 boundary conditions after applying X2 boundary conditions. Definitely necessary for outflow/reflecting conditions with Flux-CT magnetic field, questionably necessary with a more modern scheme. Default: true
• fix_corner_outer: fix the outer corner. Extremely uncommon to need this. Default: false
• check_inflow_*: test a particular boundary for inflow. Default for outflow boundaries.
• zero_flux_*: zero the flux through a particular boundary. Default for polar boundaries.
• reflect_face_vector_*: invert corresponding face-centered values on this boundary (e.g., X2 faces/B2 on X2/polar boundary). Default for polar reflecting boundaries.
• clean_face_B_*: reset corresponding faces (e.g., B1 on X1 boundaries) to eliminate divergence without respecting prior values or fluxes. Default for outflow boundaries.
• average_EMF_*: average together the perpendicular values of the EMF on this face. Used for polar boundary conditions, where EMF1 edges all occupy the same space at the pole, and for allowing coherent motion of the magnetic field at Dirichlet boundaries, where an identical EMF will not disturb the magnetic field in the boundary. Default for transmitting polar boundaries.
• zero_EMF_*: zero out EMFs on this face. Default for reflecting polar boundaries.
• reconnect_B3_*: forcibly "reconnect" B3, the $\phi$ component of magnetic field by subtracting the average, e.g. about the pole where it represents a small field loop which would cancel. Used to be used for stability of polar boundaries. Default: false
• cancel_U3_*: forcibly advect together U3, the $\phi$ component of velocity, by subtracting the average. Used to be used for stability of polar boundaries. Default: false
• cancel_T3_*: forcibly advect together T3, the angular momentum/$\phi$ component of the stress-energy tensor, by subtracting the average. Used to be used for stability of polar boundaries. Default: false

<b_field>

• solver: flux_ct, face_ct, constraint_damping, or cleanup. The mode of satisfying the constraint equation that there be no divergence of the magnetic field. The default flux_ct transport efficiently ensures that the divergence grows by, at maximum, a low multiple of the floating-point round-off error per step. face_ct ensures the same but requires more operations, with the benefit it is compatible with SMR/AMR and excised transmitting boundary conditions. constraint_damping and cleanup would require code development to be used at all, and considerable effort to be used on interesting problems.
• kill_on_large_divb: if the magnetic field divergence ever grows beyond a limit, stop the simulation.
• kill_on_divb_over: limit of acceptable divergence. Default: 1e-3
• initial_cleanup: clean the magnetic field before running the simulation, by solving the divergence-free constraint equation (equivalent to the poisson equation). Uses a Stabilized Biconjugate Gradient method (BiCGSTAB) for uniform grids. Upcoming KHARMA versions use a Multigrid preconditioned BiCGSTAB to solve on SMR/AMR grids as well. Enabled automatically when needed, only needs to be enabled manually for testing.
• implicit: evolve the magnetic field semi-implicitly by solving for the next state. Never needed currently, used to test the implicit solver.

flux_ct

• fix_polar_flux: In order for the corrected/"constrained" flux of the magnetic field through a face to be zero, certain conditions must be met in the
• fix_flux_*: Enable fix_flux_x1 if using Dirichlet conditions with magnetic field, if you wish to allow some limited magnetic flux through the face (while maintaining the Dirichlet condition). fix_flux_x2 is also available for testing in Cartesian system (spherical systems should always use fix_polar_flux). Each boundary can be individually controlled with e.g. fix_flux_inner_x1, etc.
• disable_flux_ct: Don't actually correct any fluxes of the magnetic field vector components. This allows the divergence to grow naturally for testing and comparison. Obviously, don't enable this unless you want the simulation to blow up.

face_ct

• ct_scheme: bs99, gs05_0, or gs05_c aka sg07. Method of choosing the EMF values along zone edges, which are used to calculate the change in magnetic field on zone faces in Face-CT. bs99 is Balsara & Spicer, gs05_0 is Gardiner & Stone $\mathcal{E}_z^\circ$ non-upwinded method (eqn. 40). gs05_c is the the $\mathcal{E}_z^c$ method with upwinding, eqn. 42 & 51, extended to 3D in Gardiner & Stone. Default: gs05_c

constraint_damping

• damping: Damping factor for reducing divergence variable $\psi$. Factor $\lambda$ in Dedner et al..

<inverter>

• type: kastaun, onedw, or none. The kastaun inverter implements the scheme from Kastaun et al., which is extremely robust and guarantees convergence to a reasonable set of primitive variables under a broad set of circumstances. onedw implements the $1D_W$ scheme from Noble et al.. If the kastaun implementation is too slow or (God forbid) unreliable, the interested user might consider implementing the robust Newton-Raphson solver of Cai et al.. If running a suitably non-relativistic problem, skipping inversion by specifying none may speed up execution a bit.
• err_tol: recovery error tolerance. Default: 1e-12 for kastaun, 1e-8 for onedw.
• iter_max: maximum number of iterations before the solve is marked as failed. Default: 25 for kastaun, 8 for onedw.

<floors>

All floors in KHARMA are disabled by default, except gamma_max which defaults to 50, as a maximum <= 50 is assumed in the source a few times for performance. Suitable floor "prescriptions" for torus runs can be found in the examples, and depend highly on the algorithm being used, especially the primitive variable recovery (see last section).

• disable_floors: disable all floor applications. Useful for tests, or simple simulations which will not clump or evacuate regions.
• frame: drift, normal, fluid, mixed_fluid_normal aka mixed, or mixed_fluid_drift. The rest frame in which new floor material is injected. Applies to floors on density $\rho$ and internal energy $u$. The drift frame
• rho_min_geom: geometrically falling minimum on $\rho$. Value represents the floor at $1 r_g$. Floor is scaled as $r^{-3/2}$ unless use_r_char is set, when it is scaled by $r^{-2} * 1/(1 + r/r_{char})$ (uncommon). Generally set low for tori, 1e-5 or 1e-6
• u_min_geom . Set below rho_min_geom, 1e-7 or 1e-8
• rho_min_const: constant $\rho$ floor over all space. Generally left at default 1e-20, can cause problems in spherical coordinates if too high. This floor is applied by the "Kastaun" inverter in the fluid frame, regardless of frame setting.
• u_min_const: constant $u$ floor over all space. Generally left at default 1e-20, can cause problems in spherical coordinates if too high. This floor is applied by the "Kastaun" inverter in the fluid frame, regardless of frame setting.
• u_over_rho_max: maximum temperature, enforced by adding density unless temp_adjust_u is set. Good panacea for explosions but not very accurate, a limit at 10 or worse yet 2 will give you a finished simulation, at a cost. This ceiling is applied by the "Kastaun" inverter by limiting internal energy, regardless of other settings.
• bsq_over_rho_max: magnetization ceiling, adding material to avoid advecting magnetic field in vacuum. Generally necessary: with FOFC and the Kastaun inverter, 1e3 or perhaps higher is fine, otherwise 1e2 or even lower may be necessary.
• bsq_over_u_max: limit on $\beta^{-1}$. Used to be set quite high (i.e., loosely), now generally set only a bit higher than bsq_over_rho_max to avoid adding material that is too cold, e.g. 1e4 for bsq_over_rho_max of 1e3.
• use_r_char: reduce the geometric $\rho$ and $u$ floors more quickly everywhere, and even more quickly outside a characteristic radius. Designed to prevent buildup of material from floors as it outflows.
• r_char: characteristic radius for dropping the geometric floors more quickly. Default: 10
• ktot_max: maximum entropy, always enforced by lowering the internal energy $u$. Can act as an effective temperature ceiling for low-density regions only, but is highly dependent on fluid $\gamma$. Strict values are around 3 for $\gamma$ of 4/3, 1500 for $\gamma$ of 5/3.
• temp_adjust_u: when applying the temperature ceiling, subtract from internal energy rather than adding to density.
• adjust_k: when applying density and internal energy floors, adjust the recorded entropy $k$ to match as described in Ressler et al.. Only meaningful in simulations tracking electron temperatures using dissipation, which otherwise use the scheme from that paper.
• gamma_max: maximum Lorentz factor. Tracking fast-moving material can be inaccurate, and very high velocities are often a symptom of otherwise troubled zones, e.g. with large momenta advected into a zone at the same time most density is advected out. This setting is also applied by the "Kastaun" inverter, and the value is assumed in the source code to be <= 50. Default: 50

mixed_fluid_normal

• frame_switch_r: the mixed_fluid_normal frame injects floors in the normal observer (coordinate) frame (NOF) within some radius, and in the fluid frame outside. This prevents some instabilities observed with NOF floors over large, empty domains. This controls the switch radius (which is a hard cut, not smoothed). Default: 50

mixed_fluid_drift

• frame_switch_beta: the mixed_fluid_drift frame injects floors in the drift frame above a base magnetization, but in the normal observer frame in demagnetized regions. This can improve the stability of drift frame floors if floors are hit often without much magnetic field present. This controls the maximum beta at which to use the drift frame. Default: 10

<floors_inner> and <fofc_floors>

There are two more sections which replicate the parameters from "floors," for different reasons:

• The <floors_inner> block allows specifying an entirely different set of floors to be applied within some radius. Setting floors/radius_dependent_floors to true and floors/floors_switch_r (default 50.) will apply floors from <floors_inner> at radii below the cut, and <floors> at larger radii. This is similar to the mixed_fluid_normal frame, but more flexible. Values for <floors_inner> default to the values set in <floors>.
• The <fofc_floors> block allows setting separate limit values for applying first-order flux corrections, described in the next section. Normally, corrections are applied to any zone which would fall below the limits in <floors>, but adding this block allows customizing how broadly FOFC is applied, e.g. to all regions at magnetization $\sigma$ > 1, or only in regions where the density or internal energy would be reconstructed negative (or in the case of the Kastaun inverter, reconstructed to their constant floor values).

<fofc>

First-order flux corrections (FOFC) attempt to avoid using simulation floors as often, by replacing fluxes around "troubled" zones with first-order versions, which are more diffusive but carry stronger guarantees about preserving positivity.

• on: whether to enable FOFC. Default: false
• use_glf: whether to use global Lax-Friedrichs fluxes (signal speed of $c$) in computing first-order fluxes. Default: false
• use_source_term: include the geometric source term when computing which zones will fall below the floors
• polar_cells: always enable FOFC this many cells from the pole. Default: 0
• eh_buffer: radius outside the black hole event horizon inside which FOFC is always enabled. Prevents use of HLLE fluxes or other high-order shenanigans at/inside the EH. Default: 0.1
• consistent_face_b: whether to use existing or reconstructed magnetic field values at matching faces (see flux/consistent_face_b). This can improve stability, and using reconstructed magnetic fields in the small percentage of FOFC cells is at least more defensible than enabling it everywhere. Default: false

<debug>

• verbose: level of output verbosity. 0 is quite terse, 1 enables normal archival output, 2 more debugging-oriented output.
• flag_verbose: verbosity in printing "flagged" zones: floors, inversion failures, solver, failures, flux corrections, etc. 0 omits printing, 1 prints total numbers of each flag, 2 breaks out which flags were hit.
• extra_checks: step-end checks for obviously undesirable states: 0 disables checks, 1 checks for zero or NaN signal speeds, 2 additionally checks for any negative $\rho$, $u$, and $\rho u^t$ after each step.

<parthenon/outputX>

• file_type: hdf5, rst, or hst. Type of output. Normal dump files are hdf5, files intended for restarting the simulation are rst, and the history output is hst (text file with diagnostics, e.g. flags hit, accretion rate, magnetization, magnetic field divergence). All history outputs will contain the same information, this is a Parthenon limitation.
• dt: output frequency in simulation time units
• single_precision_output: whether to output in single precision, to save disk space
• ghost_zones: whether to include ghost zones in output. Not needed for rst files any more, optional for hdf5 files.
• variables: which variables to output. Listing prefixes outputs all variables with that prefix, e.g. "prims" for prims.rho, prims.u, prims.uvec, (if present) prims.B, and any other variables with that prefix. jcon outputs the 4-current, which takes a bit of extra computation time per output. fflag indicates floor hits during the step that produced the output, pflag indicates inversion failures, and a few other flags exist for semi-implicit/Extended MHD simulations. Generally, any Parthenon field (anything added with AddField in a package's Initialize function) can be added to the list of variables to output in an hdf5 file. variables may also be specified for hst files, to add extra diagnostic columns to the output. conserved_vars tallies totals of conserved variables, luminosities totals jet and luminosity diagnostics, eh_fluxes_cell totals accretion rates at the EH using cell-centered values, and eh_fluxes_flux uses the face-centered fluxes themselves. all_reductions just enables everything.