Document standard means to run unit tests

[sadielbartholomew]

I can see that there are a set of unit tests for qgs under the qgs/model_test/ directory, yet there is no guidance on how to run them, and trying standard means leads to errors. Specifically:

• I cannot see any description within the documentation or README (or indeed elsewhere) about the intended way (i.e. commands) to run them, which is important to provide because I have seen many codebases where the run directory of tests influences the outcome, and moreover it may be that the tests need to run in a certain order, etc.;
• many of the tests error when I try common commands to initiate Python unit tests, in this case pytest in either under the root directory of the repo or qgs/model_test/, or running each via python <unit test name> directly, after directly following the installation instructions provided such that python qgs_rp.py executed in the root directory runs to completion (see detail below if helpful).
• other tests, namely test_base.py, quickly (~2s) runs and provides a success exit code without providing any output to STDOUT or STDERR such that I am left unsure as to whether the test passed or failed, or even ran.

So overall I, and other newcomers could well also be, left uncertain as to whether I am running the tests as intended, and to whether the code is providing the correct functionality.

This Issue is raised towards my review in openjournals/joss-reviews#2597.

Detail

This works:

(qgs) $ python qgs_rp.py

Model qgs (Atmosphere + orography configuration)
================================================

Initialization ...

Qgs parameters summary
======================

General Parameters:
'time_unit': days,
'rr': 287.058  [J][kg^-1][K^-1]  (gas constant of dry air),
'sb': 5.67e-08  [J][m^-2][s^-1][K^-4]  (Stefan-Boltzmann constant),

Scale Parameters:
'scale': 5000000.0  [m]  (characteristic space scale (L*pi)),
'f0': 0.0001032  [s^-1]  (Coriolis parameter at the middle of the domain),
'n': 1.3    (aspect ratio (n = 2 L_y / L_x)),
'rra': 6370000.0  [m]  (earth radius),
'phi0_npi': 0.2777777777777778    (latitude exprimed in fraction of pi),
'deltap': 50000.0  [Pa]  (pressure difference between the two atmospheric layers),

Atmospheric Parameters:
'kd': 0.1  [nondim]  (atmosphere bottom friction coefficient),
'kdp': 0.01  [nondim]  (atmosphere internal friction coefficient),
'sigma': 0.2  [nondim]  (static stability of the atmosphere),

Atmospheric Temperature Parameters:
'hd': 0.1  [nondim]  (Newtonian cooling coefficient),
'thetas[1]': 0.2    (spectral components 1 of the temperature profile),
'thetas[2]': 0.0    (spectral components of the temperature profile),
'thetas[3]': 0.0    (spectral components of the temperature profile),
'thetas[4]': 0.0    (spectral components of the temperature profile),
'thetas[5]': 0.0    (spectral components of the temperature profile),
'thetas[6]': 0.0    (spectral components of the temperature profile),
'thetas[7]': 0.0    (spectral components of the temperature profile),
'thetas[8]': 0.0    (spectral components of the temperature profile),
'thetas[9]': 0.0    (spectral components of the temperature profile),
'thetas[10]': 0.0    (spectral components of the temperature profile),

Ground Parameters:
'hk[1]': 0.0    (spectral components of the orography),
'hk[2]': 0.2    (spectral components 2 of the orography),
'hk[3]': 0.0    (spectral components of the orography),
'hk[4]': 0.0    (spectral components of the orography),
'hk[5]': 0.0    (spectral components of the orography),
'hk[6]': 0.0    (spectral components of the orography),
'hk[7]': 0.0    (spectral components of the orography),
'hk[8]': 0.0    (spectral components of the orography),
'hk[9]': 0.0    (spectral components of the orography),
'hk[10]': 0.0    (spectral components of the orography),


Starting a transient time integration...
Starting the time evolution ...
Evolution finished, writing to file evol_fields.dat
Time clock :
21.707646417 seconds

But the tests, excluding test_base.py, when run in some sensible way as a guess, seem to return an error similar to:

(qgs) $ pwd
/home/sadie/qgs
(qgs) $ pytest
============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/sadie/qgs
collected 0 items / 2 errors                                                   

==================================== ERRORS ====================================
_________________ ERROR collecting model_test/test_aotensor.py _________________
ImportError while importing test module '/home/sadie/qgs/model_test/test_aotensor.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../anaconda3/envs/qgs/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
model_test/test_aotensor.py:5: in <module>
    from params.params import QgParams
E   ModuleNotFoundError: No module named 'params'
______________ ERROR collecting model_test/test_inner_products.py ______________
ImportError while importing test module '/home/sadie/qgs/model_test/test_inner_products.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../anaconda3/envs/qgs/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
model_test/test_inner_products.py:7: in <module>
    from params.params import QgParams
E   ModuleNotFoundError: No module named 'params'
=========================== short test summary info ============================
ERROR model_test/test_aotensor.py
ERROR model_test/test_inner_products.py
!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!
============================== 2 errors in 0.23s ===============================

with test_base.py running very quickly with no output:

(qgs) $ time python test_base.py 

real	0m0.066s
user	0m0.065s
sys	0m0.001s
(qgs) $ echo $?
0

[jodemaey]

Hi,

Thank you for raising that issue.
This issue is addressed by the commit:

c6a7dee

in the joss branch. It will be merged into the master when the review process is completed and this issue will then be closed.

The tests can now be executed using pytest. test_base.py contains a base class for the tests and is not used directly.

A recast of the testing system with proper documentation is planned for the v0.3 (or v0.4 at least if we are late).

[sadielbartholomew]

Thanks @jodemaey, I have ran the tests via the newly-documented means and they all pass:

$  pytest
============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/sadie/qgs
collected 4 items                                                              

model_test/test_aotensor.py .                                            [ 25%]
model_test/test_aotensor_6x6.py .                                        [ 50%]
model_test/test_inner_products.py .                                      [ 75%]
model_test/test_inner_products_6x6.py .                                  [100%]

======================== 4 passed in 418.87s (0:06:58) =========================

So I now consider this closed.