Merge branch 'nci:develop' into feature/spearman_rank

.zenodo.json

@@ -16,7 +16,7 @@
         },
         {
             "orcid": "https://orcid.org/0009-0009-3207-4876",
-            "affiliation": "Bureau of Meteorology, Australia",
+            "affiliation": "Work undertaken while at the Bureau of Meteorology, Australia",
             "name": "Cook, Harrison"
         },
         {

README.md

@@ -24,7 +24,7 @@ Below is a **curated selection** of the metrics, tools and statistical tests inc
 | **[Spatial](https://scores.readthedocs.io/en/stable/included.html#spatial)** 	|Scores that take into account spatial structure.                 	|Fractions Skill Score.              	|
 | **[Statistical Tests](https://scores.readthedocs.io/en/stable/included.html#statistical-tests)** 	|Tools to conduct statistical tests and generate confidence intervals.                 	|Diebold Mariano.              	|
 | **[Processing Tools](https://scores.readthedocs.io/en/stable/included.html#processing-tools-for-preparing-data)**        	|Tools to pre-process data.                 	|Data matching, Discretisation, Cumulative Density Function Manipulation.              	|
-
+| **[Emerging](https://scores.readthedocs.io/en/stable/included.html#emerging)**        	|Emerging scores that are still undergoing mathematical peer review. They may change in line with the peer review process.                 	|Risk Matrix Score.             	|
 
 `scores` not only includes common scores (e.g., MAE, RMSE), it also includes novel scores not commonly found elsewhere (e.g., FIRM, Flip-Flop Index), complex scores (e.g., threshold weighted CRPS), and statistical tests (e.g., the Diebold Mariano test). Additionally, it provides pre-processing tools for preparing data for scores in a variety of formats including cumulative distribution functions (CDF). `scores` provides its own implementations where relevant to avoid extensive dependencies.
 

docs/api.md

@@ -52,6 +52,7 @@
 .. autofunction:: scores.probability.murphy_thetas
 .. autofunction:: scores.probability.roc_curve_data
 .. autofunction:: scores.probability.brier_score
+.. autofunction:: scores.probability.brier_score_for_ensemble
 .. autofunction:: scores.probability.isotonic_fit
 ```
 
@@ -110,3 +111,9 @@
 .. autofunction:: scores.pandas.continuous.mae
 ```
 
+## scores.emerging
+```{eval-rst}
+.. autofunction:: scores.emerging.risk_matrix_score
+.. autofunction:: scores.emerging.matrix_weights_to_array
+.. autofunction:: scores.emerging.weights_from_warning_scaling
+```

docs/conf.py

@@ -9,7 +9,7 @@
 
 project = "scores"
 copyright = "Licensed under Apache 2.0 - https://www.apache.org/licenses/LICENSE-2.0"
-release = "2.0.0"
+release = "2.1.0"
 
 version = __version__
 

docs/contributing.md

@@ -25,15 +25,23 @@ Please submit bug reports, feature requests and feedback as issues in GitHub: [h
 
 Please see the information provided in [SECURITY.md](SECURITY.md)
 
+## Creating Your Own Fork  of `scores` for the First Time
+
+Unless you are an advanced Git user, we would recommend you follow this process:
+
+1. First (i.e. **before** cloning the `scores` repository) create your own fork using the GitHub web user interface.
+2. Clone **your fork**. (Do not clone github.com/nci/scores).
+3. Immediately create a new local branch, with a command such as `git checkout -b branch_name`.
+
 ## Workflow for Submitting Pull Requests
 
 Prior to developing a pull request, it may be a good idea to create a [GitHub issue](https://github.com/nci/scores/issues) to capture what the pull request is trying to achieve, any pertinent details, and (if applicable) how it aligns to the roadmap. Otherwise, please explain this in the pull request.
 
 To submit a pull request, please use the following workflow: 
-1. create a fork of the `scores` repository, 
-2. create a feature branch on your fork,
-3. keep your feature branch rebased and up-to-date with the `scores` develop branch,
-4. when ready, submit a pull request to the develop branch of `scores`.
+
+1. Ensure you are working on a new feature branch in **your fork** (see [the section above](#creating-your-own-fork-of-scores-for-the-first-time)).
+2. Keep your feature branch rebased and up-to-date with the develop branch of `scores`. You can do this by first syncing the develop branch on your fork, and then rebase your feature branch against the develop branch on your fork.
+3. When ready, submit a pull request to the develop branch of github.com/nci/scores.
 
 To help disambiguate branches, some contributors like to prefix their branch names with a short numerical indentifier. This is up to the contributor and any approach to branch naming is welcome. 
 
@@ -110,6 +118,18 @@ To automate linter and fixer checks this project uses `pre-commit` which is set
 pre-commit install -t pre-commit -t pre-push
 ```
 
+## Pull Request Etiquette
+
+In general, the originator of a pull request will be the person who does all the coding work, including responding to feedback from others. Typically, feedback will be provided in the form of comments or code suggestions made through the GitHub web user interface.
+
+Sometimes, it may be pragmatic for the package maintainers to make or request a more complex code change. This typically occurs when a complex Git operation is needed to keep a pull request (PR) up to date, to resolve conflicts, or to make changes where the originator of the PR does not know how to proceed. It can also occur in the final stages of a PR lifecycle if there are small tidyups needed and time is a factor.
+
+Not every possibility can be accounted for, and the package maintainers will (if needed) push code directly to a PR to get something over the line. However, special circumstances aside, the maintainers will first leave a comment on the PR asking how the originator would like to proceed, so that there are no surprises.
+
+Most of these kinds of code change can also be handled as a PR by the reviewer onto the fork of the originator. This is slightly slower (i.e. does not take effect immediately), but allows for more control by the originator. This is probably most developer's general preference.
+
+In short - once you have made a PR, the maintainers may then take it, modify it, or include it as-is. However, every effort will be made to communicate about that process and make sure that the originator of the PR is happy with any modifications made.
+
 ## Review Processes
 
 All pull requests will undergo a review process prior to being included in the develop branch in order to ensure both the coding and the scientific validity of changes. 

docs/included.md

@@ -172,6 +172,10 @@
   - [API](api.md#scores.probability.brier_score)
   - [Tutorial](project:./tutorials/Brier_Score.md)
   - [Brier (1950)](https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2)
+* - Brier Score for Ensembles
+  - [API](api.md#scores.probability.brier_score_for_ensemble)
+  - [Tutorial](project:./tutorials/Brier_Score.md)
+  - [Ferro (2013)](https://doi.org/10.1002/qj.2270)
 * - Continuous Ranked Probability Score (CRPS) for Cumulative Distribution Functions (CDFs)
   -    
   - 
@@ -743,4 +747,36 @@
   - [Wikipedia](https://en.wikipedia.org/wiki/Root-mean-square_deviation)
 ```
 
+## Emerging
+
+```{Caution} 
+  This section of the API contains implementations of novel metrics that are still undergoing mathematical peer review. These implementations may change in line with the peer review process.
+```
 
+```{list-table}
+:header-rows: 1
+
+* - Name (Alphabetical order)
+  - API
+  - Tutorial
+  - Reference(s)
+* - Risk Matrix Score
+  - 
+  - 
+  - 
+* -  
+    - Risk Matrix Score
+  - [API](api.md#scores.emerging.risk_matrix_score)
+  - [Tutorial](project:./tutorials/Risk_Matrix_Score.md)
+  - Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework with consistent evaluation. In preparation.
+* -  
+    - Risk Matrix Score - Matrix Weights to Array
+  - [API](api.md#scores.emerging.matrix_weights_to_array)
+  - [Tutorial](project:./tutorials/Risk_Matrix_Score.md)
+  - Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework with consistent evaluation. In preparation.
+* -  
+    - Risk Matrix Score - Warning Scaling to Weight Array
+  - [API](api.md#scores.emerging.weights_from_warning_scaling)
+  - [Tutorial](project:./tutorials/Risk_Matrix_Score.md)
+  - Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework with consistent evaluation. In preparation.
+```

docs/related_works.md

@@ -2,7 +2,7 @@
 
 Here are some related packages that may be of interest.
 
-- **`scoringrules`**
+- **`Scoringrules`**
 	- Open source Python package that provides verification metrics with backends for [`NumPy`](https://github.com/numpy/numpy), [`JAX`](https://github.com/jax-ml/jax), [`PyTorch`](https://github.com/pytorch/pytorch) and [`TensorFlow`](https://github.com/tensorflow/tensorflow).
 	- [Software Repository](https://github.com/frazane/scoringrules); [Documentation](https://frazane.github.io/scoringrules/)
 	- Reference: [Zanetta and Allen (2024)](https://frazane.github.io/scoringrules/#citation) 	

docs/release_notes.md

@@ -1,5 +1,60 @@
 # Release Notes (What's New)
 
+## Version 2.0.0 (December 7, 2024)
+
+For a list of all changes in this release, see the [full changelog](https://github.com/nci/scores/compare/1.3.0...2.0.0). Below are the changes we think users may wish to be aware of.
+
+### Breaking Changes
+
+- The function `scores.probability.tw_crps_for_ensemble` previously took an optional (mis-spelled) argument `chainging_func_kwargs`. The spelling has been corrected and the argument is now `chaining_func_kwargs`. See [PR #780](https://github.com/nci/scores/pull/780) and [PR #772](https://github.com/nci/scores/pull/772).
+- For those who develop on `scores`, you will need to update your installation of the `scores` package with `pip install -e .[all]`, to get updated versions of `black`, `pylint` and `mypy`. See [PR #768](https://github.com/nci/scores/pull/768), [PR #769](https://github.com/nci/scores/pull/769) and [PR #771](https://github.com/nci/scores/pull/771).
+
+### Features
+
+- Added three new metrics:
+	- Brier score for ensembles: `scores.probability.brier_score_for_ensemble`. See [PR #735](https://github.com/nci/scores/pull/735).
+	- Negative predictive value: `scores.categorical.BasicContingencyManager.negative_predictive_value`. See [PR #759](https://github.com/nci/scores/pull/759).
+	- Positive predictive value: `scores.categorical.BasicContingencyManager.positive_predictive_value`. See [PR #761](https://github.com/nci/scores/pull/761) and [PR #756](https://github.com/nci/scores/pull/756).
+- Also added one new emerging metric and two supporting functions:
+	- Risk matrix score: `scores.emerging.risk_matrix_scores`.
+	- Risk matrix score - matrix weights to array: `scores.emerging.matrix_weights_to_array`.
+	- Risk matrix score - warning scaling to weight array: `scores.emerging.weights_from_warning_scaling`.  
+	See [PR #724](https://github.com/nci/scores/pull/724) and [PR #794](https://github.com/nci/scores/pull/794).  
+- A new method called `format_table` was added to the class `BasicContingencyManager` to improve visualisation of 2x2 contingency tables. The tutorial `Binary_Contingency_Scores` was updated to demonstrate the use of this function. See [PR #775](https://github.com/nci/scores/pull/775).
+- The functions `scores.processing.comparative_discretise`, `scores.processing.binary_discretise` and `scores.processing.binary_discretise_proportion` now accept either a string indicating the choice of operator to be used, or an operator from the [Python core library `operator` module](https://docs.python.org/3/library/operator.html). Using one of the operators from the Python core module is recommended, as doing so is more reliable for a variety of reasons. Support for the use of a string may be removed in future. See [PR #740](https://github.com/nci/scores/pull/740) and [PR #758](https://github.com/nci/scores/pull/758).
+
+### Documentation
+
+- Added "The Risk Matrix Score" tutorial. See [PR #724](https://github.com/nci/scores/pull/724) and [PR #794](https://github.com/nci/scores/pull/794). 
+- Updated the "Brier Score" tutorial to include a new section about the Brier score for ensembles. See [PR #735](https://github.com/nci/scores/pull/735).
+- Updated the "Binary Categorical Scores and Binary Contingency Tables (Confusion Matrices)" 
+  tutorial:
+  - Included "positive predictive value" in the list of binary categorical scores.
+  - Included "negative predictive value" in the list of binary categorical scores.
+  - Demonstrated the use of the new `format_table` method for visualising 2x2 contingency tables.  
+  See [PR #759](https://github.com/nci/scores/pull/759) and [PR #775](https://github.com/nci/scores/pull/775). 
+- Updated the “Contributing Guide”:
+	- Added a new section: "Creating Your Own Fork of `scores` for the First Time".
+	- Updated the section: "Workflow for Submitting Pull Requests".
+	- Added a new section: "Pull Request Etiquette".  
+	See [PR #787](https://github.com/nci/scores/pull/787).
+- Updated the README:
+	- Added a link to a video of a PyCon AU 2024 conference presentation about `scores`. See [PR #783](https://github.com/nci/scores/pull/783).
+	- Added a link to the archives of `scores` on Zenodo. See [PR #784](https://github.com/nci/scores/pull/784).
+- Added `Scoringrules` to "Related Works". See [PR #746](https://github.com/nci/scores/pull/746), [PR #766](https://github.com/nci/scores/pull/766) and [PR #789](https://github.com/nci/scores/pull/789).
+
+### Internal Changes
+
+- Removed scikit-learn as a dependency. `scores` has replaced the use of scikit-learn with a similar function from SciPy (which was an existing `scores` dependency). This change was manually tested and found to be faster. See [PR #774](https://github.com/nci/scores/pull/774).
+- Version pinning of dependencies in release files (the wheel and sdist files used by PyPI and conda-forge) is now managed and set by the `hatch_build` script. This allows development versions to be free-floating, while being more specific about dependencies in releases. The previous process also aimed to do this, but was error-prone. A new entry called `pinned_dependencies` was added to pyproject.toml to specify the release dependencies. See [PR #760](https://github.com/nci/scores/pull/760).
+
+### Contributors to this Release
+
+Arshia Sharma* ([@arshiaar](https://github.com/arshiaar)), A.J. Fisher* ([@AJTheDataGuy](https://github.com/AJTheDataGuy)), Liam Bluett* ([@lbluett](https://github.com/lbluett)), Jinghan Fu* ([@JinghanFu](https://github.com/JinghanFu)), Sam Bishop* ([@techdragon](https://github.com/techdragon)), Robert J. Taggart ([@rob-taggart](https://github.com/rob-taggart)), Tennessee Leeuwenburg ([@tennlee](https://github.com/tennlee)), Stephanie Chong ([@Steph-Chong](https://github.com/Steph-Chong)) and 
+Nicholas Loveday ([@nicholasloveday](https://github.com/nicholasloveday)).    
+
+\* indicates that this release contains their first contribution to `scores`.
+
 ## Version 1.3.0 (November 15, 2024)
 
 For a list of all changes in this release, see the [full changelog](https://github.com/nci/scores/compare/1.2.0...1.3.0). Below are the changes we think users may wish to be aware of.

src/scores/__init__.py

@@ -13,7 +13,7 @@
 import scores.sample_data
 import scores.stats.statistical_tests  # noqa: F401
 
-__version__ = "2.0.0"
+__version__ = "2.1.0"
 
 __all__ = [
     "scores.categorical",

src/scores/categorical/contingency_impl.py

@@ -21,17 +21,25 @@
 # pylint: disable=too-many-lines
 
 import operator
+import warnings
 from abc import ABC, abstractmethod
 from typing import Optional
 
 import numpy as np
+import pandas as pd
 import xarray as xr
 
 import scores.utils
 from scores.typing import FlexibleArrayType, FlexibleDimensionTypes
 
 DEFAULT_PRECISION = 8
 
+HIGH_DIMENSION_HTML_CONTINGENCY_WARNING = """
+Note - you are trying to get a representation of a higher-dimensionality contingency table 
+(e.g. with preserved dimensions). This is currently not supported, so a simpler view is being
+returned. If you would like a 2x2 table rendering, try again with all dimensions reduced.
+"""
+
 
 class BasicContingencyManager:  # pylint: disable=too-many-public-methods
     """
@@ -153,6 +161,71 @@ def get_table(self) -> xr.DataArray:
         """
         return self.xr_table  # type: ignore  # mypy doesn't recognise when np has been overriden by xarray
 
+    def format_table(
+        self, positive_value_name: str = "Positive", negative_value_name: str = "Negative"
+    ) -> pd.DataFrame | xr.DataArray:
+        """Formats a contingency table from a 2x2 contingency manager into a confusion matrix.
+
+        Args:
+            positive_value_name: A string with the value to be displayed on
+                the contingency table visual. The default value is 'Positive'.
+            negative_value_name: A string with the value to be displayed on
+                the contingency table visual. The default value is 'Negative'.
+
+        Returns:
+            pandas.DataFrame: A Pandas DataFrame representing 2x2 binary contingency table if possible.
+            xr.DataArray: An xarray DataArray showing higher-dimension contingency table information otherwise
+
+        This function retrieves the contingency table, extracts the relevant counts, and constructs
+        a 2x2 confusion matrix represented as a Pandas DataFrame. This function does not support
+        HTML rendering of tables where dimensionality has been preserved.
+
+        Example:
+            Assuming `self.xr_table` returns an xr.Array:
+
+            .. code-block :: python
+
+                array([ 5., 11.,  1.,  1., 18.])
+
+                Coordinates:
+
+                    contingency
+                    (contingency)
+                    <U11
+                    'tp_count' ... 'total_count'
+
+            This function will return:
+
+            .. code-block :: python
+
+                    Positive  Negative Total
+            Positive       5       1       6
+            Negative       1      11       12
+            Total          6      12       18
+        """
+
+        table = self.xr_table
+
+        # Reshape into a 2x2 confusion matrix directly via numpy
+        if table.shape == (5,):
+            confusion_matrix = np.array([table[0], table[2], table[3], table[1]]).reshape((2, 2))
+        else:
+            warnings.warn(HIGH_DIMENSION_HTML_CONTINGENCY_WARNING)
+            return self.get_table()
+
+        # Create a pandas DataFrame for better presentation
+        df = pd.DataFrame(
+            confusion_matrix.astype(int),
+            columns=[positive_value_name + " Observed", negative_value_name + " Observed"],
+            index=[positive_value_name + " Forecast", negative_value_name + " Forecast"],
+        )
+
+        # Add totals
+        df["Total"] = df.sum(axis=1)
+        df.loc["Total"] = df.sum(axis=0)
+
+        return df
+
     def accuracy(self) -> xr.DataArray:
         """
         Identical to :py:func:`fraction_correct`.

src/scores/categorical/multicategorical_impl.py

@@ -3,7 +3,7 @@
 """
 
 from collections.abc import Sequence
-from typing import Optional, Union
+from typing import Iterable, Optional, Union
 
 import numpy as np
 import xarray as xr

src/scores/emerging/__init__.py

@@ -1,6 +1,4 @@
 """
-This module is currently empty.
-
 This module is intended to hold code for new metrics that may not yet have a publication
 and may not yet be in their final form, but which still have interest and value to
 the verification community.
@@ -9,3 +7,15 @@
 "Discussions" area of the github page (https://github.com/nci/scores/discussions) or
 raise a new issue (https://github.com/nci/scores/issues)
 """
+
+from scores.emerging.risk_matrix import (
+    matrix_weights_to_array,
+    risk_matrix_score,
+    weights_from_warning_scaling,
+)
+
+__all__ = [
+    "risk_matrix_score",
+    "matrix_weights_to_array",
+    "weights_from_warning_scaling",
+]