Skip to content

Commit

Permalink
Update and add admonitions for emerging metrics (#794)
Browse files Browse the repository at this point in the history
* Update admonitions for emerging metrics in API, tutorial and README, plus add an admonition to included page in docs
  • Loading branch information
Steph-Chong authored Dec 5, 2024
1 parent 33df085 commit 2e60196
Show file tree
Hide file tree
Showing 4 changed files with 18 additions and 11 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Below is a **curated selection** of the metrics, tools and statistical tests inc
| **[Spatial](https://scores.readthedocs.io/en/stable/included.html#spatial)** |Scores that take into account spatial structure. |Fractions Skill Score. |
| **[Statistical Tests](https://scores.readthedocs.io/en/stable/included.html#statistical-tests)** |Tools to conduct statistical tests and generate confidence intervals. |Diebold Mariano. |
| **[Processing Tools](https://scores.readthedocs.io/en/stable/included.html#processing-tools-for-preparing-data)** |Tools to pre-process data. |Data matching, Discretisation, Cumulative Density Function Manipulation. |
| **[Emerging](https://scores.readthedocs.io/en/latest/included.html#emerging)** |Emerging scores that are under development and have not yet been peer-reviewed. They may change at any time. |Risk Matrix Score. |
| **[Emerging](https://scores.readthedocs.io/en/latest/included.html#emerging)** |Emerging scores that are still undergoing mathematical peer review. They may change in line with the peer review process. |Risk Matrix Score. |

`scores` not only includes common scores (e.g., MAE, RMSE), it also includes novel scores not commonly found elsewhere (e.g., FIRM, Flip-Flop Index), complex scores (e.g., threshold weighted CRPS), and statistical tests (e.g., the Diebold Mariano test). Additionally, it provides pre-processing tools for preparing data for scores in a variety of formats including cumulative distribution functions (CDF). `scores` provides its own implementations where relevant to avoid extensive dependencies.

Expand Down
4 changes: 4 additions & 0 deletions docs/included.md
Original file line number Diff line number Diff line change
Expand Up @@ -741,6 +741,10 @@

## Emerging

```{Caution}
This section of the API contains implementations of novel metrics that are still undergoing mathematical peer review. These implementations may change in line with the peer review process.
```

```{list-table}
:header-rows: 1
Expand Down
21 changes: 12 additions & 9 deletions src/scores/emerging/risk_matrix.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ def risk_matrix_score(
) -> XarrayLike:
"""
Caution:
This is an emerging score that is under development and has not yet been peer-reviewed.
It may change at any time.
This is an implementation of a novel metric that is still undergoing mathematical peer review.
This implementation may change in line with the peer review process.
Calculates the risk matrix score of Taggart & Wilke (2024).
Expand Down Expand Up @@ -104,7 +104,8 @@ def risk_matrix_score(
ValueError: if ``threshold_assignment`` is not "upper" or lower".
References:
- Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework with consistent evaluation. In preparation.
- Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework
with consistent evaluation. In preparation.
See also:
:py:func:`scores.emerging.matrix_weights_to_array`
Expand Down Expand Up @@ -266,8 +267,8 @@ def matrix_weights_to_array(
) -> xr.DataArray:
"""
Caution:
This function is part of an emerging score that is under development and has not yet
been peer-reviewed. It may change at any time.
This function is part of an implementation of a novel metric that is still undergoing
mathematical peer review. This implementation may change in line with the peer review process.
Generates a 2-dimensional xr.DataArray of weights for each decision point, which is used for
the :py:func:`scores.emerging.risk_matrix_score` calculation.
Expand Down Expand Up @@ -298,7 +299,8 @@ def matrix_weights_to_array(
ValueError: if ``prob_threshold_coords`` aren't strictly between 0 and 1.
References:
- Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework with consistent evaluation. In preparation.
- Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework
with consistent evaluation. In preparation.
Examples:
Returns weights for each risk matrix decision point, where weights increase with increasing
Expand Down Expand Up @@ -351,8 +353,8 @@ def weights_from_warning_scaling(
) -> xr.DataArray:
"""
Caution:
This function is part of an emerging score that is under development and has not yet
been peer-reviewed. It may change at any time.
This function is part of an implementation of a novel metric that is still undergoing
mathematical peer review. This implementation may change in line with the peer review process.
Given a warning scaling matrix, assessment weights and other inputs,
returns the weights for each risk matrix decision point as an xarray data array. The returned
Expand Down Expand Up @@ -395,7 +397,8 @@ def weights_from_warning_scaling(
ValueError: if ``assessment_weights`` aren't strictly positive.
References:
- Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework with consistent evaluation. In preparation.
- Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework
with consistent evaluation. In preparation.
Examples:
Returns weights for each risk matrix decision point, for the SHORT-RANGE scaling matrix of
Expand Down
2 changes: 1 addition & 1 deletion tutorials/Risk_Matrix_Score.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
"\n",
"\n",
"> **CAUTION:**\n",
"> This is an emerging score that is under development and has not yet been peer-reviewed. It may change at any time.\n",
"> The risk matrix score is a novel metric that is still undergoing mathematical peer review. The implementation may change in line with the peer review process.\n",
"\n",
"The Risk Matrix Score provides a consistent way of scoring forecasts and warnings within a risk matrix framework (Taggart & Wilke 2024). A lower score is better.\n",
"\n",
Expand Down

0 comments on commit 2e60196

Please sign in to comment.