Cyeva is a Python open-source tool library jointly developed by the Colorful Clouds Technology weather team and community contributors for quickly evaluating the accuracy of deterministic forecasts of meteorological elements.
Cyeva aims to make automated evaluation of the accuracy of deterministic meteorological forecasts straightforward. It integrates commonly used deterministic forecast accuracy evaluation metrics. Its internal algorithms widely use NumPy's vector operations, thus having high computational efficiency for large data sets.
$ pip install cyevaNote: Since this project is currently in beta, it is not a stable version. It may undergo incompatible changes in future releases. Therefore, it's recommended to specify the version during installation, e.g., pip install cyeva==0.2.3.
First, choose the desired version on the release page, extract it, navigate to the project directory, and then execute:
$ python setup.py installCyeva has specially designed objects to handle relevant indicators for temperature, wind, and precipitation.
For temperature, which is a continuous variable, we are usually interested in its RMSE (Root Mean Square Error), MAE (Mean Absolute Error), and other similar metrics. In cyeva, we can refer to the following examples to calculate these kinds of metrics.
import numpy as np
from cyeva import TemperatureComparison
np.random.seed(0) # Specify a random seed to ensure that the results obtained are consistent.
obs = np.sin(np.arange(100)) * 20 + np.random.random(100) * 5 * np.random.choice([1, -1]) # Simulate real temperature by overlaying a sine array with a random array.
fcst = obs + np.random.random(100) * 5 * np.random.choice([1, -1]) # Limit the forecast to within plus or minus 5°C of the observation; the results are better in such cases.
temp = TemperatureComparison(obs, fcst, unit='degC')
print('accuracy ration within 1 degC:', temp.calc_diff_accuracy_ratio(limit=1)) # 1-degree accuracy (deviation within 1°C)
print('accuracy ration within 2 degC:', temp.calc_diff_accuracy_ratio(limit=2)) # 2-degree accuracy (deviation within 2°C)
print('rss:', temp.calc_rss()) # Residual Sum of Squares
print('rmse:', temp.calc_rmse()) # Root Mean Square Error
print('mae:', temp.calc_mae()) # Mean Absolute Error
print('chi square:', temp.calc_chi_square()) # Chi Square (χ2)One characteristic of precipitation is that it is not continuous, so when assessing its accuracy, in addition to common metrics, other metrics (such as TS scores) are usually introduced. Moreover, precipitation has clear levels, so it is necessary to make distinctions according to different levels. In cyeva, we can refer to the following examples to calculate metrics used for assessing precipitation.
import numpy as np
from cyeva import PrecipitationComparison
np.random.seed(0) # Specify a random seed to ensure that the results obtained are consistent.
obs = np.random.random(int(100)) * 50
fcst = np.random.random(int(100)) * 50
precip = PrecipitationComparison(obs, fcst, unit='mm')
print('rss:', precip.calc_rss()) # Residual Sum of Squares
print('rmse:', precip.calc_rmse()) # Root Mean Square Error
print('mae:', precip.calc_mae()) # Mean Absolute Error
print('chi square:', precip.calc_chi_square()) # Chi Square (χ2)
print('accuracy ratio:', precip.calc_accuracy_ratio()) # Accuracy(0 level)
print('binary accuracy ratio:', precip.calc_binary_accuracy_ratio()) # Binary accuracy(Binary classes between sunny or rainy weather)
print('false alarm ratio:', precip.calc_false_alarm_ratio()) # False alarm
print('miss ratio:', precip.calc_miss_ratio()) # Missing
print('accuracy ratio:', precip.calc_accuracy_ratio(kind='3h', lev='3')) # Accuracy (3-hours time interval/level 3/heavy rain)
for inv in ['1h', '3h', '12h', '24h']: # Accuracy on different time intervals
for lev in range(7):
lev_str = str(lev)
levp_str = f'+{lev_str}'
print(f'accuracy ratio({inv}|{lev_str}):', precip.calc_accuracy_ratio(kind=inv, lev=lev_str))
if lev > 0:
print(f'accuracy ratio({inv}|{levp_str}):', precip.calc_accuracy_ratio(kind=inv, lev=levp_str))
print('ts:', precip.calc_ts()) # TS score（1-hr time interval/binary classes/default）
for inv in ['1h', '3h', '12h', '24h']: # TS score under different time intervals with different level criteria
for lev in range(7):
lev_str = str(lev)
levp_str = f'+{lev_str}'
print(f'ts({inv}|{lev_str}):', precip.calc_ts(kind=inv, lev=lev_str))
if lev > 0:
print(f'ts({inv}|{levp_str}):', precip.calc_ts(kind=inv, lev=levp_str))
print('ets:', precip.calc_ets()) # ETS score（1-hr time interval/binary classes/ETS score/default）
for inv in ['1h', '3h', '12h', '24h']: # ETS score under different time intervals with different level criteria
for lev in range(7):
lev_str = str(lev)
levp_str = f'+{lev_str}'
print(f'ets({inv}|{lev_str}):', precip.calc_ets(kind=inv, lev=lev_str))
if lev > 0:
print(f'ets({inv}|{levp_str}):', precip.calc_ets(kind=inv, lev=levp_str))
print('bias:', precip.calc_bias_score()) # bias score（1-hr time interval/binary classes/bias score/default）
for inv in ['1h', '3h', '12h', '24h']: # bias score under different time intervals with different level criteria
for lev in range(7):
lev_str = str(lev)
levp_str = f'+{lev_str}'
print(f'bias({inv}|{lev_str}):', precip.calc_bias_score(kind=inv, lev=lev_str))
if lev > 0:
print(f'bias({inv}|{levp_str}):', precip.calc_bias_score(kind=inv, lev=levp_str))For wind, which is a vector element, we need to provide both speed and direction information. Therefore, the data array passed in when instantiating the object will differ from that of temperature and precipitation. There are also some metrics specifically designed for wind assessment, such as the rates of wind being stronger or weaker than the standard, etc. In cyeva, we can refer to the following examples to calculate metrics used for assessing wind.
import numpy as np
from cyeva import WindComparison
np.random.seed(0)
obs_spd = np.random.random(100) * 10
obs_dir = np.random.random(100) * 360
fct_spd = np.random.random(100) * 10
fct_dir = np.random.random(100) * 360
wind = WindComparison(obs_spd, fct_spd, obs_dir, fct_dir)
print('difference accuracy ratio within 1 m/s:', wind.calc_diff_accuracy_ratio(limit=1)) # 1m/s accuracy (speed deviation within 1m/s)
print('difference accuracy ratio within 2 m/s:', wind.calc_diff_accuracy_ratio(limit=2)) # 2m/s accuracy (speed deviation within 2m/s)
print('wind speed rss:', wind.calc_rss()) # Residual Sum of Squares（by speed/default）
print('wind direction rss:', wind.calc_rss(kind='direction')) # Residual Sum of Squares（by direction）
print('wind speed rmse:', wind.calc_rmse()) # Root Mean Square Error（by speed/default）
print('wind direction rmse:', wind.calc_rmse(kind='direction')) # Root Mean Square Error（by direction）
print('wind speed mae:', wind.calc_mae()) # Mean Absolute Error（by speed/default）
print('wind direction mae:', wind.calc_mae(kind='direction')) # Mean Absolute Error（by direction）
print('wind speed chi square:', wind.calc_chi_square()) # Chi Square(χ2)
print('wind direction chi square:', wind.calc_chi_square(kind='direction')) # Chi Square(χ2)（by direction）
print('wind direction score:', wind.calc_dir_score()) # Direction score
print('wind speed score:', wind.calc_speed_score()) # Speed score
print('wind scale accuracy ratio:', wind.calc_wind_scale_accuracy_ratio()) # Wind scale accuracy
print('wind speed accuracy ratio within 2m/s:', wind.calc_speed_accuracy_ratio()) # Speed accuracy(2m/s deviation/default)
print('wind speed accuracy radio within 3m/s:', wind.calc_speed_accuracy_ratio(limit=3)) # Speed accuracy(3m/s deviation)
print('wind scale stronger ratio:', wind.calc_wind_scale_stronger_ratio()) # The ratio with a stronger wind scale forecast
print('wind scale weaker ratio:', wind.calc_wind_scale_weaker_ratio()) # The ratio with a weaker wind scale forecastFor explanations, formulas, and other information about the various evaluation algorithms implemented in this project, please refer to the metrics section in the cyeva documentation.