BUG REPORT: The custom eval metric get untransformed model input score.

[MichaelYin1994]

On XGBoost python library (version 1.4.2, also in 1.4.0), custom eval metric get an untransformed model score, instead of the validation prediction probability.

Code to reproduce:

import numpy as np
import xgboost as xgb
from sklearn.metrics import accuracy_score

def custom_eval_metric(y_pred_proba, dtrain):
    y_true_label = dtrain.get_label()
    print(y_pred_proba.min(), y_pred_proba.max())

    acc_score = accuracy_score(
        y_true_label.reshape(-1, 1),
        (y_pred_proba > 0.5).astype(int).reshape(-1, 1))

    return 'custom_acc', -1 * acc_score


if __name__ == '__main__':
    # Generating random training data
    X_train = np.random.random(500)
    X_train = X_train.reshape((100, 5))
    y_train = (np.random.randint(0, 100, 100) > 50).astype(int)

    X_val = np.random.random(500)
    X_val = X_val.reshape((100, 5))
    y_val = (np.random.randint(0, 100, 100) > 50).astype(int)

    # training XGBoost classifier
    xgb_params = {
        "n_estimators": 20,
        "max_depth": 5,
        "learning_rate": 0.03,
        "verbosity": 0,
        "objective": "binary:logistic",
        "booster": "gbtree",
        "colsample_bytree": 0.98,
        "colsample_bylevel": 0.97,
        "subsample": 0.98,
        "disable_default_eval_metric": 1,
        "random_state": 2048}

    # Fit a xgboost model, you can see at each iteration, the eval metric
    # get the untransformed model output score
    model = xgb.XGBClassifier(**xgb_params)
    model.fit(
        X_train, y_train,
        eval_set=[(X_val, y_val)],
        early_stopping_rounds=15,
        verbose=True,
        eval_metric=custom_eval_metric)

Output:

-0.035172414 0.027272727
[0]	validation_0-custom_acc:-0.50000
-0.07448528 0.04746426
[1]	validation_0-custom_acc:-0.50000
-0.10820254 0.067466184
[2]	validation_0-custom_acc:-0.50000
-0.1436399 0.08694458
[3]	validation_0-custom_acc:-0.50000
-0.18393388 0.09330373
[4]	validation_0-custom_acc:-0.50000
-0.22471038 0.11267628
[5]	validation_0-custom_acc:-0.50000
-0.26337633 0.13032766
[6]	validation_0-custom_acc:-0.50000
-0.2959541 0.1534172
[7]	validation_0-custom_acc:-0.50000
-0.2985757 0.17225054
[8]	validation_0-custom_acc:-0.50000
-0.33194867 0.18320686
[9]	validation_0-custom_acc:-0.50000
-0.36432728 0.20864208
[10]	validation_0-custom_acc:-0.50000
-0.39650607 0.20381181
[11]	validation_0-custom_acc:-0.50000
-0.4287486 0.2129844
[12]	validation_0-custom_acc:-0.50000
-0.41276354 0.23032054
[13]	validation_0-custom_acc:-0.50000
-0.43424797 0.2607328
[14]	validation_0-custom_acc:-0.50000
-0.465707 0.28491184

Expected behaviour:

You can see negative score in each iteration. The expected output should range between 0 and 1.

Bug analysis:

See line 1643 in core.py, XGBoost version 1.4.2. This problem is caused by an unsuitable predict method parameter(output_margin=True).

[hcho3]

This is actually an expected behavior. Quoted from the release note for 1.2.0:

Breaking: Custom evaluation metric now receives raw prediction (#5954)
Previously, the custom evaluation metric received a transformed prediction result when used with a classifier. Now the custom metric will receive a raw (untransformed) prediction and will need to transform the prediction itself. See demo/guide-python/custom_softmax.py for an example.
This change is to make the custom metric behave consistently with the custom objective, which already receives raw prediction (#5564).

[MichaelYin1994]

The question remains: How can we transform the raw score of the prediction that custom_eval_metric received to the prediction probability in the binary classification problem?

I have read the API codes and searched for answers for a half of a day, none of them gave an answer.

Thanks for your relying.

[hcho3]

@MichaelYin1994 Take a look at https://github.com/dmlc/xgboost/blob/master/demo/guide-python/custom_objective.py. In particular:

preds = 1.0 / (1.0 + np.exp(-preds))  # transform raw leaf weight