[BUG] Logistic regression does not return fit status

[tfeher]

Describe the bug
Logistic regression uses the QN solver. The QN solver defines a set of return codes

cuml/cpp/src/glm/qn/qn_util.cuh

Lines 40 to 46 in 39e1bb2

	enum OPT_RETCODE {
	OPT_SUCCESS = 0,
	OPT_NUMERIC_ERROR = 1,
	OPT_LS_FAILED = 2,
	OPT_MAX_ITERS_REACHED = 3,
	OPT_INVALID_ARGS = 4
	};

While qn_fit returns these codes to the caller, qnFit ignores the returned error code. This way the Python layer is not informed of the fit status.

If the solver exits after the first iteration with numerical error, the Python user is not informed about the error, and only sees insufficent accuracy of the model, like in this issue.

PR #2543 improved the C++ side logging to print a warning message if the solver is not converged, and error message if numerical error is found. But this might be hidden for user who is running a Jupyter notebook.

Steps/Code to reproduce bug

from sklearn.datasets import load_breast_cancer
from cuml.linear_model import LogisticRegression

X, y = load_breast_cancer(return_X_y=True)

cu_clf = LogisticRegression(max_iter=10)
cu_clf.fit(X, y.astype(float))

The C++ layer prints a warning message on the standard output: L-BFGS: max iterations reached, but no such message visible in a jupyter notebook.

In contrast Scikit learn

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression as sk_log_reg

X, y = load_breast_cancer(return_X_y=True)

sk_clf = sk_log_reg(max_iter=10)
sk_clf.fit(X, y)

gives the following output in a jupyter notebook:

/home/tfeher/gpfs/anaconda3/envs/cuml_branch15a/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

Expected behavior
Give informative message about fit status for Jupyter notebook users.

[teju85]

What if we added a _fit_status member variable inside LogisticRegression class? That can be used to programmatically know the result of the training.

[github-actions]

This issue has been marked rotten due to no recent activity in the past 90d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

[github-actions]

This issue has been marked stale due to no recent activity in the past 30d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be marked rotten if there is no activity in the next 60d.

[tfeher]

The warning message is correctly displayed in the Jupyter notebook:

[W] [14:42:45.849061] L-BFGS: max iterations reached

This basically solves the main inssue here. We might want to add a few notes what to do in case the max iterations are reached, along the line what sklearn does

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html

I am preparing a small PR to implement this.

Since sklearn does not return a fit status, I think we can also skip it.