[BUG] Linear Regression: Estimator predictions deviate from scikit-learn results for some inputs

[csadorf]

Describe the bug

The linear regression predictions between the cuml and the scikit-learn implementation deviate for certain input values.

Steps/Code to reproduce bug

1. Download lg_cuml_sk_deviation_minimal_example_input.zip
2. Execute:

import numpy as np
from sklearn.linear_model import LinearRegression
from cuml import LinearRegression as cuLinearRegression

npzfile = np.load("lg_cuml_sk_deviation_minimal_example_input.zip")
X_train, X_test, y_train, y_test = npzfile = [npzfile[f] for f in npzfile.files]

estimator = LinearRegression()
estimator.fit(X_train, y_train)
sk_predict = estimator.predict(X_test)

estimator = cuLinearRegression()
estimator.fit(X_train, y_train)
cu_predict = estimator.predict(X_test)

absolute_deviation = np.sum(np.abs((cu_predict - sk_predict)))
print(absolute_deviation)

Expected behavior

The deviation between regression predictions should be below the expected tolerance.

Environment details (please complete the following information):

• Environment location: Docker
• Linux Distro/Architecture: Ubuntu 20.04 amd64
• GPU Model/Driver: V100-SMX2 495.29.05
• CUDA: 11.5
• Method of cuDF & cuML install: from source

Additional context

The issue was discovered as part of the extension of tests with hypothesis (#4952).