[BUG] RF : regression in GPU predict accuracy compared to CPU predict and scikit-learn predict

[venkywonka]

Describe the bug
After #3869 , some cuml predictions on GPU (FIL) seems to be less accurate than CPU predictions and tends to break tolerance (when set to 2%) of some of the tests in test_random_forest.py.

printing the cuml_acc and fil_acc respectively for each build version after running the above test at this line:

• before Enable probability output from RF binary classifier (alternative implementaton) #3869 (at commit 474e2e7 )
  • CPU prediction accuracy: 0.9285714285714286, GPU prediction accuracy: 0.9285714285714286
• after Enable probability output from RF binary classifier (alternative implementaton) #3869 (at commit 92484fb)
  • CPU prediction accuracy: 0.9285714285714286, GPU prediction accuracy: 0.8857142857142857

Steps/Code to reproduce bug

• pytest cuml/python/cuml/test/test_random_forest.py --run_quality --run_stress --run_unit -v

• or this particular test:
  pytest $CUML_PATH/python/cuml/test/test_random_forest.py::test_rf_classification[small_clf0-auto-1-float32-0.9] -v --run_quality -- run_stress --run_unit

Expected behavior
tests expected to pass

Environment details (please complete the following information):

• Environment location: [Bare-metal]
• Linux Distro/Architecture: [Ubuntu 18.04 amd64]
• GPU Model/Driver: [V100 and driver 460.35]
• CUDA: [11.0]
• Method of cuDF & cuML install: [from source]
  • If method of install is [from source], provide versions of cmake & gcc/g++ and commit hash of build:
    • cmake: 3.20.1, gcc/g++: 9.3.0, commit hash: 92484fb99aa390dc24aeb86a67abf259bfaadf55