Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] RandomForestClassifier No attribute predict_model_on_cpu when it should ? #5008

Closed
jeanmidevacc opened this issue Nov 17, 2022 · 2 comments · Fixed by #5148
Closed

[BUG] RandomForestClassifier No attribute predict_model_on_cpu when it should ? #5008

jeanmidevacc opened this issue Nov 17, 2022 · 2 comments · Fixed by #5148
Labels
? - Needs Triage Need team to review and classify bug Something isn't working Multi-GPU Issues & PRs related to multi-GPU functionality

Comments

@jeanmidevacc
Copy link

jeanmidevacc commented Nov 17, 2022 •
edited
Loading

Describe the bug
Hello Nvidia,

I am encountering an issue during my usage of from cuml.dask.ensemble import RandomForestClassifier when aftethe fit operation I am trying to leverage the attribute predict_model_on_cpu

On the instalaltion it seems to exists when I am calling the help command

image

But when I want to use it , the attribute doesn't exist

image

Steps/Code to reproduce bug
Locally having parquet files to read with dask cudf ,
setup data one column that can be seen as a target and other column are features in float32

Overall there is the code doing the data prep

df = dask_cudf.read_parquet(location_ml_dataset_local)
X_dask_cudf, y_dask_cudf = persist_across_workers(client,
                                                  [df[features].astype('float32'),
                                                   df["target"]])

Expected behavior
Manage to make prediction from a CPU or break for memory issue but not founding the function seems not right

Environment details (please complete the following information):

  • Environment location: AWS p3.8xlarge
  • Linux Distro/Architecture:
    starting from the AWS AMI ami-031863ae88fab9aeb
[ec2-user@ip-10-153-128-125 ec2]$  cat /etc/*-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
Amazon Linux release 2 (Karoo)
  • GPU Model/Driver: GPU NVIDIA Volta V100 and 450.142.00
  • CUDA: [11.2]
  • Method of cuDF & cuML install: conda
# packages in environment at /home/ec2-user/custom-miniconda/miniconda/envs/rapids_p38:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
alembic                   1.8.1                    pypi_0    pypi
anyio                     3.6.2              pyhd8ed1ab_0    conda-forge
arrow-cpp                 9.0.0           py38he270906_2_cpu    conda-forge
asttokens                 2.1.0                    pypi_0    pypi
aws-c-cal                 0.5.11               h95a6274_0    conda-forge
aws-c-common              0.6.2                h7f98852_0    conda-forge
aws-c-event-stream        0.2.7               h3541f99_13    conda-forge
aws-c-io                  0.10.5               hfb6a706_0    conda-forge
aws-checksums             0.1.11               ha31a3da_7    conda-forge
aws-sdk-cpp               1.8.186              hecaee15_4    conda-forge
backcall                  0.2.0                    pypi_0    pypi
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
backports.zoneinfo        0.2.1            py38h0a891b7_7    conda-forge
bokeh                     3.0.2              pyhd8ed1ab_0    conda-forge
boto3                     1.26.11                  pypi_0    pypi
botocore                  1.29.11                  pypi_0    pypi
brotlipy                  0.7.0           py38h0a891b7_1005    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.9.24            ha878542_0    conda-forge
cachetools                5.2.0              pyhd8ed1ab_0    conda-forge
certifi                   2022.9.24          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1           py38h4a40e3a_2    conda-forge
charset-normalizer        2.1.1                    pypi_0    pypi
click                     8.1.3           unix_pyhd8ed1ab_2    conda-forge
cloudpickle               2.2.0              pyhd8ed1ab_0    conda-forge
contourpy                 1.0.6            py38h43d8883_0    conda-forge
cryptography              38.0.3           py38h2b5fc30_0    conda-forge
cubinlinker               0.2.0            py38h7144610_1    rapidsai
cuda-cccl                 11.8.89                       0    nvidia
cuda-command-line-tools   11.8.0                        0    nvidia
cuda-compiler             11.8.0                        0    nvidia
cuda-cudart               11.8.89                       0    nvidia
cuda-cudart-dev           11.8.89                       0    nvidia
cuda-cuobjdump            11.8.86                       0    nvidia
cuda-cupti                11.8.87                       0    nvidia
cuda-cuxxfilt             11.8.86                       0    nvidia
cuda-documentation        11.8.86                       0    nvidia
cuda-driver-dev           11.8.89                       0    nvidia
cuda-gdb                  11.8.86                       0    nvidia
cuda-libraries            11.8.0                        0    nvidia
cuda-libraries-dev        11.8.0                        0    nvidia
cuda-memcheck             11.8.86                       0    nvidia
cuda-nsight               11.8.86                       0    nvidia
cuda-nsight-compute       11.8.0                        0    nvidia
cuda-nvcc                 11.8.89                       0    nvidia
cuda-nvdisasm             11.8.86                       0    nvidia
cuda-nvml-dev             11.8.86                       0    nvidia
cuda-nvprof               11.8.87                       0    nvidia
cuda-nvprune              11.8.86                       0    nvidia
cuda-nvrtc                11.8.89                       0    nvidia
cuda-nvrtc-dev            11.8.89                       0    nvidia
cuda-nvtx                 11.8.86                       0    nvidia
cuda-nvvp                 11.8.87                       0    nvidia
cuda-profiler-api         11.8.86                       0    nvidia
cuda-python               11.8.0           py38h3fd9d12_0    nvidia
cuda-sanitizer-api        11.8.86                       0    nvidia
cuda-toolkit              11.8.0                        0    nvidia
cuda-tools                11.8.0                        0    nvidia
cuda-visual-tools         11.8.0                        0    nvidia
cudatoolkit               11.2.72              h2bc3f7f_0    nvidia
cudf                      22.10.01        cuda_11_py38_gca9a422da9_2    rapidsai
cuml                      22.10.01        cuda11_py38_ge3f4f57d1_0    rapidsai
cupy                      11.3.0           py38h405e1b6_0    conda-forge
cycler                    0.11.0                   pypi_0    pypi
cython                    0.29.32                  pypi_0    pypi
cytoolz                   0.12.0           py38h0a891b7_1    conda-forge
dask                      2022.9.2           pyhd8ed1ab_0    conda-forge
dask-core                 2022.9.2           pyhd8ed1ab_0    conda-forge
dask-cuda                 22.10.00        py38_g382e519_0    rapidsai
dask-cudf                 22.10.01        cuda_11_py38_gca9a422da9_2    rapidsai
dask-glm                  0.2.0                      py_1    conda-forge
dask-ml                   2022.5.27          pyhd8ed1ab_0    conda-forge
dask-sql                  2022.10.1        py38h4029515_1    conda-forge
databricks-cli            0.17.3                   pypi_0    pypi
debugpy                   1.6.3                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
distributed               2022.9.2           pyhd8ed1ab_0    conda-forge
dlpack                    0.5                  h9c3ff4c_0    conda-forge
docker                    5.0.3                    pypi_0    pypi
entrypoints               0.4                      pypi_0    pypi
executing                 1.2.0                    pypi_0    pypi
faiss-proc                1.0.0                      cuda    rapidsai
fastapi                   0.87.0             pyhd8ed1ab_0    conda-forge
fastavro                  1.7.0            py38h0a891b7_0    conda-forge
fastrlock                 0.8              py38hfa26641_3    conda-forge
flask                     2.2.2                    pypi_0    pypi
fonttools                 4.38.0                   pypi_0    pypi
freetype                  2.12.1               hca18f0e_0    conda-forge
fsspec                    2022.11.0          pyhd8ed1ab_0    conda-forge
gds-tools                 1.4.0.31                      0    nvidia
gflags                    2.2.2             he1b5a44_1004    conda-forge
gitdb                     4.0.9                    pypi_0    pypi
gitpython                 3.1.29                   pypi_0    pypi
glog                      0.6.0                h6f12383_0    conda-forge
greenlet                  2.0.1                    pypi_0    pypi
grpc-cpp                  1.47.1               hbad87ad_6    conda-forge
gunicorn                  20.1.0                   pypi_0    pypi
h11                       0.14.0             pyhd8ed1ab_0    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        4.13.0                   pypi_0    pypi
importlib-resources       5.10.0                   pypi_0    pypi
ipykernel                 6.17.1                   pypi_0    pypi
ipython                   8.6.0                    pypi_0    pypi
itsdangerous              2.1.2                    pypi_0    pypi
jedi                      0.18.1                   pypi_0    pypi
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
jmespath                  1.0.1                    pypi_0    pypi
joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h166bdaf_2    conda-forge
jupyter-client            7.4.7                    pypi_0    pypi
jupyter-core              5.0.0                    pypi_0    pypi
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4                    pypi_0    pypi
krb5                      1.19.3               h3790be6_0    conda-forge
lcms2                     2.14                 h6ed2654_0    conda-forge
ld_impl_linux-64          2.38                 h1181459_1  
lerc                      4.0.0                h27087fc_0    conda-forge
libabseil                 20220623.0      cxx17_h48a1fff_5    conda-forge
libblas                   3.9.0           16_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_8    conda-forge
libbrotlidec              1.0.9                h166bdaf_8    conda-forge
libbrotlienc              1.0.9                h166bdaf_8    conda-forge
libcblas                  3.9.0           16_linux64_openblas    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcublas                 11.11.3.6                     0    nvidia
libcublas-dev             11.11.3.6                     0    nvidia
libcudf                   22.10.01        cuda11_gca9a422da9_2    rapidsai
libcufft                  10.9.0.58                     0    nvidia
libcufft-dev              10.9.0.58                     0    nvidia
libcufile                 1.4.0.31                      0    nvidia
libcufile-dev             1.4.0.31                      0    nvidia
libcuml                   22.10.01        cuda11_ge3f4f57d1_0    rapidsai
libcumlprims              22.10.00        cuda11_gfdb85e0_0    nvidia
libcurand                 10.3.0.86                     0    nvidia
libcurand-dev             10.3.0.86                     0    nvidia
libcurl                   7.86.0               h7bff187_1    conda-forge
libcusolver               11.4.1.48                     0    nvidia
libcusolver-dev           11.4.1.48                     0    nvidia
libcusparse               11.7.5.86                     0    nvidia
libcusparse-dev           11.7.5.86                     0    nvidia
libdeflate                1.14                 h166bdaf_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               h9b69904_4    conda-forge
libfaiss                  1.7.0           cuda112h5bea7ad_8_cuda    conda-forge
libffi                    3.4.2                h295c915_4  
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgfortran-ng            12.2.0              h69a702a_19    conda-forge
libgfortran5              12.2.0              h337968e_19    conda-forge
libgoogle-cloud           2.1.0                h9ebe8e8_2    conda-forge
liblapack                 3.9.0           16_linux64_openblas    conda-forge
libllvm11                 11.1.0               he0ac6c6_5    conda-forge
libnghttp2                1.47.0               hdcd2b5c_1    conda-forge
libnpp                    11.8.0.86                     0    nvidia
libnpp-dev                11.8.0.86                     0    nvidia
libnvjpeg                 11.9.0.86                     0    nvidia
libnvjpeg-dev             11.9.0.86                     0    nvidia
libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
libpng                    1.6.38               h753d276_0    conda-forge
libprotobuf               3.20.2               h6239696_0    conda-forge
libraft-distance          22.10.01        cuda11_gf7d2335_0    rapidsai
libraft-headers           22.10.01        cuda11_gf7d2335_0    rapidsai
libraft-nn                22.10.01        cuda11_gf7d2335_0    rapidsai
librmm                    22.10.01        cuda11_gd98b8719_0    rapidsai
libssh2                   1.10.0               haa6b8db_3    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libthrift                 0.16.0               h491838f_2    conda-forge
libtiff                   4.4.0                h55922b4_4    conda-forge
libutf8proc               2.8.0                h166bdaf_0    conda-forge
libwebp-base              1.2.4                h166bdaf_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxgboost                1.6.2dev.rapidsai22.10       cuda_11_1    rapidsai
libzlib                   1.2.13               h166bdaf_4    conda-forge
llvm-openmp               15.0.5               he0ac6c6_0    conda-forge
llvmlite                  0.39.1           py38h38d86a4_1    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lz4                       4.0.2            py38h1bf946c_0    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
mako                      1.2.4                    pypi_0    pypi
markupsafe                2.1.1            py38h0a891b7_2    conda-forge
matplotlib                3.6.2                    pypi_0    pypi
matplotlib-inline         0.1.6                    pypi_0    pypi
mlflow                    1.28.0                   pypi_0    pypi
msgpack-python            1.0.4            py38h43d8883_1    conda-forge
multipledispatch          0.6.0                      py_0    conda-forge
nccl                      2.14.3.1             h0800d71_0    conda-forge
ncurses                   6.3                  h5eee18b_3  
nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
nsight-compute            2022.3.0.22                   0    nvidia
numba                     0.56.3           py38h9a4aae9_0    conda-forge
numpy                     1.23.4           py38h7042d01_1    conda-forge
nvtx                      0.2.3            py38h0a891b7_2    conda-forge
oauthlib                  3.2.2                    pypi_0    pypi
openjpeg                  2.5.0                h7d73246_1    conda-forge
openssl                   1.1.1s               h166bdaf_0    conda-forge
orc                       1.7.6                h6c59b99_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.5.1            py38h8f669ce_1    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
parso                     0.8.3                    pypi_0    pypi
partd                     1.3.0              pyhd8ed1ab_0    conda-forge
pexpect                   4.8.0                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pillow                    9.2.0            py38h9eb91d8_3    conda-forge
pip                       22.2.2           py38h06a4308_0  
platformdirs              2.5.4                    pypi_0    pypi
prometheus-client         0.15.0                   pypi_0    pypi
prometheus-flask-exporter 0.21.0                   pypi_0    pypi
prompt-toolkit            3.0.32             pyha770c72_0    conda-forge
protobuf                  3.20.2           py38hfa26641_0    conda-forge
psutil                    5.9.4            py38h0a891b7_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptxcompiler               0.7.0            py38h7525318_2    conda-forge
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
py-xgboost                1.6.2dev.rapidsai22.10  cuda_11_py38_1    rapidsai
pyarrow                   9.0.0           py38h097c49a_2_cpu    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydantic                  1.10.2           py38h0a891b7_1    conda-forge
pygments                  2.13.0             pyhd8ed1ab_0    conda-forge
pyjwt                     2.6.0                    pypi_0    pypi
pylibraft                 22.10.01        cuda11_py38_gf7d2335_0    rapidsai
pynvml                    11.4.1             pyhd8ed1ab_0    conda-forge
pyopenssl                 22.1.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.8.15               h3fd9d12_0  
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-tzdata             2022.6             pyhd8ed1ab_0    conda-forge
python_abi                3.8                      2_cp38    conda-forge
pytz                      2022.6             pyhd8ed1ab_0    conda-forge
pytz-deprecation-shim     0.1.0.post0      py38h578d9bd_3    conda-forge
pyyaml                    6.0              py38h0a891b7_5    conda-forge
pyzmq                     24.0.1                   pypi_0    pypi
querystring-parser        1.2.4                    pypi_0    pypi
raft-dask                 22.10.01        cuda11_py38_gf7d2335_0    rapidsai
re2                       2022.06.01           h27087fc_0    conda-forge
readline                  8.2                  h5eee18b_0  
requests                  2.28.1                   pypi_0    pypi
rmm                       22.10.01        cuda11_py38_gd98b8719_0    rapidsai
s2n                       1.0.10               h9b69904_0    conda-forge
s3transfer                0.6.0                    pypi_0    pypi
scikit-learn              1.1.3            py38h4c4ba11_1    conda-forge
scipy                     1.9.3            py38h8ce737c_2    conda-forge
setuptools                65.5.0           py38h06a4308_0  
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     5.0.0                    pypi_0    pypi
snappy                    1.1.9                hbd366e4_2    conda-forge
sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
spdlog                    1.8.5                h4bd325d_1    conda-forge
sqlalchemy                1.4.44                   pypi_0    pypi
sqlite                    3.39.3               h5082296_0  
sqlparse                  0.4.3                    pypi_0    pypi
stack-data                0.6.1                    pypi_0    pypi
starlette                 0.21.0             pyhd8ed1ab_0    conda-forge
tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tk                        8.6.12               h1ccaba5_0  
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tornado                   6.2                      pypi_0    pypi
tqdm                      4.64.1                   pypi_0    pypi
traitlets                 5.5.0                    pypi_0    pypi
treelite                  3.0.0            py38h8e2129e_1    conda-forge
treelite-runtime          3.0.0                    pypi_0    pypi
typing-extensions         4.4.0                hd8ed1ab_0    conda-forge
typing_extensions         4.4.0              pyha770c72_0    conda-forge
tzdata                    2022f                h191b570_0    conda-forge
tzlocal                   4.2              py38h578d9bd_2    conda-forge
ucx                       1.13.1               h538f049_0    conda-forge
ucx-proc                  1.0.0                       gpu    rapidsai
ucx-py                    0.28.00         py38_g8292636_0    rapidsai
urllib3                   1.26.11            pyhd8ed1ab_0    conda-forge
uvicorn                   0.19.0           py38h578d9bd_0    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
websocket-client          1.4.2                    pypi_0    pypi
werkzeug                  2.2.2                    pypi_0    pypi
wheel                     0.37.1             pyhd3eb1b0_0  
xgboost                   1.6.2dev.rapidsai22.10  cuda_11_py38_1    rapidsai
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xyzservices               2022.9.0           pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h5eee18b_0  
yaml                      0.2.5                h7f98852_2    conda-forge
zict                      2.2.0              pyhd8ed1ab_0    conda-forge
zipp                      3.10.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               h166bdaf_4    conda-forge
zstd                      1.5.2                h6239696_4    conda-forge

Additional context
@jeanmidevacc jeanmidevacc added ? - Needs Triage Need team to review and classify bug Something isn't working labels Nov 17, 2022
@dantegd dantegd added the Multi-GPU Issues & PRs related to multi-GPU functionality label Dec 4, 2022
@dantegd
Copy link
Member

dantegd commented Dec 4, 2022

Thanks for the issue @jeanmidevacc, indeed this is a bug in the multiGPU RFClassifer code. We will have a fix soon and it should be available in the nightly builds as well.

@jeanmidevacc
Copy link
Author

Hello @dantegd , thanks for the update :)

@lowener lowener mentioned this issue Jan 23, 2023
Merged
@rapids-bot rapids-bot bot closed this as completed in #5148 Feb 5, 2023
rapids-bot bot pushed a commit that referenced this issue Feb 5, 2023
@lowener
Add _predict_model_on_cpu for RandomForestClassifier (#5148)
8f15cc1 
Closes #5008

Authors:
  - Micka (https://github.com/lowener)
  - Dante Gama Dessavre (https://github.com/dantegd)

Approvers:
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #5148
jakirkham pushed a commit to jakirkham/cuml that referenced this issue Feb 27, 2023
@lowener
Add _predict_model_on_cpu for RandomForestClassifier (rapidsai#5148)
fac0e24 
Closes rapidsai#5008

Authors:
  - Micka (https://github.com/lowener)
  - Dante Gama Dessavre (https://github.com/dantegd)

Approvers:
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: rapidsai#5148
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
? - Needs Triage Need team to review and classify bug Something isn't working Multi-GPU Issues & PRs related to multi-GPU functionality
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add _predict_model_on_cpu for RandomForestClassifier lowener/cuml
2 participants
@dantegd @jeanmidevacc