[TRACKER][BUG] Integer-based indexing causing failures when data grows very large. #2459
Labels
bug
Something isn't working
Comments
cjnolet
added
bug
|
Logistic regression seems to throw an error before the algorithm even executes and it seems to be from CuPy: a = np.random.random((1000000, 2100))
from cuml.linear_model import LogisticRegression
model = LogisticRegression()
model.fit(a, a[:,0]) |
|
I executed the Failed:
Did not fail:
|
cjnolet
changed the title
|
I believe the solution for most of these should be to convert any and all variables representing number of array elements (e.g., n_rows * n_cols) to |
|
This issue has been marked rotten due to no recent activity in the past 90d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. |
|
This issue has been marked stale due to no recent activity in the past 30d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be marked rotten if there is no activity in the next 60d. |
dantegd
changed the title
|
Testing these on 2108 w/ 1Mx2.1k (float32):
|
rapids-bot bot
pushed a commit
that referenced
this issue
Jul 23, 2021
Addresses #2459 (likely not all of it) Authors: - Corey J. Nolet (https://github.com/cjnolet) Approvers: - Dante Gama Dessavre (https://github.com/dantegd) URL: #3983
|
I really thought PCA/TSVD had been fixed in 21.08 but it does appear the issue still lingers: Definitely still seems to be an overflow somewhere. I'm guessing in a 32-bit int. |
|
Related: #4105 |
This was referenced Feb 4, 2022
rapids-bot bot
pushed a commit
that referenced
this issue
Feb 10, 2022
The following script results in an illegal memory access: ```python import cupy as cp from cuml.ensemble import RandomForestClassifier X = cp.random.random((1000000, 2500),dtype=cp.float32) assert(X.size > 2**31) y = cp.zeros(X.shape[0]) y[::2]=1.0 model = RandomForestClassifier() model.fit(X,y) pred = model.predict(X) ``` Fixed by some targeted casting of integers to 64 bit. See #2459. Authors: - Rory Mitchell (https://github.com/RAMitchell) Approvers: - Venkat (https://github.com/venkywonka) - Corey J. Nolet (https://github.com/cjnolet) URL: #4563
vimarsh6739
pushed a commit
to vimarsh6739/cuml
that referenced
this issue
Oct 9, 2023
…3983) Addresses rapidsai#2459 (likely not all of it) Authors: - Corey J. Nolet (https://github.com/cjnolet) Approvers: - Dante Gama Dessavre (https://github.com/dantegd) URL: rapidsai#3983
vimarsh6739
pushed a commit
to vimarsh6739/cuml
that referenced
this issue
Oct 9, 2023
The following script results in an illegal memory access: ```python import cupy as cp from cuml.ensemble import RandomForestClassifier X = cp.random.random((1000000, 2500),dtype=cp.float32) assert(X.size > 2**31) y = cp.zeros(X.shape[0]) y[::2]=1.0 model = RandomForestClassifier() model.fit(X,y) pred = model.predict(X) ``` Fixed by some targeted casting of integers to 64 bit. See rapidsai#2459. Authors: - Rory Mitchell (https://github.com/RAMitchell) Approvers: - Venkat (https://github.com/venkywonka) - Corey J. Nolet (https://github.com/cjnolet) URL: rapidsai#4563
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Below is a reproducible example on the 0.15 nightly of cuML w/ CUDA 10.2.
There are several estimators that fail with either
illegal memory access was encounteredorinvalid configuration argument, both of which seem to indicate an integer overflow might be occurring when representing the size of the underlying array.Here's the exception
In the above example, I used a size of
1Mx5000, which is >>2^31number of samples. I also tried with1Mx2500, which is also >2^31.Just to rule out the possibility that this error only occurs in the case of oversubscribing the GPU memory, I tried with
1Mx2100, (which is<2^31but still requires>32gbof GPU memory to train):As a result of the behavior outlined above, I have a very strong suspicion the are integers being used to represent array sizes that should be promoted to
size_t. This should also affectTruncatedSVD.Keeping a list to track the places where this occurs so far:
The text was updated successfully, but these errors were encountered: