Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fallback memory resource for TCC devices #257

Merged
merged 14 commits into from
Dec 6, 2024

Conversation

ksimpson-work
Copy link
Contributor

@ksimpson-work ksimpson-work commented Nov 28, 2024

For devices which don't support memory pools, we need to provide an alternate default memory resource.

This basic WAR implementation works. I used a colossus lease for a Tesla T4 on Friday Nov 29 and these were the results:

using the DefaultAsyncMempool --> python - m pytest tests/test_memory.py
=============================================== short test summary info ===============================================
FAILED tests/test_memory.py::test_buffer_initialization - cuda.core.experimental._utils.CUDAError: CUDA_ERROR_NOT_SUPPORTED: operation not supported

using the implementation in this branch --> python - m pytest tests/test_memory.py
collected 4 items

tests\test_memory.py .... (SUCCESS)

close #208

@ksimpson-work ksimpson-work added P0 High priority - Must do! feature New feature or request cuda.core Everything related to the cuda.core module labels Nov 28, 2024
@ksimpson-work ksimpson-work added this to the cuda.core beta 2 milestone Nov 28, 2024
@ksimpson-work ksimpson-work self-assigned this Nov 28, 2024
Copy link

copy-pr-bot bot commented Nov 28, 2024

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ksimpson-work ksimpson-work added bug Something isn't working and removed feature New feature or request labels Nov 28, 2024
@ksimpson-work ksimpson-work force-pushed the ksimpson/tcc_memory_resource branch from 42346c5 to 319a372 Compare November 29, 2024 19:17
cuda_core/cuda/core/experimental/_memory.py Outdated Show resolved Hide resolved
cuda_core/cuda/core/experimental/_memory.py Outdated Show resolved Hide resolved
@ksimpson-work ksimpson-work marked this pull request as ready for review December 3, 2024 21:26
cuda_core/cuda/core/experimental/_device.py Outdated Show resolved Hide resolved
cuda_core/cuda/core/experimental/_memory.py Outdated Show resolved Hide resolved
cuda_core/cuda/core/experimental/_memory.py Outdated Show resolved Hide resolved
@ksimpson-work
Copy link
Contributor Author

re-tested using synchronous malloc and free on Tesla T4 colossus instance.

on main branch:

(test_env) C:\cuda-python\cuda_core>python -m pytest tests\test_memory.py

platform win32 -- Python 3.12.7, pytest-8.3.4, pluggy-1.5.0
rootdir: C:\cuda-python\cuda_core
configfile: pyproject.toml
collected 4 items

tests\test_memory.py FFFF [100%]

with change:

(test_env) C:\cuda-python\cuda_core>python -m pytest tests\test_memory.py

platform win32 -- Python 3.12.7, pytest-8.3.4, pluggy-1.5.0
rootdir: C:\cuda-python\cuda_core
configfile: pyproject.toml
collected 4 items

tests\test_memory.py .... [100%]

@ksimpson-work ksimpson-work requested a review from leofang December 4, 2024 17:36
@leofang
Copy link
Member

leofang commented Dec 6, 2024

/ok to test

@leofang
Copy link
Member

leofang commented Dec 6, 2024

Windows failures are known (#271) and irrelevant. Let's merge. Thanks, Keenan!

@leofang leofang merged commit c6a1a94 into main Dec 6, 2024
9 of 12 checks passed
@leofang leofang deleted the ksimpson/tcc_memory_resource branch December 6, 2024 22:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working cuda.core Everything related to the cuda.core module P0 High priority - Must do!
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Check TCC/WDDM modes during Device initialization
2 participants