Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix bf16 support issues #2238

Closed
wants to merge 2 commits into from
Closed

Fix bf16 support issues #2238

wants to merge 2 commits into from

Commits on Dec 28, 2023

  1. switch between hip and cuda c++ lib so load (pytorch#2236) 

    Summary:

- Switch to hip related TARGETS (w/ _hip suffix) when AMD GPU build is used.
- Add "supports_python_dlopen = True," to support dlopen on related deps.
- Add missing deps like `"//deeplearning/fbgemm/fbgemm_gpu:split_table_batched_embeddings_hip",`

Reviewed By: q10, zoranzhao

Differential Revision: D52435932
    @jianyuh @facebook-github-bot
    jianyuh authored and facebook-github-bot committed Dec 28, 2023
    Configuration menu
    Copy the full SHA
    6507269 View commit details
    Browse the repository at this point in the history

  2. Fix bf16 support issues (pytorch#2238) 

    Summary:

For bf16 related cuda code, we have the following macro to distinguish between v100 vs. a100 (pre-a100 cuda/NV GPU doesn't support BF16):
```
#if !(                                                  \
    ((defined(CUDA_VERSION) && CUDA_VERSION < 11000) || \
     (defined(__CUDA_ARCH__) && (__CUDA_ARCH__ < 800))))
```
macro.

For AMD GPU (rocm), it will lead to always false. However, on the MI250 / MI300 GPU we have in house, they have BF16 supports. We re-enable BF16 for RoCM related usages.

Reviewed By: houseroad, jiawenliu64

Differential Revision: D52438898
    @jianyuh @facebook-github-bot
    jianyuh authored and facebook-github-bot committed Dec 28, 2023
    Configuration menu
    Copy the full SHA
    0f82766 View commit details
    Browse the repository at this point in the history