ch4/ofi: Lazily register FI_MULTI_RECV buffers

[raffenet]

Pull Request Description

Avoid consuming GPU resources during initialization by using regular malloc for FI_MULTI_RECV buffers. Do lazy registration when the first GPU communication is detected.

Author Checklist

• Provide Description
  Particularly focus on why, not what. Reference background, issues, test failures, xfail entries, etc.
• Commits Follow Good Practice
  Commits are self-contained and do not do two things at once.
  Commit message is of the form: module: short description
  Commit message explains what's in the commit.
• Passes All Tests
  Whitespace checker. Warnings test. Additional tests via comments.
• Contribution Agreement
  For non-Argonne authors, check contribution agreement.
  If necessary, request an explicit comment from your companies PR approval manager.

[raffenet]

test:mpich/ch4/ofi

[raffenet]

test:mpich/ch4/ofi
test:mpich/ch4/gpu/ofi

[raffenet]

test:mpich/ch4/ofi
test:mpich/ch4/gpu/ofi

[raffenet]

TODO: move CVAR for tmp buf registration into mpir_gpu.h utils. Will add patch and then this PR should be good.

[raffenet]

test:mpich/ch4/ofi
test:mpich/ch4/gpu/ofi

[raffenet]

test:mpich/ch4/ofi
test:mpich/ch4/gpu

[hzhou]

@raffenet Can we do a gpu test with the CVAR disabling the host registration?

[raffenet]

@raffenet Can we do a gpu test with the CVAR disabling the host registration?

Actually the CVAR disables registration by default. I can add a dummy commit to re-enable registration and re-run, if desired.

[hzhou]

@raffenet Can we do a gpu test with the CVAR disabling the host registration?

Actually the CVAR disables registration by default. I can add a dummy commit to re-enable registration and re-run, if desired.

I see. I was hoping some of the GPU testing failures can be addressed by not registering the buffer. But a bummer.

Can you confirm that we fixed the GPU memory issue? Since we do that lazy register, I think we can leave the CVAR default on, right?

[raffenet]

@raffenet Can we do a gpu test with the CVAR disabling the host registration?

Actually the CVAR disables registration by default. I can add a dummy commit to re-enable registration and re-run, if desired.

I see. I was hoping some of the GPU testing failures can be addressed by not registering the buffer. But a bummer.

Can you confirm that we fixed the GPU memory issue? Since we do that lazy register, I think we can leave the CVAR default on, right?

Yes, I think we can default it to on. I'll double check a hello world program and confirm we don't take up any resources.

[raffenet]

test:mpich/ch4/ofi
test:mpich/ch4/gpu

[raffenet]

nvidia-smi output from a program held in wait loop after MPI_Init. Looks like we are no longer creating any resources.

[raffenet@pmrs-gpu-240-01]~% nvidia-smi
Thu Mar  2 09:09:08 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro RTX 4000     Off  | 00000000:5E:00.0 Off |                  N/A |
| 30%   32C    P8    10W / 125W |      3MiB /  8192MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Quadro RTX 4000     Off  | 00000000:D8:00.0 Off |                  N/A |
| 30%   37C    P8     8W / 125W |      3MiB /  8192MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

[raffenet]

Test results are consistent with registration turned back on. Outstanding question is, do we want to have an additional switch in ch4/ofi to disable registration of FI_MULTI_RECV buffers in case, say, the provider is already handling it?

[hzhou]

Test results are consistent with registration turned back on. Outstanding question is, do we want to have an additional switch in ch4/ofi to disable registration of FI_MULTI_RECV buffers in case, say, the provider is already handling it?

I see. You mean when provider will register the multi-recv buffer? I tend to believe that CUDA or any GPU runtime will cache the address and additional registration should be no-op. In any case, I would suggest that let's not worry about such case until they become a fact, and make decision then.

[hzhou]

LGTM

[raffenet]

Test results are consistent with registration turned back on. Outstanding question is, do we want to have an additional switch in ch4/ofi to disable registration of FI_MULTI_RECV buffers in case, say, the provider is already handling it?

I see. You mean when provider will register the multi-recv buffer? I tend to believe that CUDA or any GPU runtime will cache the address and additional registration should be no-op. In any case, I would suggest that let's not worry about such case until they become a fact, and make decision then.

Works for me.