[PyTorch] Add CUDA graph tests with FP8 weight caching

[timmoon10]

Description

Our CUDA graph infrastructure (#575) supports FP8 weight caching when training with multiple gradient accumulation steps. While adding tests for this functionality (see #820 (comment)), I ran into subtle correctness issues because te.make_graphed_callables resets grad buffers after capturing graphs, so the gradient buffer filled in the backward pass is different from the gradient buffer used by the optimizer. Note that we didn't detect this before because Megatron-LM and Nemo explicitly manage gradient buffers, e.g. with a distributed optimizer.

This PR modifies the CUDA graph tests to initialize grads before make_graphed_callables and it avoids resetting grads within make_graphed_callables.

Type of change

• Documentation change (change only to the documentation, either a fix or a new content)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Changes

Please list the changes introduced in this PR:

• Add CUDA graph tests for FP8 weight caching
• Do not reset parameter gradients after graph capture

Checklist:

• I have read and followed the contributing guidelines
• The functionality is complete
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[timmoon10]

Note that users must also initialize grads before calling torch.cuda.make_graphed_callables, or else they'll run into similar correctness issues. Consider the following example:

import torch
torch.set_default_device('cuda')

# Construct linear module
model = torch.nn.Linear(1, 1, bias=False)
with torch.no_grad():
    model.weight.fill_(1)
    # model.weight.grad = torch.empty_like(model.weight)  # Uncomment to fix bug

# Capture CUDA graph
x = torch.ones((1, 1), requires_grad=True)
model = torch.cuda.make_graphed_callables(model, (x,))

# Training steps
for step in range(3):
    if model.weight.grad is not None:
        model.weight.grad.zero_()
    x = torch.ones((1, 1), requires_grad=True)
    y = model(x)
    y.backward(torch.ones((1, 1)))
    print(f"{step=}, {model.weight.grad.item()=}")

I expect the weight gradient to always be 1. However:

step=0, model.weight.grad.item()=1.0
step=1, model.weight.grad.item()=2.0
step=2, model.weight.grad.item()=2.0

[timmoon10]

/te-ci pytorch

[nvcforster]

I left a question about coverage, otherwise LGTM.

[nvcforster]

In dropping the adam optimizer from the test parameterization, are we losing any test coverage w.r.t. cuda graphs?

[timmoon10]

This test doesn't really test convergence, but just checks that results match exactly with and without CUDA graphs. SGD and Adam are implemented similarly, so we just need to test one to make sure that the CUDA graph infrastructure is working correctly.