🐛[bug] Distributed training not working for PyTorchTrial when using evaluate_full_dataset and more than 1 GPU

[charles-viss]

Describe the bug

When training with a PyTorchTrial, the validation_loader for the PyTorchTrialController is set to None for all processes with rank > 0: https://github.com/determined-ai/determined/blob/main/harness/determined/pytorch/_pytorch_trial.py#L463-L465

But then when performing validation with evaluate_full_dataset, the PyTorchTrialController asserts that validation_loader must not be None for all processes: https://github.com/determined-ai/determined/blob/main/harness/determined/pytorch/_pytorch_trial.py#L995

Reproduction Steps

1.Train with a PyTorch trial using more than one GPU and evaluate_full_dataset

Expected Behavior

With evaluate_full_dataset, only the rank=0 process should require that validation_loader is not None. Could the assertion be moved after the if self.is_cheif statement? https://github.com/determined-ai/determined/blob/main/harness/determined/pytorch/_pytorch_trial.py#L996

Screenshot

N/A

Environment

N/A

Additional Context

No response

[azhou-determined]

This indeed seems like a pretty obvious bug on our part. 🤦

We'll land this fix before next release (0.37.0) slated to go out in about two weeks. Thank you for reporting this.