Fix HorovodStrategy Teardown() Edge Case

[speediedan]

🐛 Bug

HorovodStrategy.teardown() may not complete gracefully if an exception is thrown before HorovodStrategy._exit_stack is set.

    def teardown(self) -> None:
        super().teardown()
>       self._exit_stack.__exit__(None, None, None)
E       AttributeError: 'NoneType' object has no attribute '__exit__'

pytorch_lightning[/strategies/horovod.py:200](): AttributeError

Notice below that self._exit_stack isn't set until callback setup() hooks are called:

https://github.com/PyTorchLightning/pytorch-lightning/blob/58324b5197aef20eb7acb577f953c0fae7c2dc05/pytorch_lightning/strategies/horovod.py#L79-L84

To Reproduce

Test to reproduce

Expected behavior

If an exception is thrown in any callback setup() hook, the Horovod teardown should still proceed without error.

Environment

• CUDA:
  • GPU:
    • GeForce RTX 2070 SUPER
    • GeForce RTX 2070
  • available: True
  • version: 11.3
• Packages:
  • numpy: 1.21.2
  • pyTorch_debug: False
  • pyTorch_version: 1.10.1
  • pytorch-lightning: 1.6.0dev
  • tqdm: 4.62.3
• System:
  • OS: Linux
  • architecture:
    • 64bit
  • processor: x86_64
  • python: 3.7.11
  • version: docs: enable syntax highlight #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022

Additional context

I'll be uploading a PR to address this bug shortly

cc @awaelchli

[awaelchli]

@speediedan Good that you found this, thanks for sending the PR!