TPU-related docs are somewhat confusing

[dinhanhx]

📚 Documentation

In Trainer class api,

it said not to use accelerator parameter and will be removed in 1.7.0 (current stable version is 1.7.1), BUT tpu_cores said to use accelator.

In Accelerator: TPU Training (also stable version), all the examples use accelerator='tpu'.

I don't really know which one to follow. And when I head to, What is a Strategy?

I don't really understand why we need strategy="ddp_spawn" here when without it, the model is still trained on 8 cores?
Moreover, Learn more of TPU links to examples that don't use any TPU strategy?

This example should use precision='bf16'

---

If you enjoy Lightning, check out our other projects! ⚡

• Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

• Lite: enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.

• Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.

• Bolts: Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.

• Lightning Transformers: Flexible interface for high-performance research using SOTA Transformers leveraging PyTorch Lightning, Transformers, and Hydra.

cc @Borda @carmocca @JackCaoG @steventk-g @Liyang90 @rohitgr7 @kaushikb11

[rohitgr7]

it said not to use accelerator parameter and will be removed in 1.7.0 (current stable version is 1.7.1), BUT tpu_cores said to use accelator.

it says passing strategy arguments ('ddp', dp', etc..) to accelerator is deprecated, not the complete parameter.

I don't really understand why we need strategy="ddp_spawn" here when without it, the model is still trained on 8 cores?

yes, but it depends upon devices if you don't set the strategy flag explicitly. With devices=8, it uses ddp_spawn else trains on one TPU core.

This example should use precision='bf16'

yes, it will through a warning and use bf16. But it's a little weird, maybe we should deprecate 16 for TPU and remove it later?
it was added here by @carmocca

[carmocca]

maybe we should deprecate 16 for TPU and remove it later?

I had the same question in #10020 (comment) (the comment liked as "fixed" by #10026). But I did not end up deprecating it to "ease transition between accelerator environments" (https://github.com/Lightning-AI/lightning/pull/10026/files#diff-2d97e0bfd5a239885ea0702336cbf3f6dbdaee6a5e255f5ca4da15cb5e260ea9R613)

I somewhat lean towards deprecation in case it ever becomes supported. The docs should be updated regardless of this decision

[rohitgr7]

But I did not end up deprecating it to "ease transition between accelerator environments"

you mean someone changing from CPU or GPU to TPU?

[carmocca]

Yes

[dinhanhx]

Re @

it says passing strategy arguments ('ddp', dp', etc..) to accelerator is deprecated, not the complete parameter.

This was on me. I misread the sentences. It means passing strategy options.

Thanks for the rest

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions - the Lightning Team!