TPU CI process flakiness

[DuYicong515]

🐛 Bug

CI-TPU tests are constantly failing because of resource issue.
Common reasons are

1. TPU resources are not available right now. Log messages look like No resources found in default namespace.
2. During the execution of the tests, the TPU machine got taken away. This will show us with an incomplete log + message like Exited with code exit status 1, CircleCI received exit code 1

In addition, the TPU tests logs are not very useful for contributor to debug issues right now.
For example in #12151(CI-TPU testing link), test_trainer_config_device_ids, test_accelerator_tpu, test_set_devices_if_none_tpu were all passing. However, multiple tests unrelated to this PR were failing with logs related to Cannot replicate if number of devices (1) is different from 8 or can't move model to device. All those failed tests seem to fit models after the 3 PR relevant tests are run. However, the logs aren't very helpful to debug why unrelated tests are failing.

To Reproduce

Submit a PR and run CI-TPU tests. Example failures are in https://app.circleci.com/pipelines/github/PyTorchLightning/pytorch-lightning?branch=pull%2F12151&filter=all

Expected behavior

1. TPU tests pass/fail are deterministic. The resource issues need to be looked at
2. Have more clear contributor guidelines on how to work with TPU testing. Examples include when @pl_multi_process_testshould be used in TPU tests, how can we optimize TPU tests performance to avoid timeouts, how do we tell timeouts from resource being taken in the middle
3. Improve TPU CI process log messages

cc @carmocca @akihironitta @Borda @kaushikb11 @rohitgr7

[akihironitta]

Also, it would be very nice to see logs in real-time.

For the note, currently, we can see logs from CI only after the TPU CI job succeeds or fails (which usually takes up to 20min+) as dumped at:
https://github.com/PyTorchLightning/pytorch-lightning/blob/8b4abe4edb6912abeb3906c48bb822a3681b08c4/.circleci/config.yml#L96-L99

[carmocca]

Closing in favor of #13720