Fix slowdown on init with device_map="auto"
#4720
| Job | Run time |
|---|---|
| 1m 58s | |
| 2m 48s | |
| 3m 32s | |
| 2m 26s | |
| 2m 23s | |
| 2m 29s | |
| 2m 28s | |
| 3m 53s | |
| 3m 38s | |
| 8m 45s | |
| 1m 58s | |
| 3m 0s | |
| 3m 25s | |
| 8m 30s | |
| 2m 34s | |
| 2m 25s | |
| 2m 25s | |
| 11m 3s | |
| 3m 38s | |
| 8m 48s | |
| 1h 22m 6s |