[Core] Avoid race condition for key generation

[Michaelvll]

This is to avoid the racing condition when multiple spot, services or launch run concurrently on a new environemnt. Previously, it is possible that two process generates the public key and private key concurrently causing unmatched keys. For example,

1. process 1 writes private key
2. process 2 writes private key
3. process 2 writes public key
4. process 1 writes public key

This PR adds a lock for the key generation to avoid this race condition

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[concretevitamin]

LGTM, thanks @Michaelvll.

[concretevitamin]

Suggested change

	_SSH_KEY_GENERATION_LOCK = '~/.ssh/.sky-key.lock'
	# Lock protecting public/private sky-key generation.
	_SSH_KEY_GENERATION_LOCK = '~/.ssh/.sky-key.lock'

[concretevitamin]

How about putting all our generated files in ~/.sky/generated or a lock dir under ~/.sky?

[Michaelvll]

Good point! Moved it to ~/.sky/generated/ssh and test it in a new environment by running 5 concurrent launch.

for i in `seq 1 5`; do sky launch -c test-gcp-$i -d -y --cloud gcp --cpus 2 echo hi & done