[GCP Storage] Upgrade GCSFuse version

[Michaelvll]

We encountered an issue with the gcsfuse that the saved large model checkpoints on a mounted GCS bucket can be incomplete. It could be a problem with the old gcsfuse we were using:
in the release note of gcsfuse (discovered by @infwinston):

Fixed a critical bug in write workflow - although it was very rare, but gcsfuse was writing wrong file content while doing append operation on larger file.

https://github.com/GoogleCloudPlatform/gcsfuse/releases/tag/v0.41.11

Tested (run the relevant ones):

• Any manual or new tests for this PR (please specify below)
  • Launch a node and write things to the mounted bucket.
  • pytest tests/test_smoke.py::test_aws_storage_mounts --aws; to test the mounting of GCS bucket on AWS VM.
  • pytest tests/test_smoke.py::test_gcp_storage_mounts
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[romilbhardwaj]

Thanks! Should we bump to 0.42.3 (latest) if we're upgrading versions?

[Michaelvll]

Thanks! Should we bump to 0.42.3 (latest) if we're upgrading versions?

It is 0.42.3 in the PR changes?

Weirdly, the launched the VM is still using 0.41.9, still trying to figure out why.

[romilbhardwaj]

Since not all mounting tools may have --version flag, can we change the arg version to version_check_command and the caller can pass whatever version check command they'd like to run?

E.g., in this case, the caller would pass version_check_command = 'gcsfuse --version | grep -q 1.0'

[Michaelvll]

Good point! Fixed. Thanks!

[romilbhardwaj]

Thanks for adding the version check! Should be good to go if pytest tests/test_smoke.py::test_gcp_storage_mounts passes.