Upgrade cloud.google.com/go/storage to v1.16.0

[mdgreenfield]

Hello, I was looking at some strange GCS storage errors and in looking at the cloud.google.com/go/storage currently used in Vault (v1.6.0) I noticed it was outdated and there are several updates to the transport and retry logic that hopefully can make the GCS storage backend a little more robust when under load.

One of the errors I was seeing was an EOF error but I don't have the exact error message any more. It looked like the following changelog entry in v1.15.0 may have fixed the issue:

storage: retry io.ErrUnexpectedEOF (#3957) (f6590cd)

[fcrespofastly]

We see some errors while hitting renew-self:

        * failed to persist lease entry: 1 error occurred:
        * error closing connection: Post "https://storage.googleapis.com/upload/storage/v1/b/my-vault-storage/o?alt=json&prettyPrint=false&projection=full&uploadType=multipart": context canceled

We saw similar issues on other services using GCS to store data and the response from Google support was that GCS will close the connection when it's busy and clients should implement retrying logic.

Not sure if this is exactly the same issue @mdgreenfield is exposing, but curious to see if this fixes it.

[heatherezell]

Hi folks! Is this still an issue in newer versions of Vault? Please let me know so I can bubble it up accordingly. Thanks!

[raskchanky]

Since it's been awhile with no response, I'm going to close this issue. Please reopen if there's more to add here.