Add an option for crc32/md5 client side validation for Storage.readAllBytes

[aozarov]

From @Capstan "Read-side validation of checksums (crc32c) is a very nice-to-have".

[mziccard]

Just to be sure: the crc32/md5 values should be user provided right? I could not find a way of getting the service's ones when downloading object's data.

[Capstan]

We have an internal bug filed to provide it when downloading via headers. You can also get them via object metadata lookups, but that means a second API call if you don't already have the metadata.

[Capstan]

(The XML API already provides these via headers; it'd be a JSON API improvement to add them.)

[mziccard]

@Capstan Yes I noticed that with the XML API we could access them via headers. However we would lose some other options.
We also avoided to do "extra" metadata requests elsewhere in gcloud-java and I think we should stick to this practice.

How to you see allowing the user to provide an explicity value for both crc32c and md5 until the JSON API adds the missing headers? Something like:

storage.readAllBytes(BlobId blob, BlobReadOption.crc32cMatch("42"));

When using functional blob (which contains metadata) we could instead have BlobSourceOption.crc32cMatch():

blob.content(BlobReadOption.crc32cMatch());

[aozarov]

@mziccard I think we should wait with this issue until the Json API supports it.
@Capstan is there a way for us to track it?

[Capstan]

@azarov I don't believe we have an external mirror of Google's internal bug tracking database. If you know otherwise, or some linked db, I'm happy to use that.

@mziccard As for user-provided, we're not necessarily expecting users to have kept track of the hashes of their objects in a local datastore (though they well could), so this is less about specifying them as conditions and more about validating that the data you read is consistent. Bits can be flipped all the way from storage medium out to the network, even if TLS is used, and even then on the client's machine from the network buffer to the consumption point. A 200 only means the service thinks it's yielding you the right data: the final determination can only be made by the client that the checksums are in agreement with the received data.

[mziccard]

@Capstan I see your point. Nevertheless having the user to explicitly provide those values was a way to avoid doing an extra get metadata under the hood: the user might have already got metadata or can get them himself before downloading the object.

As @aozarov said we should probably wait for the headers to be supported by the JSON API.

[garrettjonesgoogle]

@Capstan any update?

[garrettjonesgoogle]

This has been added to our feature backlog: https://github.com/GoogleCloudPlatform/google-cloud-java/wiki/Feature-backlog . This issue will be closed but is linked in the backlog and can continue to be used for comment and discussion.