Use apitools for uploads/downloads

[silvolu]

uploads_test.py and downloads_test.py should provide good examples for it.

[tseaver]

I'm working on this, but maybe need some help from @craigcitro . The bits of apitools we need to do the upload / download still have dependencies on protorpc, even outside code generation / generated code:

apitools.base.py.base_api

• defines two classes derived from protorpc.messages.Message
• uses protorpc.messages.ValidationError and protorpc.messages.MessageFields classes for error handling / validation.
• Its _LoadClass uses the protorpc.message_types module for "simple" / scalar types.

apitools.base.py.encoding

• depends heavily on protorpc.messages.Message (its whole raison-d'etre is marshalling JSON using messages as schema).
• uses the protorpc.message_types module, but in only one place (and the usage is somewhat odd, excluding "normal" marshalling of protorpc.message_types.DateTimeField)
• uses the protorpc.protorpc.ProtoJson class to do a "deep copy", and has a helper class derived from it (used from MessageToJson() and JsonToMessage().

apitools.base.py.extra_types

• depends heavily on protorpc.messages.Message.
• re-exports protorpc.message_types.DateTimeMessage
• adds custom marshalling for protorpc.message_types.DateTimeField.
• uses the protorpc.protorpc.ProtoJson class in its cusoom marshlling of protorpc.message_types.DateTimeField.

For us to use apitools, we really need either a version of google-protorpc that supports our desired Python versions, or else we need to fork the needed bits of both into our codebase. Note that the protorpc modules we use (unlike most of the rest) don't pull in a dependency on GAE, so maybe we should be pulling them out somehow.

[tseaver]

@dhermes you've got some residual memories of protorpc -- maybe you can suggest something?

[craigcitro]

I'm actively working on getting the protorpc fixes pushed through -- it's going well, i've just got a few too many balls in the air right now.

how far away is protorpc for python3 support? (you'd think i would know, but haven't looked yet.)

[tseaver]

It is hard to estimate, given that running the tests is an unknown quantity. I've done some work today on adding tox support, but the tests don't all pass in the absence of the not-explicitly-depended-on GAE / protobuf code. I will post a gist showing the modernize results (after cleaning it up a bit).

[dhermes]

RE: "residual memories" I never got tests running outside of Google's BUILD environment, so I am in the same boat as @tseaver.

As for the usefulness of protorpc, I am not sold on it if we are only using it in generated code.

The value in user-land was that people could create protorpc.Message objects which behaved "exactly" like a typed object (something obviously not supported in Python).

We could write something much simpler given a JSON schema:
https://gist.github.com/dhermes/2cbf4352146239e1aca1

I don't know if @craigcitro has feelings on this, but I'd guess he's thought about it more than I have.

[tseaver]

As for the usefulness of protorpc, I am not sold on it if we are only using it in generated code.

We use a fairly narrow slice of it in non-generated code (see above).

[tseaver]

@craigcitro here is the gist I promised (I'd be glad to kick this in as another Hg branch):

https://gist.github.com/tseaver/6d28ef139b2a0dcb2eea

[silvolu]

Couldn't we just use transfer.py for uploads and downloads?
There shouldn't be no need for protorpc in that case.

[tseaver]

@silvolu looks like we could: that would use http_wrapper, exceptions, and util as well, and none of those import anything from protorpc. We still can't ship with a dependency on protorpc if it doesn't support the versions we want: I guess we could just fork those four modules (they don't have tests) into a "vendored" sub-package for now.

[silvolu]

Yeah, that's the plan B mentioned in the doc I sent along.

[tseaver]

@silvolu @craigcitro I'm not sure we can make use of apitools.base.py.transfer directly without the support of the stuff in apitools.base.py.base_api. In particular, transfer.Upload.InitializeUpload() and transfer.Download.InitializeDownload() APIs expect to be handed http_request and client arguments which are set up inside base_api.BaseApiService._RunMethod(), which has lots more hair than we should be replicating.

[tseaver]

@craigcitro maybe I should rephrase that as: "can you point me to an example that uses only apitools.base.py.transfer (and its imports) to do upload / download to gcloud-storage?

[craigcitro]

@tseaver the transfer code was indeed built to be used with base_api. i'll try and take a look at whether/how to disentangle those today.