Compile protobuf messages to multiple languages for easier client development

[clevinson]

Summary

Add a build script to the SDK for code generation for .proto files into several languages as packages. This can be executed through github actions on release tags/nightlies, and integrated with "Github Packages" to then publish packages on registries like npm, maven, etc.

Problem Definition

Client developers need to be able to easily send transactions, and query state from a Cosmos SDK chain. Since Stargate (v0.40.0), the preferred client approach here requires clients to compile proto files into generated code, so they can build messages, transactions, or generate sign bytes locally and then communicate via grpc (or alternatively, a corresponding REST endpoint generated via grpc gateway).

An easy way to reduce overhead on clients is for us to take care of the code generation for common client languages, and publish packages with this generated code.

Proposal

Use github action scripts (triggered nightly and on release tags), which avoids us having generated client code checked-in to the repo. This can be integrated with github packages to allow for publishing of packages for use by client developers.

The script could compile generated code in the following languages:

• js
• Java (android)
• ObjC/Swift (ios)
• Rust
• C++
• C#
• python

The focus of these packages is on canonical types that clients can reference for published versions, as well as to have up to date versions for testing code that’s on master and RC's.

If we wanted to build out client libraries with basic grpc wiring and/or signbytes functions, that could be done later with a separate repo and build artifact.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[aaronc]

Thanks @clevinson .

Just to make a few things clear, the SDK would be responsible for:

• having proper CI build scripts to publish these artifacts
• making sure language-specific options are correct (ex. java_outer_classname, etc.)
• (optionally) stripping all gogoproto extensions

With this proposal the SDK would NOT:

• have any generated code checked into git - the generated code would live in a temporary build/ folder and just get created in CI for publishing canonical packages
• have any custom code for other languages (SignBytes implementation, gRPC wiring, etc.) - this stuff is for separate repos that we don't maintain

I would note that it is already our responsibility to set language-specific options in our .proto files correctly. We just don't know what these are because haven't generated the Java or Python code for example.

The benefits I see to doing this are:

• a canonical set of basic types that are updated regularly so that clients can just worry about their code, not our updating our .proto files. With nightly and tagged builds clients could more easily test off master and RC's
• we have immediate visibility into the correct options (ex. java_outer_classname) for our code so clients don't need to tell us this

[aaronc]

It would be nice to have some client input here actually.

@ethanfrey @webmaster128 would the CosmJs and CosmWasm teams find it useful if there were say auto-published NPM packages or rust crates with just the generated code?

[webmaster128]

Thank you for the ping @aaronc. The .proto definition is a great universal interface language that many developers from many different ecosystem agree on. I appreciate the good intention, but from my perspective this proposal is a step in the wrong direction. This gets you into client development for too many too diverse technology, where you probably don't want to be in.

Generating client code from a given set of proto files is highly subjective. While one client or client library might be happy with your code generation choices, the next one might not – even in the same languages. Here are some of the decisions we make for code generation that we want to control and the SDK team would have a hard time getting right in dialogue with external teams:

• File selection: If a client only supports a subset of the functionality, you don't want to include more auto-generated code than necessary.
• .proto preprocessing: As mentioned above, you might want to strip off some annotations. In Weave times, we replaced all // comments with /// have them interpreted as doc comments by protobuf.js. Some projects might need license headers. You get the point.
• Protobuf library choice: At least in the JS ecosystem, the generated code is bound to one protobuf library. Usually multiple exist with different pros and cons. Different clients choose different libraries.
• Code generation settings: You configure your code generator, which is not trivial. E.g. one decision we just made this week is to not use --keep-case, such that all field names are transformed to camelCase. This is nice for working with the generated code in JS, but breaks Amino JSON signing. We mitigate this problem by handling messages differently than the other proto structures. Another JS library might want to make a different choice.

The next big problem is that even of this works somehow for common Cosmos SDK .protos, you need a concept to expand this to custom proto files from custom chains. Even the most simple message custom message type means you have generated code for some of your functionality, but not all of it. So you need to create a custom code generation process anyways.

I agree not every app developer should do all of this work from scratch. A more scalable solution for this is having fully featured client libraries built by experts in the particular technology, like CosmJS is for JS.

If you want to support client developers, I see auto-generating versioned, online .proto docs as a better tool. This promotes .proto files as the primary interface and encourages writing properly documented .proto files. See e.g. here for all the weave types.

[aaronc]

Thanks for the detailed feedback @webmaster128 ! Sounds like too big a can of worms.

Maybe let's close this issue @clevinson and open one for proto docs?

What tool did you use for that @webmaster128 ?

[webmaster128]

Thanks for the detailed feedback @webmaster128 ! Sounds like too big a can of worms.

My pleasure, as always.

What tool did you use for that @webmaster128 ?

See those scrips:

• https://github.com/iov-one/weave/blob/master/Makefile#L79-L81
• https://github.com/iov-one/weave/blob/master/scripts/clean_protos.sh
• https://github.com/iov-one/weave/blob/master/scripts/build_protodocs_docker.sh

[aaronc]

I'd kind of like to reopen this now that buf schema registry exists and all the cosmos types are now in the registry: https://buf.build/cosmos/cosmos-sdk and they have remote plugin execution: https://docs.buf.build/bsr/remote-generation/remote-plugin-execution. Generating some basic setups for the official plugins should be quite easy now and lower the barrier to entry. People can always make a custom build in a different artifact if they have more custom needs or help us properly customize the buf.gen.yaml.

[jc0803kevin]

Use Proto in the release（cosmos-sdk-cosmovisor-v1.3.0）

However, compiling the Proto in Java causes an error。
This should be an incompatibility problem in Proto。

[tac0turtle]

Could you post the output for us to verify the issue?

[webmaster128]

@jc0803kevin @marbar3778 could you move this issue into a separate ticket? #7717 is about a major strategy decision and it would be great to not start debugging language/tooling specific problems here.

On the topic of #7717: To me the development of cosmjs-types and now Telescope shows that in practice code generation is and should be done by users who deeply understand their language and dev environment. The .proto files should remain the interface we all agree on. From there we have one or two language specific code generators or libraries that all app builders can utilize. I still don't see a need for the SDK team to own code generation.

[tac0turtle]

I 100% agree with this. Closing the issue for now. Lets reopen this if it becomes a larger discussion again