Proto binding support for additional languages

[egaxhaj]

Summary

Add support for generating and publishing code for proto files in multiple languages.

Problem Definition

Currently the SDK provides support for generating go code from the proto files. It would be beneficial to also provide support for more languages. Languages such as Java, Kotlin, Rust, Python, etc.

The changes proposed here are a first step towards achieving it. Check out this branch for an implementation.

What problems may be addressed by introducing this feature?

Engineers can use a prefered language.

What benefits does the SDK stand to gain by including this feature?

Provide multi-language proto binding support.
Grow the community beyond go engineers.

Are there any disadvantages of including this feature?

Application developers need to be aware of language nuances. Will require adding custom support for unsupported languages.

Proposal

1. Use Gradle with Kotlin DSL as the build tool for multi-language support.

2. Add Gradle support to the project.

   // High level structure
   cosmos-sdk
       ...
       ├── buildSrc
       ├── buildProto
       ├── gradle
       ├── build.gradle.kts
       ├── settings.gradle.kts
       ├── gradle.properties
       ...
   
   

   buildSrc/ is where complex build logic for custom tasks or plugins goes. Gradle automatically compiles and tests this code and puts it in the classpath of the build script. Code residing in buildSrc directory is executed and compiled first before build.gradle.kts runs. Read the Gradle docs for more detail.

   Adding custom build language specific support.

   buildSrc
   ├── build.gradle.kts
   └── src
       └── main
           └── kotlin
               ├── Dependencies.kt  // Constant values
               ├── Version.kt       // Constant values
               └── cosmos
                   └── rust
                       ├── ProtobufRustGrpcPlugin.kt
                       └── ProtobufRustGrpcTask.kt
                   └── <other language>
                       ├── OtherLanguageGrpcPlugin.kt
                       └── OtherLanguageGrpcTask.kt
   

   ProtobufRustGrpcPlugin.kt

   package cosmos.rust
   
   import org.gradle.api.Plugin
   import org.gradle.api.Project
   import org.gradle.kotlin.dsl.register
   
   class ProtobufRustGrpcPlugin : Plugin<Project> {
       override fun apply(project: Project) {
           project.tasks.register(
               "protobuf-rust-grpc",
               ProtobufRustGrpcTask::class
           ) {
               this.group = "protobuf"
               this.description = "Generate proto source files for Rust."
           }
       }
   }
   
   

   ProtobufRustGrpcTask.kt

   package cosmos.rust
   
   import org.gradle.api.DefaultTask
   import org.gradle.api.tasks.TaskAction
   
   open class ProtobufRustGrpcTask : DefaultTask() {
   
       @TaskAction
       fun generateRustGrpc() {
           // ...
       }
   }
   

   buildProto/ contains the subproject build scripts for the supported languages. This is where build logic for each language binding resides. Java and Kotlin use the protobuf-gradle-plugin provided by google. For languages where community support is lacking, the custom method described in the buildSrc section is applied.

   buildProto
   ├── README.md
   ├── java
   │   └── build.gradle.kts
   ├── kotlin
   │   └── build.gradle.kts
   └── rust
       └── build.gradle.kts
   

   java/build.gradle.kts

   import com.google.protobuf.gradle.generateProtoTasks
   import com.google.protobuf.gradle.id
   import com.google.protobuf.gradle.ofSourceSet
   import com.google.protobuf.gradle.plugins
   import com.google.protobuf.gradle.protobuf
   import com.google.protobuf.gradle.protoc
   
   tasks.jar {
       // JAR base name. The full JAR name format ${baseName}-${x.x.x}.jar
       // See ./cosmos-sdk/settings.gradle.kts for `rootProject.name`
       baseName = "${rootProject.name}-proto-java"
   }
   
   tasks.withType<Javadoc> { enabled = true }
   
   tasks.withType<JavaCompile> {
       sourceCompatibility = JavaVersion.VERSION_11.toString()
       targetCompatibility = sourceCompatibility
   }
   
   // For more advanced options see: https://github.com/google/protobuf-gradle-plugin
   protobuf {
       protoc {
           // The artifact spec for the Protobuf Compiler
           artifact = Libraries.ProtocArtifact
       }
       plugins {
           // Optional: an artifact spec for a protoc plugin, with "grpc" as
           // the identifier, which can be referred to in the "plugins"
           // container of the "generateProtoTasks" closure.
           id(PluginIds.Grpc) {
               artifact = Libraries.GrpcArtifact
           }
       }
       generateProtoTasks {
           all().forEach { task ->
               task.plugins {
                   id(PluginIds.Grpc)
               }
           }
       }
   }
   

   kotlin/build.gradle.kts

   import com.google.protobuf.gradle.generateProtoTasks
   import com.google.protobuf.gradle.id
   import com.google.protobuf.gradle.ofSourceSet
   import com.google.protobuf.gradle.plugins
   import com.google.protobuf.gradle.protobuf
   import com.google.protobuf.gradle.protoc
   import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
   
   tasks.jar {
       baseName = "${rootProject.name}-proto-kotlin"
   }
   
   tasks.withType<Javadoc> { enabled = true }
   
   tasks.withType<JavaCompile> {
       sourceCompatibility = JavaVersion.VERSION_11.toString()
       targetCompatibility = sourceCompatibility
   }
   
   tasks.withType<KotlinCompile> {
       kotlinOptions {
           freeCompilerArgs = listOf("-Xjsr305=strict", "-Xopt-in=kotlin.RequiresOptIn")
           jvmTarget = "11"
           languageVersion = "1.5"
           apiVersion = "1.5"
       }
   }
   
   // For more advanced options see: https://github.com/google/protobuf-gradle-plugin
   protobuf {
       protoc {
           // The artifact spec for the Protobuf Compiler
           artifact = Libraries.ProtocArtifact
       }
       plugins {
           // Optional: an artifact spec for a protoc plugin, with "grpc" as
           // the identifier, which can be referred to in the "plugins"
           // container of the "generateProtoTasks" closure.
           id(PluginIds.Grpc) {
               artifact = Libraries.GrpcArtifact
           }
           id(PluginIds.GrpcKt) {
               artifact = Libraries.GrpcKotlinArtifact
           }
       }
       generateProtoTasks {
           all().forEach { task ->
               task.plugins {
                   id(PluginIds.Grpc)
                   id(PluginIds.GrpcKt)
               }
               task.builtins {
                   id(PluginIds.Kotlin)
               }
   
               task.generateDescriptorSet = true
           }
       }
   }
   

   rust/bin/grpc_rust_plugin

   #!/usr/bin/env bash
   
   # Use a pre-built docker image for Rust bindings
   docker run ...
   

   rust/build.gradle.kts

   import com.google.protobuf.gradle.generateProtoTasks
   import com.google.protobuf.gradle.id
   import com.google.protobuf.gradle.ofSourceSet
   import com.google.protobuf.gradle.plugins
   import com.google.protobuf.gradle.protobuf
   import com.google.protobuf.gradle.protoc
   
   // Register the custom task we created for Rust
   tasks.register<cosmos.rust.ProtobufRustGrpcTask>("generateRustGrpc")
   

   gradle/ contains the Gradle wrapper binaries.

   gradle
   └── wrapper
       ├── gradle-wrapper.jar
       └── gradle-wrapper.properties
   
   

   build.gradle.kts is the top level build script. It contains common dependencies for the project.

   settings.gradle.kts defines which projects are taking part in the multi-project build. Adds language bindings in a dynamic way as to avoid manual entry.

   // The name of your project.
   rootProject.name = "cosmos-sdk"
   
   // Dynamically include subprojects in `buildProto/`.
   // Uses the folder name as the project name.
   File(rootDir, "buildProto").walk().filter {
       it.isDirectory && File(it, "build.gradle.kts").isFile
   }.forEach {
       include(it.name)
       project(":${it.name}").projectDir = it
   }
   

   gradle.properties contains build script properties. This is where we also define the location of the proto files.

   kotlin.code.style=official
   kapt.use.worker.api=false
   kapt.incremental.apt=false
   org.gradle.jvmargs=-Xmx4096M
   # Source directory of the proto files that will be compiled.
   # A comma separated list of directories of the proto files for compilation.
   # Relative paths must start from the root of the project. Use absolute paths
   # if proto files are located outside the project root folder.
   protoDirs=proto/,third_party/proto/
   

3. Generating language bindings.

   # ./gradlew clean :<language>:<task>
   
   # Java
   ./gradlew clean :java:jar
   
   # Kotlin
   ./gradlew clean :kotlin:jar
   
   # Rust
   ./gradlew clean :rust:generateRustGrpc    # our custom task
   

   Note, to see a full list of tasks use the gradle-task-tree plugin commands. (For top-level tasks, run: ./gradlew tastkTree).

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[aaronc]

Thanks @egaxhaj-figure. This is a duplicate of #7717, so if it's okay I'm going to close this and let's continue discussing there. My sense is that using buf generate is a simpler solution as I proposed in #7717 (comment). Does that seem like it will solve your needs? If we can align on an approach, I'm happy to support official bindings in multiple languages.