Massa gRPC API V2

[aoudiamoncef]

This issue tracks the work in progress on the newer Massa JsonRPC API.

Refactor the API:

1 - Specify the business needs with Core, the Innovation team, direct users (eg. Dusa) and the community Builders. Try to split endpoints as much as possible. Use snake_case everywhere (#2823). Decide if we fuse the private/public APIs and allow finer access control through other means.

2 - Write a new API that uses https://github.com/hyperium/tonic crate and will support the following features:

• Schema reflection (introspection). See https://github.com/hyperium/tonic/blob/master/tonic-reflection/tests/server.rs
• Support Http 1.1 transport. See https://github.com/hyperium/tonic/blob/master/tonic-web/README.md
• Limit size of message(s). See Config max request message size to fend off potential malicious attacks hyperium/tonic#1097
• Support gateway annotations. See Support gRPC-gateway annotations hyperium/tonic#332
• Support Json instead of Protobuf. See https://github.com/hyperium/tonic/blob/master/examples/src/json-codec/common.rs
• Support of service health check. See https://github.com/hyperium/tonic/blob/master/tonic-health/README.md
• Support mTLS. Support mTLS #3815
• Support client authorizations via Support Certificate Revocation List (CRL) for mTLS #4315
• Postman gRPC client with Massa collection. See https://www.postman.com/massalabs
• Support bulk(batch) methods
• Add better logging in the gRPC API
• Support WebSocket binding to gRPC streaming. See Split out factory and pool #2895 and stream events for SC applications (@BatiGencho)
• Support rate limiting https://github.com/benwis/tower-governor
• Support custom Tokio runtime
• Support pagination. See paginate crate Implement pagination in JsonRPC endpoints #3428
• Full configuration support from Massa config system Full configuration support from Massa config system in gRPC API #3600
• Build a CI/CD around .proto files Build a CI/CD around .proto files #3622. See https://github.com/massalabs/massa-proto
• Force minimal version of protoc in build.rs. See https://github.com/datafuselabs/databend/blob/main/src/meta/protos/build.rs
• Migrate massa core to use massa-proto-rs Migrate to massa-proto-rs in gRPC API #4050
• Add serialized_size field to SecureShares in APIs Add serialized_size field to SecureShares in APIs  #4019

3 - Let's also add the following endpoints:

• SendEndrosments Implement send_endorsements method in gRPC API #3423
• SendOperations Implement send_endorsements method in gRPC API #3642
• SentBlocks Implement send_blocks method in gRPC API #3420
• GetSelectorDraws Implement get_selector_draws method in gRPC API #3421
• GetNextBlockBestParents Implement get_next_block_best_parents method in gRPC API #3419
• GetLatestBlock
• GetLatestOperation
• GetLatestEndorsement
• GetBlocksBySlots Implement get_blocks_by_slots method in gRPC API #3425
• GetExecutedOperations
• GetLargestWallet Implement GetLargestWallets method in gRPC API #3427
• GetLargestStakers Implement getLargestStakers method in gRPC API #3485
• GetTransactionsThroughput Implement get_transactions_throughput method/subscription in gRPC API #3426
• GetBalances Implement GetBalances method in gRPC API  #3813
• GetSlots (need by Inno: slot ID, ASC execution and its gas consumption, deferred credits and its gas consumption, and block object and its gas consumption, since they have different gas limits) Implement GetSlots method in gRPC API #3593
• GetRolls
• GetHealth already done implicitly
• GetStakingActivation
• GetStakingInfos
• GetTokenInfos
• GetBlocks Implement GetBlocks method in gRPC API #3839
• GetOperations Implement GetOperations method in gRPC API #3835
• SimulateOperations already exists
• GetVersion
• GetDatastoreEntries Implement get_datastore_entries in gRPC API #3621
• GetScExecutionEvents Implement GetScExecutionEvents in gRPC API #3867

4 - As well as streaming methods for:

• NewBlocks
• NewBlocks_headers
• NewFilledBlocks
• NewOperations
• NewEndorsements Implement NewEndorsements method in gRPC API  #3833
• NewBlockCliques Implement NewBlockCliqueChanges method in gRPC API #3857
• NewSlotExecutionOutputs Implement NewSlotExecutionOutputs method in gRPC API #3810
• NewTransactionsThroughput Implement get_transactions_throughput method/subscription in gRPC API #3426
• NewStateEventsChanges Implement NewScExecutionEvents method in gRPC API  #3812

Existing WebSockets should be migrated to equivalent gRPC streams.

There is a discussion about gRPC support.
This will pave the way towards #2903 in particular, and also help @BatiGencho with execution event streaming for web3.

previously #2799 #3076

[aoudiamoncef]

Hi @massalabs/core-team,

As we are going to accelerate the development of Massa API V2, we are gathering all enhancements proposals in this issue.

Based on clients feedbacks, I made the list of methods/subscriptions/features above, we should select, drop and prioritise what's really add a value to Massa.

I'm waiting for feedbacks, feel free to update and/or comment issue.

Thanks 👍🏻

[banool]

I see you have ticked off "Support gateway annotations", how are you planning to implement that given it isn't supported in the underlying libraries?

[aoudiamoncef]

Hi @banool

We added gateway annotations in our gRPC service definition to be used by reverse proxies Eg Envoy and other languages E.g in go

In the future, if Prost/Tonic adds support for a Rust based gateway, we're already ready.