feat: add TON blockchain support #4985

Keshoid · 2024-12-11T13:29:24Z

Description

This PR adds a TON blockchain support to hyperlane project.

add hyperlane-ton crate to workspace;
update relayer & validator to support TON chains;
add hyperlane smart contracts for TON blockchain;
add ton::test under run-locally utility with e2e test (ton<->ton).

Drive-by changes

Related issues

Backward compatibility

Yes

Testing

E2E test:

cargo test  --package run-locally --features ton test-utils -- ton::test --nocapture

* init commit * Initial setup for TonProvider implementation * Initial implementation of HyperlaneProvider methods (get_block_by_hash, get_txn_by_hash, is_contract, get_balance, get_chain_metrics) * Refactor: Renamed folder, updated field name, and cleaned up unused imports * implemented get_messages method for TonApiCenter * Refactor: Added ConnectionConf and updated TON provider requests * feat(ton-provider): Added get_account_state to provider and updated methods for improved logging and response handling * feat(provider): add run_get_method for querying smart contract state * added send_message request in ton-api-trait * feat: add get_wallet_states method to Ton provider * feat(ton-mailbox): scaffold structure and stub methods for Mailbox traits * implemented default_ism and count methods * implemented delivered method in mailbox * resolve lifetimes problems in TonMailbox implementation * added implementation SequenceAwareIndexer trait and recipient_ism method in Mailbox Ton * resolve conflicts in main.rs * added get_transaction_by_message in provider * implemented process method in mailbox and some updates in recipient_ism * added import utils * feat: Implemented TOnMailboxIndexer and get_blocks to Ton provider * feat: create TonInterchainSecurityModule and raw implemented InterchainSecurityModule trait * feat: added init implementation multisig ism * feat: initial implementation of Interchain Gas Paymaster * refactor(mailbox): rework build_message method * refactor(provider): move wait_for_transaction to provider * refactor(ISM): refactor dry_run_verify and module_type methods * refactor(multisig): rework validators_and_threshold getter * refactor: update cargo.toml and mod.rs * feat: integrate base64_to_H512 conversion in TonProvider and added utility tests * refactor(fetch_logs_in_range): switch to masterchain seqno for time extraction * refactor(get_finalized_block_number): switch to workchain -1 * feat: init implementation TonSigner * feat: init implementation validator_announce * feat: implemented get_announced_storage_locations in validatorAnnounce * updates error in signers.rs * Refactor error handling, added max attempts and delay to TonConnectionConfig * refactor error handling for get_blocks in TonProvider * feature: add conversion utilities and unit tests for address linking and parsing * feature: new implementation get_announced_storage_locations in validator_announce * feature: add conversion utils with new conversion methods and updateimports tonlib -> tonlib_core * refactor(tonlib-core): migrate from tonlib to tonlib_core with updated methods * feature:(TonSigner): added create_signed_message and constructor from_mnemonic in TonSigner * refactor(contracts): replace TonWallet with TonSigner in off-chain components * Refactor hyperlane-ton under hyperlane-core * feature: integrate initial support for TON blockchain in hyperlane-base and hyperlane-core * feat(utils): add method for converting H256 to TonAddress * added using conversionUtils for parsing H256 address * refactor(chains/hyperlane-ton and hyperlane-base/settings): migrated from tonlib to tonlib_core and resolved warnings * feat: init commit e2e tests for chain hyperlane-ton * feat(contracts): add TON contracts with empty implementations (merkle_tree_hook, aggregation_ism, routing_ism) and other micro updates * feat(ton-signer): implement Debug trait for DebugWalletVersion * feat(hyperlane-base and hyperlane-core): update data creation and parsing for Ton offchain components * feat(run_locally): add initial draft for e2e tests in TON * feature(ton-provider): add handler H256 to TON address conversion in get_wallet_states * fix(cosmos_provider): update path to fix compilation with main repository * fix(lib.rs): delete mod mailbox_test * feat(run_locally ton):some updates * feat(hyperlane-base/chains): init implementation for create TonMailboxIndexer * feat(TonMailbox): init impelmentation Indexer<H256> for TonMailbox * feat(ton provider): some updates in TonProvider * feat(ton mailbox and igp): some updates * Fix compile & runtime errors * Fix * Extract http_client requests to separate fn * fix(cargo in ton): added features for tokio crate * refactor(ton-provider): updated seqno and mc_seqno types to u32 for compatibility * feat(relayer and base/contract-sync):Added additional logging for message sync and index metrics * feat(chains/hyperlane-ton):added logging * feat(run-locally/ton): some updates in config * FIx env vars * feat(run_locally/ton): some updates in generate config * Refactor * address update * updated address * FIx * updated address * fix(TonMailbox): fix problems with get_recipient_ism * feat(TonMailbox): updated the delivered method and error handling * Refactor h256 to ton_address + add ::new() fn * fix(ton-igp): added handling of the case when the interval starts or ends at 0 * feat(run-get-method): stack in run_get_method update * Fix min block index * feat(multisig): update parser in multisig ton * Fix * Always ret 1 from get_merkle_leaf_id_by_message_id * fix: update address() method * fix: updated address() method * fix: update address() method * feat(ton-validator_announce): added default implementation for announce_tokens_needed * Fix ton signer * added a stub for process_estimate_costs and updated the validator_announce address in run-locally ton * update parsing dictionary in validator announce, added a plug for the merkle tree * feat(ton-provider): added default implementation for RunGetMethodResponse * feat(merkle-tree-hok): init implementation merkle_tree_hook * refactor(ton-igp): updates error handling and made the parsing of events into a separate function * refactor(hyperlane-ton): Error handlers have been updated and a small decomposition has been done * refactor(hyperlane-ton/multisig_ism): a little refactoring * refactor(chains/hyperlane-ton): removed expect and a small refactoring * refactor: removed Result<> in the ton_address_to_h256 function * Refactoring * refactor(ton/run_locally): some refactoring * resolve conflicts in ton merkle-hook * feat(ton/run_locally): update run locally for Ton * Remove unused code * FIx warnings & compile errors * Rename local ton chains * Uncomment original code * refactor(ton/types): got rid of the wrapper over the hyperlane message * refactor(TonSigner): migrate from DebugWalletVersion to native WalletVersion and update tonlib_core version * feat(ton-provider): init implemented get_block_by_height * Add ton contracts * Remove package-lock * Uncomment cosmos code * Fix grpc.rs * refactor(ton-mailbox): streamline flow and improve error handling * refactor(ton-mailbox): refactor Mailbox flow * refactor(ton-mailbox): TonMailbox refactor flow --------- Co-authored-by: NikitaM <[email protected]>

changeset-bot · 2024-12-11T13:29:27Z

⚠️ No Changeset found

Latest commit: a39cb06

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

ameten · 2024-12-12T13:41:09Z

rust/main/Cargo.lock

When I build the project from this branch, file Cargo.lock gets updated. I wonder if the latest version of Cargo.lock was committed with this change?

ameten · 2024-12-12T13:45:52Z

altvm_contracts/ton/README.md

+
+### Build
+
+`npx blueprint build` or `yarn blueprint build`


I tested yarn blueprint build from altvm_contracts/ton dir. I have got the following error:

Usage Error: The nearest package directory (<path_to_repo>/hyperlane-monorepo/altvm_contracts/ton) doesn't seem to be part of the project declared in <path_to_repo>/hyperlane-monorepo. - If <path_to_repo>/hyperlane-monorepo isn't intended to be a project, remove any yarn.lock and/or package.json file there. - If <path_to_repo>/hyperlane-monorepo is intended to be a project, it might be that you forgot to list altvm_contracts/ton in its workspace configuration. - Finally, if <path_to_repo>/hyperlane-monorepo is fine and you intend altvm_contracts/ton to be treated as a completely separate project (not even a workspace), create an empty yarn.lock file in it. $ yarn run [--inspect] [--inspect-brk] [-T,--top-level] [-B,--binaries-only] [--require #0] <scriptName> ...

ameten · 2024-12-12T13:47:21Z

altvm_contracts/ton/contracts/imports/errors.fc

+const int error::unknown_op = 0xffff;
+const int error::wrong_mailbox_version = 100;


I wonder why one error constant is in hex while others are in dec. Is it a convention?

ameten · 2024-12-12T16:13:08Z

Could you please run the following command and fix the issues it reports?

cargo clippy -- -D warnings

ameten · 2024-12-12T16:17:16Z

rust/main/agents/relayer/src/msg/metadata/base.rs

    pub async fn get_merkle_leaf_id_by_message_id(&self, message_id: H256) -> Result<Option<u32>> {
-        let merkle_leaf = self
+        // FIXME It's a stub for hyperlane-ton e2e test debugging.
+        // Remove when debugging is finished.
+        let _merkle_leaf = self
            .db
            .retrieve_merkle_leaf_index_by_message_id(&message_id)?;
-        Ok(merkle_leaf)
+        Ok(Some(1))
+        //Ok(merkle_leaf)


I wonder if this change was intentionally committed?

ameten · 2024-12-12T16:19:50Z

I see quite a lot of logging added for debug purposes, but with info log level. I wonder if the plan is not remove them?

ameten · 2024-12-12T16:23:00Z

rust/main/chains/hyperlane-ton/Cargo.toml

+url = { workspace = true }
+pretty_env_logger = {workspace = true}
+base64 = {workspace = true}
+tracing = {workspace = true}
+num-bigint = {workspace = true}


Could we please use .workspace = true format if we don't need to mention features?

ameten · 2024-12-12T16:23:33Z

rust/main/chains/hyperlane-ton/Cargo.toml

+tokio = {workspace = true, features = ["full"]}
+
+hyperlane-core = { path = "../../hyperlane-core", features = ["async", "agent"] }
+tonlib-core = "0.22.2"


We can specify the version in workspace and use .workspace = true here.

ameten · 2024-12-12T16:24:56Z

rust/main/chains/hyperlane-ton/src/.gitignore

+runtime_test.rs
+main.rs
+mailbox_test.rs
+ism_test.rs
+announce_test.rs


I wonder how these files are generated and why we would like to avoid committing them into source control?

ameten · 2024-12-12T16:25:29Z

rust/main/chains/hyperlane-ton/src/client/fallback.rs

I wonder if we need this empty file?

ameten · 2024-12-12T16:32:15Z

rust/main/chains/hyperlane-ton/src/client/provider.rs

+use async_trait::async_trait;
+use derive_new::new;
+use log::{debug, info, warn};
+use std::error::Error;
+use std::str::FromStr;
+
+use hyperlane_core::{
+    h512_to_bytes, BlockInfo, ChainCommunicationError, ChainInfo, ChainResult, FixedPointNumber,
+    HyperlaneChain, HyperlaneCustomErrorWrapper, HyperlaneDomain, HyperlaneProvider, TxOutcome,
+    TxnInfo, TxnReceiptInfo, H256, H512, U256,
+};
+use reqwest::{Client, Response};
+use serde_json::{json, Value};
+use tokio::time::sleep;
+
+use crate::client::error::CustomHyperlaneError;
+use crate::run_get_method::StackItem;
+use crate::{
+    trait_builder::TonConnectionConf,
+    traits::ton_api_center::TonApiCenter,
+    types::{
+        account_state::AccountStateResponse, block_response::BlockResponse,
+        message::MessageResponse, message::SendMessageResponse,
+        run_get_method::RunGetMethodResponse, transaction::TransactionResponse,
+        wallet_state::WalletStatesResponse,
+    },
+    utils::conversion::ConversionUtils,
+};
+use tonlib_core::TonAddress;
+use url::Url;


Could we please adopt the following format for imports?

imports from std sorted alphabetically followed by an empty line

imports from external crates sorted alphabetically (async_trait, reqwest, tokio, url, tonlib_core) followed by an empty line

imports from hyperlane crates sorted alphabetically (hyperlane_core) followed by an empty line

imports from local crate (crate::) followed by an empty line

imports from parent crate (super::) followed by an empty line

Could we please adopt it in all source code files?

ameten · 2024-12-12T16:43:03Z

rust/main/chains/hyperlane-ton/src/client/provider.rs

+        let block = response.blocks.first().ok_or_else(|| {
+            tracing::warn!("No blocks found in the response: {:?}", response);
+            ChainCommunicationError::CustomError("No blocks found in the response".to_string())
+        })?;


We have quite a generic pattern on how define chain-specific errors and convert it into ChainCommunicationError. Could you please use it throughout the source code?

See example here: https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/rust/main/chains/hyperlane-sealevel/src/error.rs

ameten · 2024-12-12T16:44:05Z

rust/main/chains/hyperlane-ton/src/client/provider.rs

+            .get_blocks(
+                -1, // Masterchain
+                None,
+                Some(_height),


Since parameter _height is used, it should not have leading underscore in its name.

ameten · 2024-12-12T16:45:01Z

rust/main/chains/hyperlane-ton/src/client/provider.rs

+        })?;
+
+        Ok(BlockInfo {
+            hash: H256::zero(),


I wonder why this field is not populated?

ameten · 2024-12-12T16:59:14Z

rust/main/chains/hyperlane-ton/src/client/retrying.rs

I wonder why we need this empty file?

ameten · 2024-12-12T17:27:43Z

rust/main/utils/run-locally/src/ton/mod.rs

@@ -0,0 +1,279 @@
+#![allow(dead_code)] // TODO: `rustc` 1.80.1 clippy issue


Since it is a new change, could you please remove this instruction and make sure this file does not contribute into clippy errors and warnings?

ameten · 2024-12-12T17:32:02Z

rust/main/hyperlane-base/src/contract_sync/cursors/mod.rs

@@ -60,6 +62,7 @@ impl Indexable for MerkleTreeInsertion {
            HyperlaneDomainProtocol::Fuel => todo!(),
            HyperlaneDomainProtocol::Sealevel => CursorType::SequenceAware,
            HyperlaneDomainProtocol::Cosmos => CursorType::SequenceAware,
+            HyperlaneDomainProtocol::Ton => CursorType::SequenceAware, // need check


Merkle tree insertion should be SequenceAware as it is. So, comment need check can be removed.

ameten · 2024-12-13T14:13:28Z

rust/main/chains/hyperlane-ton/src/signer/signer.rs

+    }
+    pub fn from_mnemonic(


A new line before next method definition:

Suggested change

}

pub fn from_mnemonic(

}

pub fn from_mnemonic(

ameten · 2024-12-13T14:30:23Z

rust/main/chains/hyperlane-ton/src/types/message.rs

+#[derive(Debug, Deserialize)]
+pub struct MessageState {
+    pub body: String,
+    pub decoded: Option<DecodedMessage>,
+    pub hash: String,
+}
+
+#[derive(Debug, Deserialize)]
+pub struct MessageContent {
+    pub body: String,
+    pub decoded: Option<DecodedMessage>,
+    pub hash: String,
+}


I wonder what the difference is between these two struct?

ameten · 2024-12-13T14:49:25Z

rust/main/chains/hyperlane-ton/src/utils/conversion.rs

It will be great to cover all the conversions with tests.

ameten · 2024-12-13T14:50:10Z

rust/main/config/mainnet_config.json

+        "symbol": "TON",
+        "decimals": 9
+      },
+      "chainId": 0,


I suppose we need to choose chainId and domainId for TON.

ameten · 2024-12-13T14:59:17Z

rust/main/hyperlane-base/src/settings/signers.rs

 use tracing::instrument;

 use super::aws_credentials::AwsChainCredentialsProvider;
 use crate::types::utils;

 /// Signer types
-#[derive(Default, Debug, Clone)]
+#[derive(Default, Debug, Clone)] /////


I wonder why we need the comment in the end of the line?

ameten · 2024-12-13T15:06:12Z

rust/main/utils/run-locally/src/ton/deploy.rs

I wonder if we need this commented out source code.

ameten · 2024-12-13T16:00:50Z

rust/main/chains/hyperlane-ton/src/contracts/mailbox.rs

+            self.mailbox
+                .provider
+                .get_blocks(
+                    -1,          // masterchain (workchain = -1)
+                    None,        // shard
+                    None,        // Block block seqno
+                    Some(block), // Masterchain block seqno
+                    None,
+                    None,
+                    None,
+                    None,
+                    None,
+                    None,
+                    None,
+                )
+                .await
+                .map_err(|e| {
+                    ChainCommunicationError::CustomError(format!(
+                        "Failed to fetch block info for block {}: {:?}",
+                        block, e
+                    ))
+                })?
+                .blocks


I suppose that it will be useful to extract this call into a separate method and use it here as well as in the implementation of Indexer traits for other events like interchain gas payments.

ameten · 2024-12-13T16:02:11Z

rust/main/chains/hyperlane-ton/src/contracts/mailbox.rs

+            self.mailbox
+                .provider
+                .get_blocks(
+                    -1,          // masterchain (workchain = -1)


It will be great to move this literal to a constant which can be used everywhere.

Lapo4kaKek and others added 2 commits December 11, 2024 16:09

Merge remote-tracking branch 'upstream/main'

a39cb06

Keshoid requested review from tkporter, daniel-savu, yorhodes and jmrossy as code owners December 11, 2024 13:29

Keshoid changed the title ~~Add TON blockchain support~~ feat: add TON blockchain support Dec 11, 2024

ameten reviewed Dec 12, 2024

View reviewed changes

rust/main/chains/hyperlane-ton/src/client/fallback.rs

Copy link

Contributor

ameten Dec 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we need this empty file?

ameten reviewed Dec 12, 2024

View reviewed changes

rust/main/chains/hyperlane-ton/src/client/retrying.rs

Copy link

Contributor

ameten Dec 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder why we need this empty file?

ameten reviewed Dec 12, 2024

View reviewed changes

ameten reviewed Dec 13, 2024

View reviewed changes

rust/main/chains/hyperlane-ton/src/utils/conversion.rs

Copy link

Contributor

ameten Dec 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will be great to cover all the conversions with tests.

ameten reviewed Dec 13, 2024

View reviewed changes

rust/main/utils/run-locally/src/ton/deploy.rs

Copy link

Contributor

ameten Dec 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we need this commented out source code.

ameten reviewed Dec 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add TON blockchain support #4985

feat: add TON blockchain support #4985

Keshoid commented Dec 11, 2024 •

edited

Loading

changeset-bot bot commented Dec 11, 2024

ameten Dec 12, 2024

ameten Dec 12, 2024

ameten Dec 12, 2024

ameten commented Dec 12, 2024

ameten Dec 12, 2024

ameten commented Dec 12, 2024

ameten Dec 12, 2024

ameten Dec 12, 2024

ameten Dec 12, 2024

ameten Dec 12, 2024

ameten Dec 12, 2024

ameten Dec 12, 2024

ameten Dec 12, 2024

ameten Dec 12, 2024

ameten Dec 12, 2024

ameten Dec 12, 2024

ameten Dec 12, 2024

ameten Dec 13, 2024

ameten Dec 13, 2024

ameten Dec 13, 2024

ameten Dec 13, 2024

ameten Dec 13, 2024

ameten Dec 13, 2024

ameten Dec 13, 2024 •

edited

Loading

ameten Dec 13, 2024

		const int error::unknown_op = 0xffff;
		const int error::wrong_mailbox_version = 100;

		@@ -0,0 +1,279 @@
		#![allow(dead_code)] // TODO: `rustc` 1.80.1 clippy issue

feat: add TON blockchain support #4985

Are you sure you want to change the base?

feat: add TON blockchain support #4985

Conversation

Keshoid commented Dec 11, 2024 • edited Loading

Description

Drive-by changes

Related issues

Backward compatibility

Testing

changeset-bot bot commented Dec 11, 2024

⚠️ No Changeset found

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ameten commented Dec 12, 2024

Choose a reason for hiding this comment

ameten commented Dec 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ameten Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Keshoid commented Dec 11, 2024 •

edited

Loading

ameten Dec 13, 2024 •

edited

Loading