Request for Docs on how a Subnet is built from Scratch without the Avalanche Network Runner

[h4ck3rk3y]

Is there a document that you have that explains the process of subnet creation? All articles I can find on the internet use the avalanche network runner. I have seen some articles on ChainStack and others that refer to deprecated calls like keystore.CreateUser or platform.CreateSubnet

I have been going through the avalanche-cli & the network runner and so far I have figured out the following

1. NodeIDs can be precomputed using a SHA160(SHA256(staking_key, staking_cert)) and prefixing NodeID
2. SubnetIDs for the local subnet are pre-computed and when you create a new subnet it gets assigned one of the existing subnet ids that are already whitelisted using --tracked-subnets. I am happy with using pre-computed SubnetIDs but I am curious if you can link me to the process of creating these SubnetIDs
3. When a subnet is deployed, a plugin needs to be added to the plugins-dir and the name of the plugin needs to be the same as the VMID; the VMID can be computed from the ChainID using anrutils.VMID. If I just want to support subnet-evm; I can just compile that and rename that to VMID and put it in plugins-dir with the right permission
4. After the plugin is put in the right area a call to CreateBlockChains is issued & if that goes well I can get the new BlockchainID by running clusterInfo and getting the chainId for the given subnetId

What I am still stuck on

1. Does avalanchego need to be restarted after a plugin is added(assuming tracked-subnets is properly set)? Or will it kickstart any plugins in plugin_dir automatically? Based on what I see with avalanche-cli it seems like the avalanchego binary doesn't get restarted
2. Can you explain more what the inner workings of the CreateBlockChains function are? What does it do and how does it do it?

Thanks @aaronbuchwald for suggesting I create an issue here!

[aaronbuchwald]

Thanks for creating here!

1. Yup, the NodeID is derived from the staking key

2. The SubnetID is the hash of the transaction that created the Subnet. This means that if you have a wallet with 1 UTXO on the P-Chain that will fund a series of transactions, you can generate those transactions offline without issuing them and get a deterministic sequence of transactions, which will tell you the SubnetID and any BlockchainIDs that you may want to create for a test. We don't currently have example code for generating them offline, but the AvalancheGo P-Chain wallet is probably the best starting point: https://github.com/ava-labs/avalanchego/blob/master/wallet/chain/p/wallet.go#L26.

3. A subnet runs a set of blockchains (typically only one), and the node needs to have the VM necessary to run these blockchains available in order to validate the subnet. If you are just using Subnet-EVM, you should compile it and place it in the plugin directory before launching AvalancheGo. When you launch any subnet that runs Subnet-EVM, it will use the binary that is already in the plugin directory. If for some reason, you want to validate a new subnet that you don't have the plugin binary for when you started the node, you will need to restart to make it available to the node (good feature request would be hot reloading)

4. Yup

5. No, you need to restart it when either adding a new binary to plugin dir or when adding a new subnetID to tracked-subnets

6. @felipemadero could you provide context on what CreateBlockchains is doing?

[h4ck3rk3y]

@aaronbuchwald that was super helpful!

2, 3 & 5 - I can try using the pre-computed SubnetIDs that the CLI & runner run with to get a v1 out and that can just be with subnet-evm for now; as I'll set tracked-subnets before hand I wouldn't have to issue restarts just like the avalanche-cli(based on my read of the code & behavior)

I'll read through the wallet code later and see if (or maybe there are docs by then) to generate the SubnetID

Will wait for @felipemadero to elaborate on what the CreateBlockchains call does.

[StephenButtolph]

Node configuration

It is not currently possible to modify --track-subnets on the fly. However it is possible to add new VMs that have been populated in the plugins folder by using admin.LoadVMs. That being said, you will need to have loaded the VM before the CreateBlockchainTx has been issued. Otherwise the chain will have been attempted to be started which will error with a VM not found error.

Wallet issuance

The full flow for launching a permissioned blockchain is:

1. Issue a CreateSubnetTx. This will generate a new permissioned subnet on the P-chain. The SubnetID is just the TxID of this operation.
2. Issue a CreateBlockchainTx. This should reference the newly created subnet. The BlockchaibID is just the TxID of this operation.
3. Issue AddSubnetValidatorTxs for all the subnet validators that are desired. (Note: subnet validators are required to also be validators of the primary network during the full duration of their subnet staking period.)

If the goal is to launch a permissionless blockchain then:

1. Launch a permissioned blockchain
2. Issue a TransformSubnetTx. This will "lock" the subnet and prevent the creation of any new blockchains on the subnet and will prevent the issuance of any more permissioned subnet validators (via the AddSubnetValidatorTx). This requires specifying an asset that was minted on the X-chain as the staking token (C-chain minting to come in a future release).
3. Now to add new validators one would need to use the AddPermissionlessValidatorTx specifying the subnetID.

If you want to pre-generate transactions, the easiest way will be to specify your own Client implementation here. Specifically, you'll want to override the IssueTx call to keep the bytes that you need. If you want to make a PR to avalanchego to make this a more minimal interface (It only uses 2 of the many functions on the interface) then that would be a great change (but we would probably want to the same change to the X wallet.)

[h4ck3rk3y]

Hey Stephen! That's really thorough.

Node Configuration

This makes sense! To makes sure I understand correctly; If I end up using a pre computed list of subnet ids; all I have to do is put the correctly named vm into the plugins folder and then issue an admin.LoadVMs call. This won't work if the subnet isn't tracked in the tracked-subnets list?

Wallet issuance

This makes sense as well. I'll go further deep into it as I start implementing this and I will probably have more questions when I do so. In the meantime my understanding of the conversations so far; The sequence of operations is the following (assuming the subnet-evm) -

1. I launch a private testnet and have some pre-funded accounts in that testnet. Make sure the nodes are validator nodes & are fully bootstrapped. I can try to use the default genesis.json(but I will have to copy the signer/staker keys) of the avalanche-network-runner or use this to generate a valid genesis JSON
2. I transfer some AVAX from the C chain address to the pChain address of one of the genesis addresses
3. Using the address from step 2 I create a p.Wallet
4. Using the wallet created in Step 3 I issue a CreateSubnetTx
5. I place the rightly named(VMID) binary of the chosen VM in the nodes of the testnet in the plugins folder that is tracked by the Avalanche Go Client. The VMID of the binary just has to match the VMID supplied and can be foobar even
6. I then restart the avalanchego nodes with the right id in --tracked-subnets & on restart the node will pickup the binaries in the plugins dir
7. I issue a CreateBlockchainTx from the same p.Wallet created earlier; with the right VM ID, subnet ID, chainName (anything human readable). I'm not sure what fxIDs are and perhaps I can leave them empty for now? I also have to pass a genesis.json for the state of the new chain thats being created. I can probably use this one here with right addresses for initial allocation. At this point the chain is running and the nodes that have the right vm & --tracked-subnets are already validating the subnet.

At this point I have a choice I can either use AddSubnetValidatorTxs or launch a permission less blockchain and then use AddPermissionLessValidatorTx (based on my understanding this is the path the CLI takes). I could stop too as I have a chain with some validators.

For permission less(elastic?) Blockchain

8. I create a new asset on the X-Chain. I can perhaps take inspiration from here to get some of the values right
9. I transfer some of that asset from the XChain to the P-Chain
10. I issue a TransformSubnetTx this takes in a lot of arguments. Are these sensible defaults?
11. I issue a AddPermissionlessValidatorTx. The signer is empty. I can take inspiration from here. These validators need to have tracked-subnets and the VM set too.

For permissioned blockchain to add new validators

12. I issue a AddSubnetValidatorTxs; with a reasonable weight & the right subnet id. These validators need to have tracked-subnets and the VM set too.

Some questions that I might not have asked above

13. avalanche subnet deploy stops at step 7? And at step 7 I have a fully running subnet with enough validators? I need to make sure that I start with at least 5 nodes when I start it all.

I'll have to decide whether I go the pre-generated transactions route or the route above. I am biasing towards generate on runtime route as I have dived deep enough into it I feel. If I do create a minimal interface I'll make sure I PR back to the repository.

Thanks for being so patient with me. I have been programming for years but I am fairly new to blockchain dev & Avalanche.

[felipemadero]

Hi Gyanendra

Regarding the original figured-out point 2: pre-computed subnet ids are used mainly for CLI implementation (for fast local subnet deploy). They belong to a network snapshot that comes packaged into CLI (https://github.com/ava-labs/avalanche-cli/blob/main/assets/bootstrapSnapshot.tar.gz), and are not available in a network started from scratch without using such snapshot.

I think there was a confusion on this point, as it is different what the CLI does (use a subnet id for a subnet that was already created and is contained in a network snapshot) from what a custom wallet enables to do (pre generate a subnet id without creating a real subnet, by simulating UTXO consumption and construct the tx without issuing it to the blockchain - as aaron mention in his comment to 2, and stephen extends in relation to pre-generating transactions)

Regarding the original questions:

An avalanchego network needs to be restarted if:

• a change is made to tracked subnets
• a change is made to subnet configuration files
• a change is made to blockchain configuration files
• a cuange is made to so called network upgrade files

In the specific case of adding a new vm, no need to restart as stephen already told, just need
to call LoadVMs as network runner does in https://github.com/ava-labs/avalanche-network-runner/blob/80262bb1beafb633d7485dd8b57a9680ae8de9cc/local/blockchain.go#L307 (and I recommend following that code of installCustomChains to try to go deeper into what is needed to be done to set up a subnet)

2

The CreateBlockchains function of ANR creates the requested blockchains, each of one defined by
the following blockchain parameters:

vm name
genesis
optional chain config (and optional per node specific chain config)
optional network upgrade
optional blockchain alias
optional subnet id
optional subnet parameters
    optional list of nodes that will participate in the subnet
    optional subnet config

Where:

• the function call goes down into the previously referred installCustomChains() which does all the work
• subnet id/parameters refer to a blockchain subnet, a set of nodes that will serve as validators for a given
  blockchain

If the subnet id is defined, then a blockchain is created (see createBlockchainsTxs) under that previously generated subnet
(a subnet could had been previously generated by using network runner's CreateSubnets)

If the subnet id is not defined, a new subnet will we generated, will all network nodes as participants and no subnet config as default, unless optional subnet parameters are passed in. In the special case of receiving an optional list of nodes as participants where some of the nodes do not exist, first the new nodes will be added to the network, and then used as participants for the desired subnet. After all of that, the blockchain is created over the new subnet.

If blockchain config files, or network upgrade files are given, then they are first generared in the right locations, and then the network will be restarted previously to create the blockchains.

---

Regarding your second comment in the thread, yup you can try to use the network snapshot that comes with the CLI
by calling LoadSnapshot on network runner, and then you will have those predefined subnet ids available and can
call network runner's CreateBlockchains to create the subnets without restart (or issue a CreateBlockchainTx as Stephen told).

Regarding using a wallet to pre generate subnet ids: I think you probably don't want to first pre generate a subnet id by using a wallet, and you just want to create a subnet from scratch.

[felipemadero]

Hey Stephen! That's really thorough.

Node Configuration

This makes sense! To makes sure I understand correctly; If I end up using a pre computed list of subnet ids; all I have to do is put the correctly named vm into the plugins folder and then issue an admin.LoadVMs call. This won't work if the subnet isn't tracked in the tracked-subnets list?

You can't just use a pre computed list of subnet ids, you can either use a network snapshot with already have subnet ids and tracked-subnets defined, or you need to do all the steps: create a subnet, restart the network with updated tracked-subnets, and then create the blockchain.
LoadVMs will work to load the VMs on plugin dirs, but you need the subnet to be tracked for the vms start working.

Wallet issuance

This makes sense as well. I'll go further deep into it as I start implementing this and I will probably have more questions when I do so. In the meantime my understanding of the conversations so far; The sequence of operations is the following (assuming the subnet-evm) -

1. I launch a private testnet and have some pre-funded accounts in that testnet. Make sure the nodes are validator nodes & are fully bootstrapped. I can try to use the default genesis.json(but I will have to copy the signer/staker keys) of the avalanche-network-runner or use this to generate a valid genesis JSON
2. I transfer some AVAX from the C chain address to the pChain address of one of the genesis addresses

You have funds already available to a local network on PChain by using the EWOQ address as in network runner code

3. Using the address from step 2 I create a p.Wallet
4. Using the wallet created in Step 3 I issue a CreateSubnetTx
5. I place the rightly named(VMID) binary of the chosen VM in the nodes of the testnet in the plugins folder that is tracked by the Avalanche Go Client. The VMID of the binary just has to match the VMID supplied and can be foobar even
6. I then restart the avalanchego nodes with the right id in --tracked-subnets & on restart the node will pickup the binaries in the plugins dir
7. I issue a CreateBlockchainTx from the same p.Wallet created earlier; with the right VM ID, subnet ID, chainName (anything human readable). I'm not sure what fxIDs are and perhaps I can leave them empty for now? I also have to pass a genesis.json for the state of the new chain thats being created. I can probably use this one here with right addresses for initial allocation. At this point the chain is running and the nodes that have the right vm & --tracked-subnets are already validating the subnet.

You can leave fxIDs empty for now. those are for feature extensions to be run by the vm. For genesis you can use a genesis created by CLI when locally deploying a subnet, to be found on ~/.avalanche-cli/subnets/SUBNET_NAME/genesis.json

At this point I have a choice I can either use AddSubnetValidatorTxs or launch a permission less blockchain and then use AddPermissionLessValidatorTx (based on my understanding this is the path the CLI takes). I could stop too as I have a chain with some validators.

You will not have any subnet validators until you issue an AddSubnetValidatorTx and you did not issue any in the previous points. It is not enough to set tracked-subnets, you need to issue an add subnet validator tx for each node you want to be a validator of the subnet.

For permission less(elastic?) Blockchain

Yup elastic

8. I create a new asset on the X-Chain. I can perhaps take inspiration from here to get some of the values right
9. I transfer some of that asset from the XChain to the P-Chain
10. I issue a TransformSubnetTx this takes in a lot of arguments. Are these sensible defaults?
11. I issue a AddPermissionlessValidatorTx. The signer is empty. I can take inspiration from here. These validators need to have tracked-subnets and the VM set too.

For permissioned blockchain to add new validators

12. I issue a AddSubnetValidatorTxs; with a reasonable weight & the right subnet id. These validators need to have tracked-subnets and the VM set too.

Yon can also first issue an add subnet validator tx for those validators, and then restart then with proper tracked subnets and vms.

Some questions that I might not have asked above

13. avalanche subnet deploy stops at step 7? And at step 7 I have a fully running subnet with enough validators? I need to make sure that I start with at least 5 nodes when I start it all.

avalanche subnet deploy stops at step 7, but it previously added all 5 default local network nodes as subnet validators. it does that even before restarting the network.

I'll have to decide whether I go the pre-generated transactions route or the route above. I am biasing towards generate on runtime route as I have dived deep enough into it I feel. If I do create a minimal interface I'll make sure I PR back to the repository.

Probably you may want to just follow the route you described.

Thanks for being so patient with me. I have been programming for years but I am fairly new to blockchain dev & Avalanche.

[h4ck3rk3y]

Thank you so much @felipemadero

Your comments are very clear, insightful and super helpful. I think I have a fairly good understanding of how subnets launch now.

I'll try to have an n Node subnet that runs on Docker/K8S out next week. I'll write about my learnings in the process. I'll reach out you folks in case I run into any troubles there but for now I feel equipped.

[h4ck3rk3y]

EWOQ addresses in the above post refer to the addresses here - https://github.com/ava-labs/avalanchego/blob/master/genesis/genesis_local.go#L18-L21 which come prefunded with a lot of tokens

Update from conversation with Aaron on Wednesday - the VMID is a cb58 hash and can be computed using either of

1. https://github.com/ava-labs/avalanchego/blob/f7307d5fd015e75d6b40d26713c5fb6576dc6ffb/ids/id.go#L37-L44
2. https://github.com/ava-labs/avalanche-network-runner/blob/faa8894e8cfa768279cde6809eb6ca911e18cda3/utils/utils.go#L81-L89

[h4ck3rk3y]

Hey!

@aaronbuchwald Thanks again for your time last week :)

I have added a bunch of stuff to the avalanche-package since and now this launches subnets! You can launch subnets with how many nodes you want using {"node_count": N}; where N is the number of nodes you want. Otherwise it launches 5 nodes by default

FLUP on speed

The time for the docker run of avalanchego to expose 9651 and Kurtosis is very similar; around 12s on my machine.

TODO

I am tracking a lot of improvements that need to be made

1. Speed up the boot process - We're currently doing things sequentially that we could be doing in parallel; looking at this internally (product & inside the package)
2. Parametrize the ChainId for the subnet being created & the network id of the network being created (currently hardcoded to 13123, 1337 respectively
3. Support permissionless subnets
4. Support minimum resource requirements in K8S (X CPU Y Memory)
5. Parametrize chain configuration, subnet configuration further (maybe you can drop in a dir)
6. For the subnet vm I am just copying the existing VM srEXiWaHuhNyGwPUi444Tu47ZEDwxTWrbQiuD7FmgSAQ6X7Dy on 1.10.1 to the new VMID in the same folder, but I reckon this would be different for 1.10.2; so have to remove this hardcoding and others
7. Toggle to disable subnets completely and just start a normal n node cluster

For now

In the meantime with Kurtosis installed you can give it a spin using

kurtosis run github.com/kurtosis-tech/avalanche-package '{"ephemeral_ports": false, "node_count": 5}'

This would spin up a 5 node cluster but you should be able to put a random number as well. By default this runs against Docker. I can send you instructions for K8S if you want to try that out too!

Note without the "ephemeral_ports": false argument; the containers will get ephemeral ports so the chain-rpc-url in the end will be valid but only inside the enclave and not from the outside world. Working with product to print the public ephemeral port at the end without the ephemeral_ports flag.

Thank you @felipemadero @StephenButtolph for your help too!

[h4ck3rk3y]

Hey! I have added a few more things

1. You can now spin up elastic subnets, permissioned subnets or no subnets
2. I have sped up the boot process and all of them start parallely
3. Internally we are fixing some K8s related things and adding the ability to specify minimum resources

I am running into some weirdness though - up to 20 nodes its fine but if try starting a cluster with any more nodes there's some weirdness - here's the steps i take for permissioned subnets

1. Create genesis
2. Start chain with 30 nodes (kurtosis run . '{"node_count": 30}')
3. Assert nodes are healthy using the health.Health endpoint
4. Copy over the right binary with the right vmId
5. Run wallet transactions (three below)
6. Issue a CreateSubnetTx
7. Issue a CreateBlockchainTx
8. Issue AddSubnetValidatorTxs for all nodes - this part fails around the 16th/17th node for the 30 node case
9. Restart all nodes with --tracked-subnets
10. Assert health again

With 30 nodes it(main.log) was initially failing with this when I did the vm installation after the after the AddSubnetValidatorTxs but then I moved it to before the wallet transactions and that went away. Now it fails with this

From the main.log

[06-09|12:03:08.001] INFO chains/manager.go:319 creating chain {"subnetID": "2aPkmYuVqCBMGnyDBUsKExmD6SZa7JSH8sw8ekcsQ41nTj2qrp", "chainID": "NAL3xKipcGFTxKrzQZAtXStmZqN7QCqh2AZ88j9iZqmvtgtUA", "vmID": "tGBrM7iZGgNZvqPiwD9oD716rVRR9PiB6BFuG3ot3SP54ie8K"}
[06-09|12:03:13.322] ERROR chains/manager.go:349 error creating chain {"subnetID": "2aPkmYuVqCBMGnyDBUsKExmD6SZa7JSH8sw8ekcsQ41nTj2qrp", "chainID": "NAL3xKipcGFTxKrzQZAtXStmZqN7QCqh2AZ88j9iZqmvtgtUA", "chainAlias": "NAL3xKipcGFTxKrzQZAtXStmZqN7QCqh2AZ88j9iZqmvtgtUA", "vmID": "tGBrM7iZGgNZvqPiwD9oD716rVRR9PiB6BFuG3ot3SP54ie8K", "error": "error while creating vm: handshake failed: vm process not found"}

the stack trace says not committed in the wallet

error occurred while adding validators: an error occurred while adding node '16' as validator: not committed

Do you know what might be going on here @aaronbuchwald @felipemadero ? This is running against avaplatform/avalanchego:v1.10.1-Subnet-EVM-master on Docker on my M1 Mac

[h4ck3rk3y]

Update - Alright I see where the error was coming from my logic here

While doing the issue validator transaction I had hardcoded a start time delay of 1 minute from the beginning of the loop. I think we'd run a transaction with a start timestamp in the past and that would fail.

If I setup startTime per node instead of having a global one that I use; I don't get the error anymore. Still having some troubles getting things into a healthy state as I run into

ID": "NAL3xKipcGFTxKrzQZAtXStmZqN7QCqh2AZ88j9iZqmvtgtUA", "chainAlias": "NAL3xKipcGFTxKrzQZAtXStmZqN7QCqh2AZ88j9iZqmvtgtUA", "vmID": "tGBrM7iZGgNZvqPiwD9oD716rVRR9PiB6BFuG3ot3SP54ie8K", "error": "error while creating vm: handshake failed: vm process not found"}

per node on restart but that seems different from the above issue. Will investigate. Maybe I should do add validator, restart validator per validator instead.

[h4ck3rk3y]

Alright! I was able to fix what was broken and the 100 node demo on k8s runs fine :) You will be able to specify how much memory + cpu something runs with

Here's the demo https://www.loom.com/share/f4e8bfeb4c304903b9b5f8736df0591d

This doc should get you started - https://docs.kurtosis.com/k8s/

[laviniat1996]

@ashucoder9 Hi, can you please provide an update on this? Wondering if it can be closed.