Hoku localnet node

[sanderpick]

We've been relying on the cargo make tasks for local hoku development. This has been really useful for debugging on a multi-validator network. However, it's still quite cumbersome, requiring a number of steps and solid knowledge of docker to get comfortable. Further, we also need an easy way for "huku developers" (people building on hoku) to easily spin up a local node. We need a "localnet" that is to hoku as hardhat/anvil is to ethereum.

• Ideally a dev just has to run one command to get everything up and running like hardhat node
• Since we have so many processes, docker-in-docker sounds interesting (https://gopesh3652.medium.com/running-docker-in-docker-dind-a-comprehensive-guide-1fe2e328020)
• But we should first explore creating a new binary crate (or command on adm?) that just spawns threads / tokio tasks for each dependency: comet, fendermint, blobs api, iroh, eth api.

This standalone command would also allow hoku contributors (people working on hoku itself) to easily attach a debugger since docker isn't involved. We could even specify the number of validators like,

hoku node --validators 4

(or similar)

[brunocalza]

But we should first explore creating a new binary crate that just spawns threads / tokio tasks for each dependency: comet, fendermint, blobs api, iroh, eth api.

This would be perfect for Basin S3 tests (https://github.com/hokunet/basin-s3/blob/main/lib/basin_s3/tests/it_basin.rs). Right now it's kind of pita to write/run tests because I have to check out the IPC repo and deploy a network using the scripts, and this approach makes it impossible to run the tests in Github jobs.

[sanderpick]

@joewagner can you please give a status update on this one?

[joewagner]

It's currently a wip in joe/local It has been difficult, but I'm making progress.
[EDIT:]
Sorry, that isn't a very helpful update. I'll opened a draft PR here: recallnet/ipc#354
It isn't ready for a thorough review, but I will add comments and discussion on the status in the PR.

[stbrody]

Not required for audit

[sanderpick]

It's currently a wip in joe/local It has been difficult, but I'm making progress. [EDIT:] Sorry, that isn't a very helpful update. I'll opened a draft PR here: hokunet/ipc#354 It isn't ready for a thorough review, but I will add comments and discussion on the status in the PR.

great, ty! if helpful, feel free to call out the main implementation challenges / blockers... happy to discuss

[stbrody]

How important is this docker-less localnet actually? You can already run the devnet without docker. But for most external users, as well as for all our testing, CI/CD, and the testnet and mainnet nodes we run ourselves, we are using docker everywhere. With a mind to keeping scope down, this feels like relatively low-value add over just making the docker setup really robust, since that one is going to be used in a ton more places.

@sanderpick do you think this needs to be a priority right now, or can we hold off on this for a while?

[sanderpick]

there are two goals here:

• primary: give external developers an experience like hardhat / anvil for local development. i don't think docker is acceptable here--a full docker setup is quite complex
• secondary: a tool for CI/CD tests: spin an entire network and run tests against it with the SDK. we don't do this at all right now

@joewagner how are you doing here? weren't you going to share a demo?

[stbrody]

For CI/CD, we are planning to use docker for this. The docker scripts are the most built-out and well-tested right now, as they are what we are using for our testnet deploys.

For the external developers, I'd like to understand better why docker isn't acceptable for that? It's a pretty standard tool that is very well suited towards reliable deploys that are well-isolated from how the host OS is set up.

In fact, on the call that @rohhan and I had with a potential RPC provider partner a few weeks ago, they said handing them some docker images to run would be pretty ideal for them.

[sanderpick]

For CI/CD, we are planning to use docker for this. The docker scripts are the most built-out and well-tested right now, as they are what we are using for our testnet deploys.

We don't have any pure docker scripts. The localnet uses cargo make and a number of other tools like git, foundry, anvil, and random unix tools like curl.

We could docker-compose a devnet-like setup, but without additional work + some container scripts, it wouldn't have a parent network, relayer, etc.

For the external developers, I'd like to understand better why docker isn't acceptable for that? It's a pretty standard tool that is very well suited towards reliable deploys that are well-isolated from how the host OS is set up. In fact, on the call that @rohhan and I had with a potential RPC provider partner a few weeks ago, they said handing them some docker images to run would be pretty ideal for them.

Are you referring to validators? Docker is fine for them, but I don't think it's a great solution for app developers. I haven't seen a chain that leverages docker for that. The analogy is to hardhat / anvil, not geth / production nodes.

That said, I agree that we should evaluate the priority here. I'm just want to be clear everyone knows what this project is about. It's not for validators / node providers. It's primarily for app developers and secondarily for CI/CD.

It really comes down to how far along @joewagner is. If he's close, I don't want to lose this work because it would make for a pretty great dev experience.

[joewagner]

That said, I agree that we should evaluate the priority here. I'm just want to be clear everyone knows what this project is about. It's not for validators / node providers. It's primarily for app developers and secondarily for CI/CD.

It really comes down to how far along @joewagner is. If he's close, I don't want to lose this work because it would make for a pretty great dev experience.

I keep thinking I'm gonna be done with this, but when I fix one issue I find another. I'm currently stuck because cometbft won't start as a validator. I believe that this is because the fendermint command that converts the genesis file into one that cometbft can use isn't assigning the validator key pair to the one the cometbft validator is using. I think this is something simple, but haven't figured it out yet.

@sanderpick maybe we can sync and I can screen share everything, and you can get a better sense of what's left to do here?
It does seem like it will be a nice tool for app developers and easier CI/CD spin up if we can get it to work.

There is also a possibility that since a lot of the processes (docker or otherwise) are written in rust that this can simplify deployments outside of just local-dev/ci/cd. I'm not convinced this is a good idea, but this would be an avenue to that if it seemed valuable.

[stbrody]

Good call out on the distinction between node operators and app developers. That said, I'm still not convinced that docker isn't a perfectly reasonably solution for app developers, I still think docker is a pretty standard tool that most devs are used to running.

But I don't really have strong opinions here either way, I'm just looking for where we can cut scope and focus our efforts. I know a lot of work is going to happen either way on the docker-based processes, so it feels like it would be nice if we could just use that and not have to maintain another process as well.

CC @smrz2001 @sam701 who are going to be doing a lot of the work on this initiative.

[sam701]

Just to mention there is a hoku-docker-compose repo that sets up a validator node with docker-compose. It is supposed to be a simpler alternative to the ansible scripts. (@joewagner you might find helpful the init part in that repo for your current issue).

In terms of DX I can imagine it is convenient for developers to run native local binaries, so that one doesn't need to build a docker image for each modification while working on let's say fendermint. It's also easier for debugging. Not yet sure how suitable a rust binary for this task; looking forward to the demo 🚀

For CI/CD, integration tests, and final node deployment I find docker approach is more convenient, because multiple components are involved and it is straight forward to wire those components together using containers. The less straight forward part is the configuration, though.

[stbrody]

In terms of DX I can imagine it is convenient for developers to run native local binaries, so that one doesn't need to build a docker image for each modification while working on let's say fendermint

@sam701 I believe you are talking about our team's developers who are building out Recall itself. The discussion here has mostly been about app developers building on Recall

[sanderpick]

@stbrody, true, but the original goal was to cover both internal and external devs:

We've been relying on the cargo make tasks for local hoku development. This has been really useful for debugging on a multi-validator network. However, it's still quite cumbersome, requiring a number of steps and solid knowledge of docker to get comfortable. Further, we also need an easy way for "huku developers" (people building on hoku) to easily spin up a local node. We need a "localnet" that is to hoku as hardhat/anvil is to ethereum.

i know we chatted about this yesterday a bit, but for those following along: make run-localnet, which allows for spinning 3 local nodes with a real parent, takes a long time to boot. a docker-compose version would not be any better, since you're just replacing cargo make orchestration with docker-compose orchestration. you still have to build the images, etc.

[stbrody]

for my own local development on Hoku itself, I just use the devnet (ie running all the processes manually in different terminal windows)

[sanderpick]

that's fair. in many cases, it's important to have a real parent and multiple nodes.