Scripts for Gaiad network with full-mesh IBC connections

[greg-szabo]

Partial solution to #897

Description

A set of scripts with documentation to create an interconnected set of blockchain networks on a local machine using docker.

Start with the README!

---

For contributor use:

• Updated the Unreleased section of CHANGELOG.md with the issue.
• If applicable: Unit tests written, added test to CI.
• Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
• Updated relevant documentation (docs/) and code comments.
• Re-reviewed Files changed in the Github PR explorer.

[brapse]

This is so great! Starting playing with this today but got stuck. Probably something silly but after running make i get pretty far until this error comes up:

docker compose up -d
[+] Running 1/11
 ⠿ Network "full-mesh_default"  Created                                                                                                             3.0s
 ⠿ Container hermes-3-5         Starting                                                                                                            0.2s
 ⠿ Container hermes-2-5         Starting                                                                                                            0.2s
 ⠿ Container hermes-1-5         Starting                                                                                                            0.2s
 ⠿ Container hermes-2-3         Starting                                                                                                            0.2s
 ⠿ Container hermes-3-4         Starting                                                                                                            0.2s
 ⠿ Container hermes-1-2         Starting                                                                                                            0.2s
 ⠿ Container hermes-1-4         Starting                                                                                                            0.2s
 ⠿ Container hermes-1-3         Starting                                                                                                            0.2s
 ⠿ Container hermes-2-4         Starting                                                                                                            0.2s
 ⠿ Container hermes-4-5         Starting                                                                                                            0.2s
The new 'docker compose' command is currently experimental. To provide feedback or request new features please open issues at https://github.com/docker/compose-cli
Error response from daemon: network-scoped alias is supported only for containers in user defined networks
make: *** [relay-up] Error 1

Is this a local configuration thing?
I found This. Maybe we need to switch from host network to soemthing else?

[greg-szabo]

You need to upgrade your Docker CLI to v1.0.14 or later. (Relevant issue here.) This version was released only a month ago, so most probably everyone will need to upgrade.

The Makefile has a comment about this in an obscure place right above the relay-up command. I'll move that into the documentation.

Alternatively, if you just want a quick workaround, replace the docker compose commands in relay-up and relay-down with docker-compose (note the dash), which is the old compose command that didn't have this bug.

[brapse]

This is really great. Went through this but I'm still playing with it. We can probably merge with the upgrade 0.3.0 and then follow up with issues.

[brapse]

Can we update this to 0.3.0?

[colin-axner]

This is wonderful! Looking forward to trying it out!

I'd love to be able to spin up a mesh with chains using different versions of gaia to test IBC and relayer compatibility between them. Testing that a gaia v4 chain can interact with gaia v5 using the same relayer binary

Another wishlist item is using non-gaia docker images, so chains could test out their docker image before upgrading on mainnet, but I think it might be best to let non-gaia chains copy the script and replace its relevant info rather than maintaining scripts for several chains

[greg-szabo]

Thanks for the feedback! There were a lot of good points from the IBC-rs team about this, including non-gaia docker images. I'm not sure yet how we move forward because I find the current configuration fairly inflexible. We have another tool in the works that is much more flexible for developer work and allows you to choose your binaries on a per-node basis. (Issue #902 .)

So, we might continue expanding the configuration here or maybe we end up merging the two tools. But I'll definitely keep this in mind.

Small trick right now: you can do

make config-files
vi docker-compose.gaiad.yml
make network-up setup-relay

and edit the gaiad images that will be used before they are started.

[greg-szabo]

I propose we close this PR without merging. Here's the alternative I have in mind: in PR #928 I propose a different tool that I feel is more beneficial for the users. Here's a quick comparison:

full-mesh-tool:
Advantages:

• robust network setup out of the box
• recreating the exact same environment is built-in because of the pre-defined docker images and configuration

Disadvantages:

• changing the setup is cumbersome: you need to understand make, some shell scripts and possible the docker-compose.yml file format as well as how docker works in general.
  • As the above comment showed keeping the tool up-to-date requires some change management process from the team, even for upgrading hermes
  • Changing unforeseen properties is also cumbersome: there's no easy way to manage differnet binaries for different networks

gm tool:
Advantages:

• Network configuration is configurable through one transparent file: gm.toml
  • adding full nodes is easy
  • changing the binary used on a per-node basis is easy (which also shows that you can plug any type of blockchain binary in there as long as the command-line is compatible with gaiad)
• Does not use docker - lower resource requirements
• bash-completion

Disadvantages:

• Slightly higher dependency set (first-time install has two more steps)
• Slightly lower trust in the underlying tool on first run: bash is more finicky than docker configurations.
• The full workflow of a full-mesh hermes setup is not completely automatic yet. The configuration is generated, but you need to run your own create channel commands and set up the relayers. (I'm open for feedback if we want to automate that.)

Question: is docker a hard requirement for people? Would the gm tool be able to replace all the requirements associated with the full-mesh tool?

[colin-axner]

I think there are two use cases that come to mind (at least from my wishlist):

1. Developer quick-start testing automation. Both pr's attempt to address this but it seems Gaiad manager #928 is a more specialized approach to this problem
2. Simulation testing of multi-network (and various versioning) environments

I think 1) is the immediate priority (and more useful for developers of this repo) and #928 looks like a good approach for this based on the trade offs listed above. Usage of this feature might inform use cases for 2)

I think 2) is something we definitely will need to eventually solve and this pr is some good investigation into this problem. Based on some recent discussion on this topic, we determined that this level of testing was a little ahead of where we are now (although never to early to start discussion). We want a solution that supports multiple IBC implementations, relayers, and chains. It needs to be language agnostic and ideally driven by MBT testing. Docker seems like a natural fit here to provide stability. Interaction directly with docker from a user standpoint should be very limited. Interacting with this set of testing should be more along the lines of running a command with various parameters set

My interest in 2) stems from wanting to kick off a set of simulations before doing a final ibc-go release to ensure compatibility between chains running an older ibc-go version and the newer ibc-go version (including ensuring that a relayer functions between both)

I'll try to document all the testing I do before our next release to inform what features should be supported

Documenting any testing I do on this discussion thread

[greg-szabo]

Note: the latest update to the gm tool allows you to configure everything for a full-mesh setup with different network configurations and different binaries.

The docker setup auto-generated here is interesting for some cases but as the immediate priority is dev testing, I'll close this issue with the remark that we can revisit it later and expand on it as integration testing, acceptance testing and end-to-end testing becomes more focused.