Use new node contract in registry

.github/workflows/lint.yml

@@ -22,4 +22,4 @@ jobs:
       - name: golangci-lint
         uses: golangci/golangci-lint-action@v3
         with:
-          args: --timeout=5m --config dev/.golangci.yaml
+          args: --timeout=5m --config .golangci.yaml

dev/.golangci.yaml → .golangci.yaml

@@ -12,3 +12,6 @@ linters:
     - ineffassign
     - staticcheck
     - unused
+issues:
+  exclude-dirs:
+    - pkg/tools

.mockery.yaml

@@ -4,6 +4,9 @@ mockname: "Mock{{.InterfaceName}}"
 outpkg: mocks
 filename: "mock_{{.InterfaceName}}.go"
 packages:
+  github.com/xmtp/xmtpd/pkg/registry:
+    interfaces:
+      NodesContract:
   github.com/xmtp/xmtpd/pkg/indexer/blockchain:
     interfaces:
       ChainClient:

README.md

@@ -8,9 +8,9 @@
 
 `xmtpd` (XMTP daemon) is an experimental version of XMTP node software. It is **not** the node software that currently forms the XMTP network.
 
-After `xmtpd` meets specific functional requirements, the plan is for it to become the node software that powers the XMTP network. 
+After `xmtpd` meets specific functional requirements, the plan is for it to become the node software that powers the XMTP network.
 
-Some of these requirements include reaching functional parity with the current node software and reliably performing data replication without data loss. 
+Some of these requirements include reaching functional parity with the current node software and reliably performing data replication without data loss.
 
 To keep up with and provide feedback about `xmtpd` development, see the [Issues tab](https://github.com/xmtp/xmtpd/issues) in this repo.
 
@@ -46,7 +46,7 @@ dev/down
 To start the `xmtpd` node, run:
 
 ```sh
-dev/start
+dev/run
 ```
 
 ## Test the node
@@ -57,27 +57,27 @@ To run tests against the `xmtpd` node, run:
 dev/test
 ```
 
-These tests provide a full suite of unit and integration tests for the `xmtpd` repo to help ensure and maintain correctness of the code over time and to avoid regressions as the code evolves. You can explore the tests by taking a look at any files with the suffix `_test.go`. 
+These tests provide a full suite of unit and integration tests for the `xmtpd` repo to help ensure and maintain correctness of the code over time and to avoid regressions as the code evolves. You can explore the tests by taking a look at any files with the suffix `_test.go`.
 
 ## Monitor the node
 
 The `xmtpd` node build provides two options for monitoring your node.
 
 - To access your local Prometheus instance to explore node metrics, run:
 
-    ```sh
-    open http://localhost:9090
-    ```
+  ```sh
+  open http://localhost:9090
+  ```
 
-    To learn how to query node data in Prometheus, see [Metric Types in Prometheus and PromQL](https://promlabs.com/blog/2020/09/25/metric-types-in-prometheus-and-promql) and [The Anatomy of a PromQL Query](https://promlabs.com/blog/2020/06/18/the-anatomy-of-a-promql-query/).
+  To learn how to query node data in Prometheus, see [Metric Types in Prometheus and PromQL](https://promlabs.com/blog/2020/09/25/metric-types-in-prometheus-and-promql) and [The Anatomy of a PromQL Query](https://promlabs.com/blog/2020/06/18/the-anatomy-of-a-promql-query/).
 
 - To access your local Grafana instance to explore and build node dashboards, run:
 
-    ```sh
-    open http://localhost:3000
-    ```
+  ```sh
+  open http://localhost:3000
+  ```
 
-    To learn how to visualize node data in Grafana, see [Prometheus Histograms with Grafana Heatmaps](https://towardsdatascience.com/prometheus-histograms-with-grafana-heatmaps-d556c28612c7) and [How to visualize Prometheus histograms in Grafana](https://grafana.com/blog/2020/06/23/how-to-visualize-prometheus-histograms-in-grafana/).
+  To learn how to visualize node data in Grafana, see [Prometheus Histograms with Grafana Heatmaps](https://towardsdatascience.com/prometheus-histograms-with-grafana-heatmaps-d556c28612c7) and [How to visualize Prometheus histograms in Grafana](https://grafana.com/blog/2020/06/23/how-to-visualize-prometheus-histograms-in-grafana/).
 
 # Contributing
 
@@ -87,24 +87,24 @@ Please follow the [style guide](https://google.github.io/styleguide/go/decisions
 
 Submit and land a PR to https://github.com/xmtp/proto. Then run:
 
-  ```sh
-  dev/generate
-  ```
+```sh
+dev/generate
+```
 
 ## Modifying the database schema
 
 Create a new migration by running:
 
-  ```sh
-  dev/gen-migration
-  ```
+```sh
+dev/gen-migration
+```
 
 Fill in the migrations in the generated files. If you are unfamiliar with migrations, you may follow [this guide](https://github.com/golang-migrate/migrate/blob/master/MIGRATIONS.md). The database is PostgreSQL and the driver is PGX.
 
 ## Modifying database queries
 
 We use [sqlc](https://docs.sqlc.dev/en/latest/index.html) to generate the code for our DB queries. Modify the `queries.sql` file, and then run:
 
-  ```sh
-  sqlc generate
-  ```
+```sh
+sqlc generate
+```

cmd/replication/main.go

@@ -9,6 +9,7 @@ import (
 	"syscall"
 
 	"github.com/jessevdk/go-flags"
+	"github.com/xmtp/xmtpd/pkg/config"
 	"github.com/xmtp/xmtpd/pkg/registry"
 	"github.com/xmtp/xmtpd/pkg/server"
 	"github.com/xmtp/xmtpd/pkg/tracing"
@@ -18,7 +19,7 @@ import (
 
 var Commit string
 
-var options server.Options
+var options config.ServerOptions
 
 func main() {
 	if _, err := flags.Parse(&options); err != nil {
@@ -81,7 +82,7 @@ func fatal(msg string, args ...any) {
 	log.Fatalf(msg, args...)
 }
 
-func buildLogger(options server.Options) (*zap.Logger, *zap.Config, error) {
+func buildLogger(options config.ServerOptions) (*zap.Logger, *zap.Config, error) {
 	atom := zap.NewAtomicLevel()
 	level := zapcore.InfoLevel
 	err := level.Set(options.LogLevel)

dev/abigen

@@ -8,6 +8,7 @@ rm -f ./pkg/abis/*.go
 cd contracts
 
 # Generate the abi files out of the solidity code
+mkdir -p ../build
 forge inspect ./src/Nodes.sol:Nodes abi > ../build/Nodes.abi.json
 forge inspect ./src/GroupMessages.sol:GroupMessages abi > ../build/GroupMessages.abi.json
 

dev/generate

@@ -1,9 +1,11 @@
-#!/usr/bin/env sh
-set -e
+#!/usr/bin/env bash
+
+set -euo pipefail
 
 go generate ./...
-mockery
+rm -f pkg/mocks/*
 ./dev/abigen
+mockery
 
 rm -rf pkg/proto/**/*.pb.go pkg/proto/**/*.pb.gw.go pkg/proto/**/*.swagger.json
 if ! buf generate https://github.com/xmtp/proto.git#subdir=proto; then

dev/local.env

@@ -0,0 +1,11 @@
+#!/bin/bash
+
+source dev/contracts/.env
+
+export CHAIN_RPC_URL=$DOCKER_RPC_URL  # From contracts/.env
+export NODE_PRIVATE_KEY=$PRIVATE_KEY # From contracts/.env
+export WRITER_CONNECTION_STRING="postgres://postgres:xmtp@localhost:8765/postgres?sslmode=disable"
+NODES_CONTRACT_ADDRESS="$(jq -r '.deployedTo' build/Nodes.json)" # Built by contracts/deploy-local
+export NODES_CONTRACT_ADDRESS
+GROUP_MESSAGES_CONTRACT_ADDRESS="$(jq -r '.deployedTo' build/GroupMessages.json)" # Built by contracts/deploy-local
+export GROUP_MESSAGES_CONTRACT_ADDRESS

dev/run

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+set -eu
+
+. dev/local.env
+
+go run cmd/replication/main.go \
+    --db.writer-connection-string=$WRITER_CONNECTION_STRING \
+    --private-key=${NODE_PRIVATE_KEY} \
+    --contracts.nodes-address=$NODES_CONTRACT_ADDRESS \
+    --contracts.messages-address=$GROUP_MESSAGES_CONTRACT_ADDRESS \
+    --contracts.rpc-url=$CHAIN_RPC_URL

dev/up

@@ -7,10 +7,12 @@ if ! which forge &>/dev/null; then curl -L https://foundry.paradigm.xyz | bash ;
 if ! which migrate &>/dev/null; then brew install golang-migrate; fi
 if ! which golangci-lint &>/dev/null; then brew install golangci-lint; fi
 if ! which shellcheck &>/dev/null; then brew install shellcheck; fi
-if ! which mockery &>/dev/null; then brew install mockery; fi
+if ! which mockery &>/dev/null; then go install github.com/vektra/mockery/v2; fi
 if ! which sqlc &> /dev/null; then brew install sqlc; fi
 if ! which buf &> /dev/null; then brew install buf; fi
 if ! which golines &>/dev/null; then go install github.com/segmentio/golines@latest; fi
+if ! which abigen &>/dev/null; then go install github.com/ethereum/go-ethereum/cmd/abigen; fi
+
 
 dev/generate
 dev/docker/up

go.mod

@@ -9,7 +9,9 @@ require (
 	github.com/jackc/pgx/v5 v5.5.4
 	github.com/jessevdk/go-flags v1.6.1
 	github.com/pires/go-proxyproto v0.7.0
+	github.com/segmentio/golines v0.12.2
 	github.com/stretchr/testify v1.9.0
+	github.com/vektra/mockery/v2 v2.44.1
 	go.uber.org/zap v1.27.0
 	google.golang.org/genproto/googleapis/api v0.0.0-20240725223205-93522f1f2a9f
 	google.golang.org/grpc v1.65.0
@@ -25,56 +27,143 @@ require (
 	github.com/DataDog/go-libddwaf/v3 v3.2.1 // indirect
 	github.com/DataDog/go-tuf v1.0.2-0.5.2 // indirect
 	github.com/DataDog/sketches-go v1.4.5 // indirect
+	github.com/DataDog/zstd v1.4.5 // indirect
 	github.com/Microsoft/go-winio v0.6.2 // indirect
 	github.com/StackExchange/wmi v1.2.1 // indirect
+	github.com/VictoriaMetrics/fastcache v1.12.2 // indirect
+	github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 // indirect
+	github.com/alecthomas/units v0.0.0-20231202071711-9a357b53e9c9 // indirect
+	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/bits-and-blooms/bitset v1.10.0 // indirect
 	github.com/btcsuite/btcd/btcec/v2 v2.2.1 // indirect
 	github.com/btcsuite/btcd/chaincfg/chainhash v1.1.0 // indirect
 	github.com/cespare/xxhash/v2 v2.3.0 // indirect
+	github.com/chigopher/pathlib v0.19.1 // indirect
+	github.com/cockroachdb/errors v1.11.3 // indirect
+	github.com/cockroachdb/fifo v0.0.0-20240606204812-0bbfbd93a7ce // indirect
+	github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b // indirect
+	github.com/cockroachdb/pebble v1.1.1 // indirect
+	github.com/cockroachdb/redact v1.1.5 // indirect
+	github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 // indirect
 	github.com/consensys/bavard v0.1.13 // indirect
 	github.com/consensys/gnark-crypto v0.12.1 // indirect
+	github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
+	github.com/crate-crypto/go-ipa v0.0.0-20240223125850-b1e8a79f509c // indirect
 	github.com/crate-crypto/go-kzg-4844 v1.0.0 // indirect
+	github.com/dave/dst v0.27.3 // indirect
 	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
 	github.com/deckarep/golang-set/v2 v2.6.0 // indirect
 	github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 // indirect
+	github.com/deepmap/oapi-codegen v1.6.0 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/ebitengine/purego v0.6.0-alpha.5 // indirect
 	github.com/ethereum/c-kzg-4844 v1.0.0 // indirect
+	github.com/ethereum/go-verkle v0.1.1-0.20240306133620-7d920df305f0 // indirect
+	github.com/fatih/structtag v1.2.0 // indirect
+	github.com/ferranbt/fastssz v0.1.2 // indirect
+	github.com/fjl/memsize v0.0.2 // indirect
 	github.com/fsnotify/fsnotify v1.6.0 // indirect
+	github.com/gballet/go-libpcsclite v0.0.0-20190607065134-2772fd86a8ff // indirect
+	github.com/getsentry/sentry-go v0.27.0 // indirect
 	github.com/go-ole/go-ole v1.3.0 // indirect
+	github.com/gofrs/flock v0.8.1 // indirect
+	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/golang-jwt/jwt/v4 v4.5.0 // indirect
+	github.com/golang/protobuf v1.5.4 // indirect
+	github.com/golang/snappy v0.0.5-0.20220116011046-fa5810519dcb // indirect
 	github.com/google/uuid v1.6.0 // indirect
 	github.com/gorilla/websocket v1.5.0 // indirect
+	github.com/graph-gophers/graphql-go v1.5.0 // indirect
 	github.com/hashicorp/errwrap v1.1.0 // indirect
+	github.com/hashicorp/go-bexpr v0.1.10 // indirect
 	github.com/hashicorp/go-multierror v1.1.1 // indirect
+	github.com/hashicorp/hcl v1.0.1-vault-5 // indirect
+	github.com/holiman/billy v0.0.0-20240216141850-2abb0c79d3c4 // indirect
+	github.com/holiman/bloomfilter/v2 v2.0.3 // indirect
 	github.com/holiman/uint256 v1.3.0 // indirect
+	github.com/huandu/xstrings v1.4.0 // indirect
+	github.com/huin/goupnp v1.3.0 // indirect
+	github.com/iancoleman/strcase v0.2.0 // indirect
+	github.com/inconshreveable/mousetrap v1.1.0 // indirect
+	github.com/influxdata/influxdb-client-go/v2 v2.4.0 // indirect
+	github.com/influxdata/influxdb1-client v0.0.0-20220302092344-a9ab5670611c // indirect
+	github.com/influxdata/line-protocol v0.0.0-20200327222509-2487e7298839 // indirect
 	github.com/jackc/pgpassfile v1.0.0 // indirect
 	github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
 	github.com/jackc/puddle/v2 v2.2.1 // indirect
+	github.com/jackpal/go-nat-pmp v1.0.2 // indirect
+	github.com/jinzhu/copier v0.3.5 // indirect
+	github.com/klauspost/compress v1.17.1 // indirect
+	github.com/klauspost/cpuid/v2 v2.2.5 // indirect
+	github.com/kr/pretty v0.3.1 // indirect
+	github.com/kr/text v0.2.0 // indirect
 	github.com/lib/pq v1.10.9 // indirect
+	github.com/magiconair/properties v1.8.7 // indirect
+	github.com/mattn/go-colorable v0.1.13 // indirect
+	github.com/mattn/go-isatty v0.0.20 // indirect
+	github.com/mattn/go-runewidth v0.0.15 // indirect
+	github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect
+	github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d // indirect
+	github.com/minio/sha256-simd v1.0.0 // indirect
+	github.com/mitchellh/go-homedir v1.1.0 // indirect
+	github.com/mitchellh/mapstructure v1.5.0 // indirect
+	github.com/mitchellh/pointerstructure v1.2.0 // indirect
 	github.com/mmcloughlin/addchain v0.4.0 // indirect
+	github.com/olekukonko/tablewriter v0.0.5 // indirect
 	github.com/outcaste-io/ristretto v0.2.3 // indirect
+	github.com/pelletier/go-toml/v2 v2.0.9 // indirect
+	github.com/peterh/liner v1.1.1-0.20190123174540-a2c9a5303de7 // indirect
 	github.com/philhofer/fwd v1.1.2 // indirect
 	github.com/pkg/errors v0.9.1 // indirect
 	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
+	github.com/prometheus/client_golang v1.12.0 // indirect
+	github.com/prometheus/client_model v0.2.1-0.20210607210712-147c58e9608a // indirect
+	github.com/prometheus/common v0.32.1 // indirect
+	github.com/prometheus/procfs v0.7.3 // indirect
+	github.com/rivo/uniseg v0.4.4 // indirect
 	github.com/rogpeppe/go-internal v1.10.0 // indirect
+	github.com/rs/cors v1.7.0 // indirect
+	github.com/rs/zerolog v1.29.0 // indirect
+	github.com/russross/blackfriday/v2 v2.1.0 // indirect
 	github.com/secure-systems-lab/go-securesystemslib v0.7.0 // indirect
 	github.com/shirou/gopsutil v3.21.4-0.20210419000835-c7a38de76ee5+incompatible // indirect
+	github.com/sirupsen/logrus v1.9.3 // indirect
+	github.com/spf13/afero v1.9.3 // indirect
+	github.com/spf13/cast v1.5.0 // indirect
+	github.com/spf13/cobra v1.6.1 // indirect
+	github.com/spf13/jwalterweatherman v1.1.0 // indirect
+	github.com/spf13/pflag v1.0.5 // indirect
+	github.com/spf13/viper v1.15.0 // indirect
+	github.com/status-im/keycard-go v0.2.0 // indirect
 	github.com/stretchr/objx v0.5.2 // indirect
+	github.com/subosito/gotenv v1.4.2 // indirect
 	github.com/supranational/blst v0.3.11 // indirect
+	github.com/syndtr/goleveldb v1.0.1-0.20220721030215-126854af5e6d // indirect
 	github.com/tinylib/msgp v1.1.8 // indirect
 	github.com/tklauser/go-sysconf v0.3.12 // indirect
 	github.com/tklauser/numcpus v0.6.1 // indirect
+	github.com/tyler-smith/go-bip39 v1.1.0 // indirect
+	github.com/urfave/cli/v2 v2.25.7 // indirect
+	github.com/x-cray/logrus-prefixed-formatter v0.5.2 // indirect
+	github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 // indirect
 	go.uber.org/atomic v1.11.0 // indirect
 	go.uber.org/multierr v1.11.0 // indirect
 	golang.org/x/crypto v0.24.0 // indirect
 	golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa // indirect
+	golang.org/x/mod v0.17.0 // indirect
 	golang.org/x/net v0.26.0 // indirect
 	golang.org/x/sync v0.7.0 // indirect
 	golang.org/x/sys v0.21.0 // indirect
+	golang.org/x/term v0.21.0 // indirect
 	golang.org/x/text v0.16.0 // indirect
 	golang.org/x/time v0.5.0 // indirect
+	golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect
 	golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
 	google.golang.org/genproto/googleapis/rpc v0.0.0-20240723171418-e6d459c13d2a // indirect
+	gopkg.in/alecthomas/kingpin.v2 v2.2.6 // indirect
+	gopkg.in/ini.v1 v1.67.0 // indirect
+	gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
+	gopkg.in/yaml.v2 v2.4.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 	rsc.io/tmplfunc v0.0.3 // indirect
 )