Feat: Token List Parsing

Makefile

@@ -114,7 +114,6 @@ run-optimism-mainnet-construction-check:
 # For the genesis block hash, see:
 # https://github.com/ethereum-optimism/optimism/blob/5e8bc3d5b4f36f0192b22b032e25b09f23cd0985/op-node/chaincfg/chains.go
 run-optimism-mainnet:
-	make build \
 	CHAIN_CONFIG='{ "chainId": 10, "terminalTotalDifficultyPassed": true }'	\
 	MODE=ONLINE \
 	PORT=8080 \

README.md

@@ -2,32 +2,98 @@
 
 # op-rosetta • [![tests](https://github.com/mdehoog/op-rosetta/actions/workflows/unit-tests.yaml/badge.svg?label=tests)](https://github.com/mdehoog/op-rosetta/actions/workflows/unit-tests.yaml) [![lints](https://github.com/mdehoog/op-rosetta/actions/workflows/lints.yaml/badge.svg)](https://github.com/mdehoog/op-rosetta/actions/workflows/lints.yaml)
 
-> **Warning**
-> WIP; Not bedrock-compatible yet.
-
 Provides Rosetta API compatibility for Optimism Bedrock, a low-cost and lightning-fast Ethereum L2 blockchain.
 
+
 ## Overview
 
-`op-rosetta` is an executable [client](./app/client.go) extending the [rosetta-geth-sdk](https://github.com/coinbase/rosetta-geth-sdk).
+`op-rosetta` is an executable [client](./app/client.go) extending the [rosetta-geth-sdk](https://github.com/coinbase/rosetta-geth-sdk) which itself extends [rosetta-sdk-go](https://github.com/coinbase/rosetta-sdk-go).
 
 To learn more about the Rosetta API, you can find more online at [rosetta-api.org](https://www.rosetta-api.org/).
 
+
+## Setup
+
+First, clone the repository:
+
+```bash
+git clone https://github.com/mdehoog/op-rosetta.git
+```
+
+Then, configure your own `.env` file by copying the contents of `.env.example` to `.env`.
+
+```bash
+cp .env.example .env
+```
+
+The `.env` variables will need to be configured.
+
+
 ## Usage
 
 Run the `op-rosetta` client for goerli like so: `make run-optimism-goerli`.
 
+As part of the `run-optimism-goerli` step inside the [makefile](./Makefile), a few parameters are specified before running the `bin/op-rosetta` executable:
+
+```
+CHAIN_CONFIG='{ "chainId": 10, "terminalTotalDifficultyPassed": true }'	\
+MODE=ONLINE \
+PORT=8080 \
+BLOCKCHAIN=Optimism \
+NETWORK=Goerli \
+ENABLE_TRACE_CACHE=true \
+ENABLE_GETH_TRACER=true \
+GETH=${OPTIMISM_GOERLI_NODE} \
+GENESIS_BLOCK_HASH=${OPTIMISM_GOERLI_GENESIS_BLOCK_HASH} \
+bin/op-rosetta
+```
+
+These parameters are configured as follows:
+- `CHAIN_CONFIG` has the same field values as the genesis config’s. But only the `chainId` and `terminalTotalDifficultyPassed` values are needed.
+- Setting `MODE` to `ONLINE` permits outbound connections.
+- `PORT` is set to `8080` - the default connection for `rosetta-cli`.
+- `BLOCKCHAIN` is `Optimism` and `NETWORK` is `Goerli`
+- We enable geth tracing and caching with `ENABLE_TRACE_CACHE` and `ENABLE_GETH_TRACER`
+- `GETH` points to a running op-geth *archive* node. This should be set in your `.env` file by setting `OPTIMISM_GOERLI_NODE` to a node url.
+- `GENESIS_BLOCK_HASH` is the block hash of the genesis block. This should be set in your `.env` file by setting `OPTIMISM_GOERLI_GENESIS_BLOCK_HASH` to the goerli bedrock genesis block hash.
+
 A mainnet client can be run with `make run-optimism-mainnet`.
 
 > **Note**
 > Mainnet will only be supported once Optimism mainnet is upgraded to its Bedrock Release.
 
+The `run-optimism-mainnet` step inside the [makefile](./Makefile) specifies a similar set of variables as above:
+
+```
+CHAIN_CONFIG='{ "chainId": 10, "terminalTotalDifficultyPassed": true }'	\
+MODE=ONLINE \
+PORT=8080 \
+BLOCKCHAIN=Optimism \
+NETWORK=Mainnet \
+ENABLE_TRACE_CACHE=true \
+ENABLE_GETH_TRACER=true \
+GETH=${OPTIMISM_MAINNET_NODE} \
+GENESIS_BLOCK_HASH=${OPTIMISM_MAINNET_GENESIS_BLOCK_HASH} \
+bin/op-rosetta
+```
+
+These parameters are configured as follows:
+- `CHAIN_CONFIG` has the same field values as the genesis config’s. But only the `chainId` and `terminalTotalDifficultyPassed` values are needed.
+- Setting `MODE` to `ONLINE` permits outbound connections.
+- `PORT` is set to `8080` - the default connection for `rosetta-cli`.
+- `BLOCKCHAIN` is `Optimism` and `NETWORK` is `Mainnet`
+- We enable geth tracing and caching with `ENABLE_TRACE_CACHE` and `ENABLE_GETH_TRACER`
+- `GETH` points to a running op-geth *archive* node. This should be set in your `.env` file by setting `OPTIMISM_MAINNET_NODE` to a node url.
+- `GENESIS_BLOCK_HASH` is the block hash of the genesis block. This should be set in your `.env` file by setting `OPTIMISM_MAINNET_GENESIS_BLOCK_HASH` to the mainnet bedrock genesis block hash.
+
+
 ## Testing
 
 _NOTE: `op-rosetta` must be running on the specified host and port provided in the configuration file. For local testing, this can be done as described in the [Running](#running) section, which will run an instance on localhost, port 8080._
 
 See [configs/README.md](./configs/README.md) for more information.
 
+
 ## Layout
 
 ```
@@ -40,10 +106,10 @@ See [configs/README.md](./configs/README.md) for more information.
 │   └── README.md
 └── pkg
     ├── client/ -> Optimism Rosetta API client
+    ├── common/ -> Standalone common functionality
     ├── config/ -> Configuration options
     ├── handlers/ -> Rosetta API handlers
     ├── logging/ -> Logging utilities
-    ├── topics/ -> Events topic functionality
     └── utils/ -> Package utilities
 ```
 

configs/README.md

@@ -30,6 +30,45 @@ To validate with `rosetta-cli`, run one of the following commands:
 
 * `rosetta-cli check:data --configuration-file optimism/config.json` - This command validates that the Data API implementation is correct using the ethereum `optimism` node. It also ensures that the implementation does not miss any balance-changing operations.
 
+> *Note*
+>
+> Running data validation takes a couple hours to completion. Once complete, the tool prints a report of any issues it encountered.
+> If it fails to do so, there's likely an issue in the implementation of the `op-rosetta` client.
+
+
+### Bedrock Compatibility
+
+The rosetta configuration file must have its start_index set to the Bedrock fork block. If it’s set to any prior block then validation fails as op-rosetta fails to parse OVM (pre-Bedrock) transactions. Here’s a sample rosetta configuration file:
+
+```
+{
+  "network": {
+    "blockchain": "Optimism",
+    "network": "goerli"
+  },
+  "online_url": "http://localhost:5000",
+  "data_directory": "/tmp/rosetta-cli-goerli-bedrock-db",
+  "http_timeout": 300,
+  "max_retries": 30,
+  "max_online_connections": 30,
+  "max_sync_concurrency": 30,
+  "tip_delay": 120,
+  "compression_disabled": true,
+  "construction": null,
+  "data": {
+    "historical_balance_disabled": false,
+    "active_reconciliation_concurrency": 30,
+    "start_index": 4307133,
+    "log_balance_changes": true,
+    "log_transactions": true,
+    "log_blocks": true,
+    "end_conditions": {
+        "tip": true
+    }
+  }
+}
+```
+
 
 ### Reference
 

pkg/config/common.go

@@ -102,19 +102,15 @@ func LoadConfiguration() (*configuration.Configuration, error) {
 		}
 	}
 
-	tokenListJson := os.Getenv(TokenListEnv)
-	if file, err := os.ReadFile(tokenListJson); err == nil {
-		// if the envvar points to a file, read it; otherwise the envvar contents is expected to be JSON
-		tokenListJson = string(file)
-	}
-	if tokenListJson == "" {
-		tokenListJson = "[]"
+	// Graceful tokenlist unmarshalling
+	tokenListJsonFilename := os.Getenv(TokenListEnv)
+	tokenListJsonFile, err := ReadTokenConfig(tokenListJsonFilename)
+	if err != nil {
+		return nil, fmt.Errorf("unable to parse token list %s: %w", tokenListJsonFilename, err)
 	}
-
-	var payload []configuration.Token
-	err = json.Unmarshal([]byte(tokenListJson), &payload)
+	tokenWhiteList, err := UnmarshalTokenConfig([]byte(tokenListJsonFile), *chainConfig.ChainID)
 	if err != nil {
-		return nil, fmt.Errorf("error parsing %s: %w", tokenListJson, err)
+		return nil, err
 	}
 
 	config.RosettaCfg = configuration.RosettaConfig{
@@ -126,7 +122,7 @@ func LoadConfiguration() (*configuration.Configuration, error) {
 		},
 		TracePrefix:     "optrace",
 		FilterTokens:    filterTokens,
-		TokenWhiteList:  payload,
+		TokenWhiteList:  tokenWhiteList,
 		SupportsSyncing: true,
 	}
 

pkg/config/tokens.go

@@ -0,0 +1,85 @@
+package config
+
+import (
+	"encoding/json"
+	"fmt"
+	"math/big"
+	"os"
+	"strings"
+
+	"github.com/coinbase/rosetta-geth-sdk/configuration"
+)
+
+// ReadTokenConfig attempts to read file contents from a filename.
+func ReadTokenConfig(filename string) (string, error) {
+	contents := ""
+	if file, err := os.ReadFile(filename); err == nil {
+		// if the envvar points to a file, read it; otherwise the envvar contents is expected to be JSON
+		contents = string(file)
+	}
+	if contents == "" {
+		contents = "[]"
+	}
+	return contents, nil
+}
+
+// UnmarshalTokenConfig attempts to construct a list of [configuration.Token] from a JSON file.
+// Accepts variadic network IDs used to filter tokens.
+func UnmarshalTokenConfig(contents []byte, networks ...big.Int) ([]configuration.Token, error) {
+	// Try to parse the file as a list of tokens.
+	var payload []configuration.Token
+	if err := json.Unmarshal(contents, &payload); err == nil {
+		return FilterNetworks(payload, networks...), nil
+	}
+
+	// If this fails, try the backwards-compatible token json format
+	var outer map[string]interface{}
+	if err := json.Unmarshal(contents, &outer); err == nil {
+		for k, v := range outer {
+			for t, b := range v.(map[string]interface{}) {
+				if b.(bool) {
+					switch strings.ToLower(k) {
+					case "mainnet":
+						payload = append(payload, configuration.Token{
+							ChainID: 1,
+							Address: t,
+						})
+					case "testnet":
+						payload = append(payload, configuration.Token{
+							ChainID: 420,
+							Address: t,
+						})
+					case "goerli":
+						payload = append(payload, configuration.Token{
+							ChainID: 420,
+							Address: t,
+						})
+					default:
+						return nil, fmt.Errorf("unknown chain %s found when parsing json token list", k)
+					}
+				}
+			}
+		}
+		return FilterNetworks(payload, networks...), nil
+	}
+
+	return nil, fmt.Errorf("error parsing file contents as json token list")
+}
+
+// FilterNetworks filters a list of [configuration.Token] by network ID.
+func FilterNetworks(tokens []configuration.Token, networks ...big.Int) []configuration.Token {
+	if len(networks) == 0 {
+		return tokens
+	}
+
+	filtered := []configuration.Token{}
+	for _, token := range tokens {
+		for _, network := range networks {
+			if token.ChainID == network.Uint64() {
+				filtered = append(filtered, token)
+			}
+		}
+	}
+
+	return filtered
+}