Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: Token List Parsing #13

Merged
merged 4 commits into from
Feb 2, 2023
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
graceful token list parsing
  • Loading branch information
refcell committed Feb 1, 2023
commit 966d9c581cf135a43531de7944e03f0218a9ba89
1 change: 0 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -114,7 +114,6 @@ run-optimism-mainnet-construction-check:
# For the genesis block hash, see:
# https://github.com/ethereum-optimism/optimism/blob/5e8bc3d5b4f36f0192b22b032e25b09f23cd0985/op-node/chaincfg/chains.go
run-optimism-mainnet:
make build \
CHAIN_CONFIG='{ "chainId": 10, "terminalTotalDifficultyPassed": true }' \
MODE=ONLINE \
PORT=8080 \
76 changes: 71 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -2,32 +2,98 @@

# op-rosetta • [![tests](https://github.com/mdehoog/op-rosetta/actions/workflows/unit-tests.yaml/badge.svg?label=tests)](https://github.com/mdehoog/op-rosetta/actions/workflows/unit-tests.yaml) [![lints](https://github.com/mdehoog/op-rosetta/actions/workflows/lints.yaml/badge.svg)](https://github.com/mdehoog/op-rosetta/actions/workflows/lints.yaml)

> **Warning**
> WIP; Not bedrock-compatible yet.

Provides Rosetta API compatibility for Optimism Bedrock, a low-cost and lightning-fast Ethereum L2 blockchain.


## Overview

`op-rosetta` is an executable [client](./app/client.go) extending the [rosetta-geth-sdk](https://github.com/coinbase/rosetta-geth-sdk).
`op-rosetta` is an executable [client](./app/client.go) extending the [rosetta-geth-sdk](https://github.com/coinbase/rosetta-geth-sdk) which itself extends [rosetta-sdk-go](https://github.com/coinbase/rosetta-sdk-go).

To learn more about the Rosetta API, you can find more online at [rosetta-api.org](https://www.rosetta-api.org/).


## Setup

First, clone the repository:

```bash
git clone https://github.com/mdehoog/op-rosetta.git
```

Then, configure your own `.env` file by copying the contents of `.env.example` to `.env`.

```bash
cp .env.example .env
```

The `.env` variables will need to be configured.


## Usage

Run the `op-rosetta` client for goerli like so: `make run-optimism-goerli`.

As part of the `run-optimism-goerli` step inside the [makefile](./Makefile), a few parameters are specified before running the `bin/op-rosetta` executable:

```
CHAIN_CONFIG='{ "chainId": 10, "terminalTotalDifficultyPassed": true }' \
MODE=ONLINE \
PORT=8080 \
BLOCKCHAIN=Optimism \
NETWORK=Goerli \
ENABLE_TRACE_CACHE=true \
ENABLE_GETH_TRACER=true \
GETH=${OPTIMISM_GOERLI_NODE} \
GENESIS_BLOCK_HASH=${OPTIMISM_GOERLI_GENESIS_BLOCK_HASH} \
bin/op-rosetta
```

These parameters are configured as follows:
- `CHAIN_CONFIG` has the same field values as the genesis config’s. But only the `chainId` and `terminalTotalDifficultyPassed` values are needed.
- Setting `MODE` to `ONLINE` permits outbound connections.
- `PORT` is set to `8080` - the default connection for `rosetta-cli`.
- `BLOCKCHAIN` is `Optimism` and `NETWORK` is `Goerli`
- We enable geth tracing and caching with `ENABLE_TRACE_CACHE` and `ENABLE_GETH_TRACER`
- `GETH` points to a running op-geth *archive* node. This should be set in your `.env` file by setting `OPTIMISM_GOERLI_NODE` to a node url.
- `GENESIS_BLOCK_HASH` is the block hash of the genesis block. This should be set in your `.env` file by setting `OPTIMISM_GOERLI_GENESIS_BLOCK_HASH` to the goerli bedrock genesis block hash.

A mainnet client can be run with `make run-optimism-mainnet`.

> **Note**
> Mainnet will only be supported once Optimism mainnet is upgraded to its Bedrock Release.

The `run-optimism-mainnet` step inside the [makefile](./Makefile) specifies a similar set of variables as above:

```
CHAIN_CONFIG='{ "chainId": 10, "terminalTotalDifficultyPassed": true }' \
MODE=ONLINE \
PORT=8080 \
BLOCKCHAIN=Optimism \
NETWORK=Mainnet \
ENABLE_TRACE_CACHE=true \
ENABLE_GETH_TRACER=true \
GETH=${OPTIMISM_MAINNET_NODE} \
GENESIS_BLOCK_HASH=${OPTIMISM_MAINNET_GENESIS_BLOCK_HASH} \
bin/op-rosetta
```

These parameters are configured as follows:
- `CHAIN_CONFIG` has the same field values as the genesis config’s. But only the `chainId` and `terminalTotalDifficultyPassed` values are needed.
- Setting `MODE` to `ONLINE` permits outbound connections.
- `PORT` is set to `8080` - the default connection for `rosetta-cli`.
- `BLOCKCHAIN` is `Optimism` and `NETWORK` is `Mainnet`
- We enable geth tracing and caching with `ENABLE_TRACE_CACHE` and `ENABLE_GETH_TRACER`
- `GETH` points to a running op-geth *archive* node. This should be set in your `.env` file by setting `OPTIMISM_MAINNET_NODE` to a node url.
- `GENESIS_BLOCK_HASH` is the block hash of the genesis block. This should be set in your `.env` file by setting `OPTIMISM_MAINNET_GENESIS_BLOCK_HASH` to the mainnet bedrock genesis block hash.


## Testing

_NOTE: `op-rosetta` must be running on the specified host and port provided in the configuration file. For local testing, this can be done as described in the [Running](#running) section, which will run an instance on localhost, port 8080._

See [configs/README.md](./configs/README.md) for more information.


## Layout

```
@@ -40,10 +106,10 @@ See [configs/README.md](./configs/README.md) for more information.
│ └── README.md
└── pkg
├── client/ -> Optimism Rosetta API client
├── common/ -> Standalone common functionality
├── config/ -> Configuration options
├── handlers/ -> Rosetta API handlers
├── logging/ -> Logging utilities
├── topics/ -> Events topic functionality
└── utils/ -> Package utilities
```

39 changes: 39 additions & 0 deletions configs/README.md
Original file line number Diff line number Diff line change
@@ -30,6 +30,45 @@ To validate with `rosetta-cli`, run one of the following commands:

* `rosetta-cli check:data --configuration-file optimism/config.json` - This command validates that the Data API implementation is correct using the ethereum `optimism` node. It also ensures that the implementation does not miss any balance-changing operations.

> *Note*
>
> Running data validation takes a couple hours to completion. Once complete, the tool prints a report of any issues it encountered.
> If it fails to do so, there's likely an issue in the implementation of the `op-rosetta` client.


### Bedrock Compatibility

The rosetta configuration file must have its start_index set to the Bedrock fork block. If it’s set to any prior block then validation fails as op-rosetta fails to parse OVM (pre-Bedrock) transactions. Here’s a sample rosetta configuration file:

```
{
"network": {
"blockchain": "Optimism",
"network": "goerli"
},
"online_url": "http://localhost:5000",
"data_directory": "/tmp/rosetta-cli-goerli-bedrock-db",
"http_timeout": 300,
"max_retries": 30,
"max_online_connections": 30,
"max_sync_concurrency": 30,
"tip_delay": 120,
"compression_disabled": true,
"construction": null,
"data": {
"historical_balance_disabled": false,
"active_reconciliation_concurrency": 30,
"start_index": 4307133,
"log_balance_changes": true,
"log_transactions": true,
"log_blocks": true,
"end_conditions": {
"tip": true
}
}
}
```


### Reference

20 changes: 8 additions & 12 deletions pkg/config/common.go
Original file line number Diff line number Diff line change
@@ -102,19 +102,15 @@ func LoadConfiguration() (*configuration.Configuration, error) {
}
}

tokenListJson := os.Getenv(TokenListEnv)
if file, err := os.ReadFile(tokenListJson); err == nil {
// if the envvar points to a file, read it; otherwise the envvar contents is expected to be JSON
tokenListJson = string(file)
}
if tokenListJson == "" {
tokenListJson = "[]"
// Graceful tokenlist unmarshalling
tokenListJsonFilename := os.Getenv(TokenListEnv)
tokenListJsonFile, err := ReadTokenConfig(tokenListJsonFilename)
if err != nil {
return nil, fmt.Errorf("unable to parse token list %s: %w", tokenListJsonFilename, err)
}

var payload []configuration.Token
err = json.Unmarshal([]byte(tokenListJson), &payload)
tokenWhiteList, err := UnmarshalTokenConfig(tokenListJsonFile)
if err != nil {
return nil, fmt.Errorf("error parsing %s: %w", tokenListJson, err)
return nil, err
}

config.RosettaCfg = configuration.RosettaConfig{
@@ -126,7 +122,7 @@ func LoadConfiguration() (*configuration.Configuration, error) {
},
TracePrefix: "optrace",
FilterTokens: filterTokens,
TokenWhiteList: payload,
TokenWhiteList: tokenWhiteList,
SupportsSyncing: true,
}

63 changes: 63 additions & 0 deletions pkg/config/tokens.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
package config

import (
"encoding/json"
"fmt"
"os"

"github.com/coinbase/rosetta-geth-sdk/configuration"
)

// ReadTokenConfig attempts to read file contents from a filename.
func ReadTokenConfig(filename string) (string, error) {
contents := ""
if file, err := os.ReadFile(filename); err == nil {
// if the envvar points to a file, read it; otherwise the envvar contents is expected to be JSON
contents = string(file)
}
if contents == "" {
contents = "[]"
}
return contents, nil
}

// UnmarshalTokenConfig attempts to construct a list of [configuration.Token] from a JSON file.
func UnmarshalTokenConfig(contents string) ([]configuration.Token, error) {
refcell marked this conversation as resolved.
Show resolved Hide resolved
// Try to parse the file as a list of tokens.
var payload []configuration.Token
if err := json.Unmarshal([]byte(contents), &payload); err == nil {
return payload, nil
}

// If this fails, try the backwards-compatible token json format
var outer map[string]interface{}
if err := json.Unmarshal([]byte(contents), &outer); err == nil {
refcell marked this conversation as resolved.
Show resolved Hide resolved
for k, v := range outer {
for t, _ := range v.(map[string]interface{}) {
fmt.Printf("t: %s\n", t)
switch k {
case "Mainnet":
payload = append(payload, configuration.Token{
ChainID: 1,
Address: t,
})
case "Testnet":
payload = append(payload, configuration.Token{
ChainID: 420,
Address: t,
})
case "Goerli":
payload = append(payload, configuration.Token{
ChainID: 420,
Address: t,
})
default:
return nil, fmt.Errorf("unknown chain %s found when parsing json token list", k)
}
}
}
return payload, nil
}

return nil, fmt.Errorf("error parsing file contents as json token list")
}
94 changes: 94 additions & 0 deletions pkg/config/tokens_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
package config_test

import (
"testing"

SdkConfiguration "github.com/coinbase/rosetta-geth-sdk/configuration"
"github.com/mdehoog/op-rosetta/pkg/config"

"github.com/stretchr/testify/suite"
)

// TokensTestSuite is a test suite for tokens utilities.
type TokensTestSuite struct {
suite.Suite
}

// TestTokens runs the TokensTestSuite.
func TestTokens(t *testing.T) {
suite.Run(t, new(TokensTestSuite))
}

// TestUnmarshalOldTokenConfig tests unmarshalling the outdated token config json format.
func (testSuite *TokensTestSuite) TestUnmarshalOldTokenConfig() {
contents := `{
"Mainnet" : {
"0x4200000000000000000000000000000000000042": true,
"0x7f5c764cbc14f9669b88837ca1490cca17c31607": true
},
"Testnet" : {
"0xda10009cbd5d07dd0cecc66161fc93d7c9000da1": true
},
"Goerli" : {
"0x7f5c764cbc14f9669b88837ca1490cca17c31607": true
}
}`
c, err := config.UnmarshalTokenConfig(contents)
testSuite.NoError(err)
testSuite.Equal([]SdkConfiguration.Token{
{
ChainID: 1,
Address: "0x4200000000000000000000000000000000000042",
},
{
ChainID: 1,
Address: "0x7f5c764cbc14f9669b88837ca1490cca17c31607",
},
{
ChainID: 420,
Address: "0xda10009cbd5d07dd0cecc66161fc93d7c9000da1",
},
{
ChainID: 420,
Address: "0x7f5c764cbc14f9669b88837ca1490cca17c31607",
},
}, c)
}

// TestUnmarshalTokenConfig tests unmarshalling the token config json format.
func (testSuite *TokensTestSuite) TestUnmarshalTokenConfig() {
contents := `[
{
"chainId": 1,
"address": "0x7f5c764cbc14f9669b88837ca1490cca17c31607",
"name": "USD Coin",
"symbol": "USDC",
"decimals": 6
},
{
"chainId": 1,
"address": "0x94b008aa00579c1307b0ef2c499ad98a8ce58e58",
"name": "USDT",
"symbol": "USDT",
"decimals": 6
}
]`
c, err := config.UnmarshalTokenConfig(contents)
testSuite.NoError(err)
testSuite.Equal([]SdkConfiguration.Token{
{
ChainID: 1,
Address: "0x7f5c764cbc14f9669b88837ca1490cca17c31607",
Name: "USD Coin",
Symbol: "USDC",
Decimals: 6,
},
{
ChainID: 1,
Address: "0x94b008aa00579c1307b0ef2c499ad98a8ce58e58",
Name: "USDT",
Symbol: "USDT",
Decimals: 6,
},
}, c)
}