Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rate-limiting, Transaction to Whitelisting & Routing Server #108

Merged
merged 14 commits into from
Apr 27, 2020

Conversation

willmeister
Copy link

@willmeister willmeister commented Apr 25, 2020

Description

These changes support rate-limiting, Transaction "to" field whitelisting, and a routing server

The full node will now start in one of 3 different configuration modes:

  • Routing Server: This server is meant to sit in front our actual RPC server. It handles rate-limiting, transaction "to" field whitelist checks, and routing the request to the appropriate downstream server (see next few bullet points for more info)
  • Transaction Node: This is a full node that can technically handle all standard RPC requests, but if configured to do so, it will only be routed Transactions and requests that are tightly-coupled to transactions. This node will also handle all of the L1 & L2 subscription, contract deployment, logic, etc.
  • Read-only Node: This is a full node that can technically handle all standard RPC requests, but it will only be used to handle read-only requests, including calls, receipt / log fetching, etc. It will not connect to L1 at all or do any of the L1 <--> L2 communication logic. It is meant to be idempotent and horizontally scalable

The idea is that we will probably run servers in all 3 modes. We'll have the router to handle rate limiting and routing, and it will in-turn free up the single processing thread in the Transaction Node for just dealing with transactions to give the best latency possible. The read-only node will serve less important requests and do so without interfering with the Transaction Node. If it gets bogged down, we can horizontally scale it.

This also includes adding

  • A TimeBucketedCounter class to handle rolling counting over a time range
  • Changes to fullnode.ts to detect whether the server to be run is the Router, Transaction node, or Read-only node
  • AccountRateLimiter to handle the rate limiting checking for the Router

Metadata

Fixes

Contributing Agreement

…ode & and optional read-only node

- Time Bucketed Counter to handle rolling counting over a time range
- Changes to fullnode.ts to detect whether the server to be run is the Router, Transaction node, or Read-only node
- AccountRateLimiter to handle the rate limiting checking for the Router
Copy link
Collaborator

@ben-chain ben-chain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Woot woot, this is looking GTM! Left a couple small comments, only other thing I'd say is that we have a loooot of activity on web3-rpc-handler so sorry in advance for the conflicts 😅. Looks like the main modification there in this PR is a lock and timing logs though, so hopefully it's easy enough to merge 🤞

Comment on lines 20 to 22
Web3RpcMethods.getTransactionByHash,
Web3RpcMethods.getBlockByNumber,
Web3RpcMethods.getBlockByHash,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not positive, but aren't these read only methods too? Any reason these are sent to transaction handler?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, they use in-memory state that is populated in the sendRawTransaction, so they have to be in the same process.

method === Web3RpcMethods.sendRawTransaction
) {
try {
tx = parseTransaction(params[0])
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having recently spent some time debugging/working on the sendTransaction endpoint, I can shed some color here--I believe this parsing will always fail for eth_sendTransaction requests. The reason is because the standard JSON RPC parameters for that method are not actually equivalent to a transaction.

In general, I think the fullnode will only need expose the sendTransaction endpoint for the test-handler and never on a deployed instance. CC @karlfloersch to sanity check that, but if so it should be sufficient to just remove sendTransaction endpoint.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh ok. That works. Is that merged in? Surprised tests pass, but I'll route it to the read-only one.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep we should only expose sendTransaction on our test web3 handler & sendRawTransaction will be the only tx sending endpoint I believe for the demo

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. Designated it as a test endpoint.

Comment on lines 160 to 163
!!tx &&
!!this.toAddressWhitelist.length &&
!(tx.to in this.toAddressWhitelist) &&
tx.from !== this.deployAddress
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@karlfloersch and I recently discussed that it would be a pretty awesome to be able to limit the destination to certain methods as well as addresses. In practice I guess that would look like adding a methodIdWhitelist for each toAddressWhitelist and comparing first 4 bytes of tx.data here.

I don't think it's critical--but if not too hard to add, could be a valuable extra level of defense 😀

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added an RPC method whitelist, but not a tx data methodId whitelist. Would take a bit of time to configure, so leaving that as a nice-to-have if we get to it.

@willmeister willmeister merged commit d745c3c into master Apr 27, 2020
@willmeister willmeister deleted the YAS-330/ThrottlingAndRouting branch April 27, 2020 18:59
snario pushed a commit that referenced this pull request Apr 14, 2021
* rm unused stateTransitionIndex
protolambda pushed a commit to protolambda/optimism that referenced this pull request May 1, 2022
…t-L1-attribs-auth

Add missing auth test to L1Block spec
ClaytonNorthey92 added a commit to hemilabs/optimism that referenced this pull request Jun 27, 2024
b5b564702 popm/wasm: add wasm-opt target to optimise wasm binary (ethereum-optimism#146)
27a5081e3 tbc: allow seeds to be overwritten (ethereum-optimism#158)
18afbc403 Fix missing panic (ethereum-optimism#156)
05cee0afb tbc: remove height as the terminal condition for indexers (ethereum-optimism#152)
4402060d6 Use hemilabs/websocket fork of nhooyr.io/websocket (ethereum-optimism#153)
3df5001c4 Add initial CODEOWNERS file (ethereum-optimism#149)
a4685a57f popm/wasm: improve and tidy up Go code (ethereum-optimism#144)
41a0009c1 Add forking detection to TBC (ethereum-optimism#101)
bbeed8bac ignore ulimits in tbc when on localnet (ethereum-optimism#142)
4e1914cc6 Stopgap to re-fetch blocks with no children (ethereum-optimism#131)
12eefff06 e2e: fix issues detected by staticcheck (ethereum-optimism#134)
80153372f Add more documentation for TBC and related RPC protocol (ethereum-optimism#86)
723b63704 bfg: fix issues reported by staticcheck (ethereum-optimism#132)
509fb1be6 electrumx: fix unhandled error in NewJSONRPCRequest (ethereum-optimism#133)
b19af1e1f hemictl: use sort.Strings(...) instead of sort.Sort(sort.StringSlice(...)) (S1032) (ethereum-optimism#123)
cf7c570f9 popmd: remove unused 'handle' func and return err in connectBFG (ethereum-optimism#124)
fa2a4c4de ci: fix bug that makes version type always 'unstable' (ethereum-optimism#125)
ddc39655a ci: use org DOCKERHUB_TOKEN secret (ethereum-optimism#128)
6457d16f5 bfgd: drop btc_blocks_can refresh triggers for l2_keystones/pop_basis (ethereum-optimism#120)
2b3c096bd popm: simplify receivedKeystones variable in tests (ethereum-optimism#116)
6140bbf24 tbc: move RPC tests to a separate test file (ethereum-optimism#114)
c76571ff0 Add configuration option for static fees to web PoP miner (ethereum-optimism#119)
233817e65 popm: use simple conversion instead of unnecessary fmt.Sprintf (S1025) (ethereum-optimism#115)
2879f5fa7 e2e: improve name for variable with type `time.Duration` (ST1011) (ethereum-optimism#117)
4f31965e6 database,e2e: remove use of deprecated io/ioutil (SA1019) (ethereum-optimism#118)
e9e090696 tbc: do not recreate outpoint for ScriptHashByOutpoint. (ethereum-optimism#109)
52eefb136 ignore l2 keystones notifications in tests (ethereum-optimism#112)
eb607994c Sync Docker image environment variables with daemon configs (ethereum-optimism#111)
57281bc5d added a way to monitor and sanity-test localnet (ethereum-optimism#106)
ea1a1c94c bfgd: fix loops unconditionally exited after one interation (SA4004) (ethereum-optimism#108)
29f116fb4 Add pprof http server to daemons (ethereum-optimism#105)
18a315d7e Added README for generating forks in BTC regtest for TBC fork resolution testing (ethereum-optimism#100)
55b8f52cb more robust nextPort (ethereum-optimism#103)
48f8b293f tbcapi: use reverse byte order for hashes in serialised, tidy up (ethereum-optimism#104)
b7e9f5ecb restart initialblocks on-failure (ethereum-optimism#102)
f9d52d423 Update required Go version to v1.22.2 (ethereum-optimism#96)
f6808aa5b Fix op-proposer op-node dependency condition (fixes ethereum-optimism#98) (ethereum-optimism#99)
a882529e8 Kill all pending block downloads if a peer fails (ethereum-optimism#95)
eb66345e1 e2e: tidy up docker-compose file (ethereum-optimism#81)
34766f725 Track pending blocks with ttl package (ethereum-optimism#90)
6ed3eb88b Added configurable fee-per-vB to PoP Miner (ethereum-optimism#91)
509e31fbc Replace os.Kill with syscall.SIGTERM in signal.Notify calls (ethereum-optimism#87)

git-subtree-dir: heminetwork
git-subtree-split: b5b564702e8d3bedcdf0e0a52c22e383d7fd4dbe
bap2pecs pushed a commit to babylonlabs-io/optimism that referenced this pull request Jul 31, 2024
github-merge-queue bot pushed a commit that referenced this pull request Oct 22, 2024
…rc20 implementation (#12476)

* feat: add superchain erc20 bridge (#61)

* feat: add superchain erc20 bridge

* fix: interfaces and versions

* refactor: optimism superchain erc20 redesign (#62)

* refactor: use oz upgradeable erc20 as dependency

* chore: update interfaces

* fix: tests based on changes

* refactor: remove op as dependency

* feat: add check for supererc20 bridge on modifier

* chore: update tests and interfaces

* chore: update stack vars name on test

* chore: remove empty gitmodules file

* chore: update superchain weth errors

* test: add superchain erc20 bridge tests (#65)

* test: add superchain erc20 bridge tests

* test: add optimism superchain erc20 beacon tests

* test: remove unnecessary test

* test: tests fixes

* test: tests fixes

* chore: update missing bridge on natspec (#69)

* chore: update missing bridge on natspec

* fix: natspecs

---------

Co-authored-by: agusduha <[email protected]>

* fix: remove superchain erc20 base (#70)

* refactor: update isuperchainweth (#71)


---------

Co-authored-by: agusduha <[email protected]>

* feat: rename mint/burn and add SuperchainERC20 (#74)

* refactor: rename mint and burn functions on superchain erc20

* chore: rename optimism superchain erc20 to superchain erc20

* feat: create optimism superchain erc20 contract

* chore: update natspec and errors

* fix: superchain erc20 tests

* refactor: make superchain erc20 abstract

* refactor: move storage and erc20 metadata functions to implementation

* chore: update interfaces

* chore: update superchain erc20 events

* fix: tests

* fix: natspecs

* fix: add semmver lock and snapshots

* fix: remove unused imports

* fix: natspecs

---------

Co-authored-by: 0xDiscotech <[email protected]>

* fix: refactor zero check (#76)

* fix: pre pr

* chore: add new solady version and import it for erc20

* fix: undo forge std changes

* chore: re run pre pr script

* fix: semver natspec check failure (#79)

* fix: semver natspec check failure

* fix: ignore mock contracts in semver natspec script

* fix: error message

* feat: add crosschain erc20 interface (#80)

* feat: add crosschain erc20 interface

* fix: refactor interfaces

* fix: superchain bridge natspec (#83)

* fix: superchain weth natspec (#84)

Co-authored-by: 0xng <[email protected]>
Co-authored-by: 0xParticle <[email protected]>
Co-authored-by: gotzenx <[email protected]>

* fix: stop inheriting superchain interfaces (#85)

* fix: stop inheriting superchain interfaces

* fix: move events and erros into the implementation

* fix: make superchainERC20 inherits from crosschainERC20

* fix: superchain bridge rename (#86)

* fix: fee vault compiler error (#87)

* fix: remove unused imports

* chore: run pre-pr and update vendor interface

* fix: refactor common errors (#90)

* fix: refactor common errors

* fix: remove unused version

* feat: add permit2 on optimism superchain erc20

* chore: run pre-pr script

* fix: reuse unauthorized error (#92)

* fix: superchain erc20 factory conflicts

* fix: rename crosschain functions (#94)

* chore: run pre-pr

* chore: run pre-pr

* chore: run pre-pr

* feat: add new tests on optimism superchain erc20

* fix: vars and params naming on newly added tests

* fix: var name

* feat: support permit2 on optimism superchain erc20 and upgrade solady's erc20 implementation (#97)


---
Co-Authored-by: AgusDuha <[email protected]>

* chore: use ierc20 alias for ierc20 solady interface (#108)

---------

Co-authored-by: AgusDuha <[email protected]>
Co-authored-by: agusduha <[email protected]>
Co-authored-by: 0xng <[email protected]>
Co-authored-by: 0xParticle <[email protected]>
Co-authored-by: gotzenx <[email protected]>
samlaf pushed a commit to samlaf/optimism that referenced this pull request Nov 10, 2024
…rc20 implementation (ethereum-optimism#12476)

* feat: add superchain erc20 bridge (ethereum-optimism#61)

* feat: add superchain erc20 bridge

* fix: interfaces and versions

* refactor: optimism superchain erc20 redesign (ethereum-optimism#62)

* refactor: use oz upgradeable erc20 as dependency

* chore: update interfaces

* fix: tests based on changes

* refactor: remove op as dependency

* feat: add check for supererc20 bridge on modifier

* chore: update tests and interfaces

* chore: update stack vars name on test

* chore: remove empty gitmodules file

* chore: update superchain weth errors

* test: add superchain erc20 bridge tests (ethereum-optimism#65)

* test: add superchain erc20 bridge tests

* test: add optimism superchain erc20 beacon tests

* test: remove unnecessary test

* test: tests fixes

* test: tests fixes

* chore: update missing bridge on natspec (ethereum-optimism#69)

* chore: update missing bridge on natspec

* fix: natspecs

---------

Co-authored-by: agusduha <[email protected]>

* fix: remove superchain erc20 base (ethereum-optimism#70)

* refactor: update isuperchainweth (ethereum-optimism#71)


---------

Co-authored-by: agusduha <[email protected]>

* feat: rename mint/burn and add SuperchainERC20 (ethereum-optimism#74)

* refactor: rename mint and burn functions on superchain erc20

* chore: rename optimism superchain erc20 to superchain erc20

* feat: create optimism superchain erc20 contract

* chore: update natspec and errors

* fix: superchain erc20 tests

* refactor: make superchain erc20 abstract

* refactor: move storage and erc20 metadata functions to implementation

* chore: update interfaces

* chore: update superchain erc20 events

* fix: tests

* fix: natspecs

* fix: add semmver lock and snapshots

* fix: remove unused imports

* fix: natspecs

---------

Co-authored-by: 0xDiscotech <[email protected]>

* fix: refactor zero check (ethereum-optimism#76)

* fix: pre pr

* chore: add new solady version and import it for erc20

* fix: undo forge std changes

* chore: re run pre pr script

* fix: semver natspec check failure (ethereum-optimism#79)

* fix: semver natspec check failure

* fix: ignore mock contracts in semver natspec script

* fix: error message

* feat: add crosschain erc20 interface (ethereum-optimism#80)

* feat: add crosschain erc20 interface

* fix: refactor interfaces

* fix: superchain bridge natspec (ethereum-optimism#83)

* fix: superchain weth natspec (ethereum-optimism#84)

Co-authored-by: 0xng <[email protected]>
Co-authored-by: 0xParticle <[email protected]>
Co-authored-by: gotzenx <[email protected]>

* fix: stop inheriting superchain interfaces (ethereum-optimism#85)

* fix: stop inheriting superchain interfaces

* fix: move events and erros into the implementation

* fix: make superchainERC20 inherits from crosschainERC20

* fix: superchain bridge rename (ethereum-optimism#86)

* fix: fee vault compiler error (ethereum-optimism#87)

* fix: remove unused imports

* chore: run pre-pr and update vendor interface

* fix: refactor common errors (ethereum-optimism#90)

* fix: refactor common errors

* fix: remove unused version

* feat: add permit2 on optimism superchain erc20

* chore: run pre-pr script

* fix: reuse unauthorized error (ethereum-optimism#92)

* fix: superchain erc20 factory conflicts

* fix: rename crosschain functions (ethereum-optimism#94)

* chore: run pre-pr

* chore: run pre-pr

* chore: run pre-pr

* feat: add new tests on optimism superchain erc20

* fix: vars and params naming on newly added tests

* fix: var name

* feat: support permit2 on optimism superchain erc20 and upgrade solady's erc20 implementation (ethereum-optimism#97)


---
Co-Authored-by: AgusDuha <[email protected]>

* chore: use ierc20 alias for ierc20 solady interface (ethereum-optimism#108)

---------

Co-authored-by: AgusDuha <[email protected]>
Co-authored-by: agusduha <[email protected]>
Co-authored-by: 0xng <[email protected]>
Co-authored-by: 0xParticle <[email protected]>
Co-authored-by: gotzenx <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants