Ethereum 2.0 ETL

Ethereum 2.0 ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Do you just want to query Ethereum data right away? Use the public dataset in BigQuery.

Full documentation available here.

Quickstart

Install Ethereum 2.0 ETL:

pip install ethereum2-etl

Sync your own node or request access to node on Infura.

Export beacon blocks, attestations, deposits, proposer slashings, attester slashings, voluntary exits (JSON Schema, CSV Schema, Reference):

> ethereum2etl export_beacon_blocks --start-block 0 --end-block 200 \
--output-dir output --output-format json \
--provider-uri https://projectid:[email protected]

Export beacon validators (JSON Schema, CSV Schema, Reference):

> ethereum2etl export_beacon_validtors --epoch 10 \
--output-dir output --output-format json \
--provider-uri https://projectid:[email protected]

Export beacon committees (JSON Schema, CSV Schema, Reference):

> ethereum2etl export_beacon_committees --start-epoch 0 --end-epoch 10 \
--output-dir output --output-format json \
--provider-uri https://projectid:[email protected]

Find other commands here. Read this article for example SQL queries.

For the latest version, check out the repo and call

> pip install -e . 
> python ethereum2etl.py

Useful Links

• Schema
• Command Reference
• Documentation
• Airflow DAGs

Running Tests

> pip install -e .[dev]
> export ETHEREUM2ETL_PROVIDER_URI=https://projectid:[email protected]
> pytest -vv

Running Tox Tests

> pip install tox
> tox

Running in Docker

1. Install Docker https://docs.docker.com/install/

2. Build a docker image

    > docker build -t ethereum2-etl:latest .
    > docker image ls
   

3. Run a container out of the image

    > docker run -v $HOME/output:/ethereum2-etl/output ethereum2-etl:latest export_beacon_blocks -s 0 -e 200 -p https://projectid:[email protected]
    > docker run -v $HOME/output:/ethereum2-etl/output ethereum2-etl:latest export_beacon_blocks -s 2020-08-04 -e 2020-08-05 -p https://projectid:[email protected]
   

Projects using Ethereum ETL

• Google - Public BigQuery Ethereum datasets
• Nansen - Blockchain analytics platform
• Medalla Data Challenge - analyses of the Ethereum 2.0 blockchain