vincijy / ethereum2-etl Goto Github PK

View Code? Open in Web Editor NEW

Python scripts for ETL (extract, transform and load) jobs for Ethereum 2.0 beacon blocks, attestations, deposits, slashings, validators, committees. Data is available in Google BigQuery

License: MIT License

Python 99.58% Dockerfile 0.42%

ethereum2-etl's Introduction

Ethereum 2.0 ETL

Ethereum 2.0 ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Do you just want to query Ethereum data right away? Use the public dataset in BigQuery.

Full documentation available here.

Quickstart

Install Ethereum 2.0 ETL:

pip install ethereum2-etl

Sync your own node or request access to node on Infura.

Export beacon blocks, attestations, deposits, proposer slashings, attester slashings, voluntary exits (JSON Schema, CSV Schema, Reference):

> ethereum2etl export_beacon_blocks --start-block 0 --end-block 200 \
--output-dir output --output-format json \
--provider-uri https://projectid:[email protected]

Export beacon validators (JSON Schema, CSV Schema, Reference):

> ethereum2etl export_beacon_validtors --epoch 10 \
--output-dir output --output-format json \
--provider-uri https://projectid:[email protected]

Export beacon committees (JSON Schema, CSV Schema, Reference):

> ethereum2etl export_beacon_committees --start-epoch 0 --end-epoch 10 \
--output-dir output --output-format json \
--provider-uri https://projectid:[email protected]

Find other commands here. Read this article for example SQL queries.

For the latest version, check out the repo and call

> pip install -e . 
> python ethereum2etl.py

Useful Links

Running Tests

> pip install -e .[dev]
> export ETHEREUM2ETL_PROVIDER_URI=https://projectid:[email protected]
> pytest -vv

Running Tox Tests

> pip install tox
> tox

Running in Docker

Install Docker https://docs.docker.com/install/

Build a docker image

 > docker build -t ethereum2-etl:latest .
 > docker image ls

Run a container out of the image

 > docker run -v $HOME/output:/ethereum2-etl/output ethereum2-etl:latest export_beacon_blocks -s 0 -e 200 -p https://projectid:[email protected]
 > docker run -v $HOME/output:/ethereum2-etl/output ethereum2-etl:latest export_beacon_blocks -s 2020-08-04 -e 2020-08-05 -p https://projectid:[email protected]