klaytn / klaytn-etl

Python scripts for ETL (extract, transform and load) jobs for Klaytn blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions.

Klaytn ETL

Klaytn ETL lets you convert Klaytn blockchain data into convenient formats like JSONs, CSVs and relational databases. This is a fork of Ethereum ETL.

Full documentation available here.

Notice: Klaytn ETL is still on the beta version. However, CLIs are all functional.

Quickstart

Install Klaytn ETL:

pip3 install klaytn-etl-cli

Export blocks and transactions

> klaytnetl export_blocks_and_transactions --start-block 0 --end-block 5000 \
--blocks-output blocks.json --transactions-output transactions.json

Export ERC20 and ERC721 transfers

> klaytnetl export_token_transfers --start-block 0 --end-block 5000 \
--output token_transfers.json

Export traces

> klaytnetl export_traces --start-block 0 --end-block 5000 \
--output traces.json

Find other commands here.

For the latest version, check out the repo and call

> pip3 install -e . 
> python3 klaytnetl.py

Running in Docker

1. Install Docker https://docs.docker.com/install/

2. Build a docker image

   > docker build -t klaytn-etl:latest .
   > docker image ls

3. Run a container out of the image

   > docker run -v $HOME/output:/klaytn-etl/output klaytn-etl:latest export_all -s 0 -e 5499999 -b 100000
   > docker run -v $HOME/output:/klaytn-etl/output klaytn-etl:latest export_all -s 2018-01-01 -e 2018-01-01

klaytn-etl's Issues

evmdasm.disassembler [ERROR] - invalid instruction: PUSH29

I found stdout that looks like error. Can we confirm this line is critical error?

The script I executed is here.

#!/bin/bash

ETL_PROVIDER_URL=http://a.b.c.d:8551/
ETL_AMOUNT=10000
ETL_START_BLOCK=70024000
ETL_END_BLOCK=$(($ETL_START_BLOCK+$ETL_AMOUNT-1))

python klaytnetl.py export_trace_group --start-block $ETL_START_BLOCK --end-block $ETL_END_BLOCK --provider-uri $ETL_PROVIDER_URL --enrich True -b 1 -w 5 -t 300 --file-format json --file-maxlines 10000 --traces-output klaytn/raw/traces/network_type_name=cypress/ --contracts-output klaytn/raw/contracts/network_type_name=cypress/ --tokens-output klaytn/raw/tokens/network_type_name=cypress/ --log-percentage-step 1

The stdout is here.

Symbolic Execution not available: No module named 'mythril'
pyetherchain not available: No module named 'pyetherchain'
2022-10-02 08:26:28,204 - ProgressLogger [INFO] - Started work. Items to process: 10000.
2022-10-02 08:26:29,834 - ProgressLogger [INFO] - 100 items processed. Progress is 1%.
2022-10-02 08:26:31,470 - ProgressLogger [INFO] - 200 items processed. Progress is 2%.
2022-10-02 08:26:32,875 - ProgressLogger [INFO] - 300 items processed. Progress is 3%.
2022-10-02 08:26:34,219 - ProgressLogger [INFO] - 400 items processed. Progress is 4%.
2022-10-02 08:26:35,835 - ProgressLogger [INFO] - 500 items processed. Progress is 5%.
2022-10-02 08:27:08,235 - ProgressLogger [INFO] - 600 items processed. Progress is 6%.
2022-10-02 08:27:34,985 - ProgressLogger [INFO] - 700 items processed. Progress is 7%.
2022-10-02 08:28:01,963 - ProgressLogger [INFO] - 800 items processed. Progress is 8%.
2022-10-02 08:28:25,313 - ProgressLogger [INFO] - 900 items processed. Progress is 9%.
2022-10-02 08:28:51,527 - ProgressLogger [INFO] - 1000 items processed. Progress is 10%.
2022-10-02 08:29:17,003 - ProgressLogger [INFO] - 1100 items processed. Progress is 11%.
2022-10-02 08:29:42,574 - ProgressLogger [INFO] - 1200 items processed. Progress is 12%.
2022-10-02 08:30:07,880 - ProgressLogger [INFO] - 1300 items processed. Progress is 13%.
2022-10-02 08:30:28,666 - ProgressLogger [INFO] - 1400 items processed. Progress is 14%.
2022-10-02 08:30:52,595 - ProgressLogger [INFO] - 1500 items processed. Progress is 15%.
2022-10-02 08:31:17,934 - ProgressLogger [INFO] - 1600 items processed. Progress is 16%.
2022-10-02 08:31:41,702 - ProgressLogger [INFO] - 1700 items processed. Progress is 17%.
2022-10-02 08:32:05,170 - ProgressLogger [INFO] - 1800 items processed. Progress is 18%.
2022-10-02 08:32:25,646 - ProgressLogger [INFO] - 1900 items processed. Progress is 19%.
2022-10-02 08:32:48,561 - ProgressLogger [INFO] - 2000 items processed. Progress is 20%.
2022-10-02 08:33:09,279 - ProgressLogger [INFO] - 2100 items processed. Progress is 21%.
2022-10-02 08:33:31,779 - ProgressLogger [INFO] - 2200 items processed. Progress is 22%.
2022-10-02 08:33:54,341 - ProgressLogger [INFO] - 2300 items processed. Progress is 23%.
2022-10-02 08:34:18,236 - ProgressLogger [INFO] - 2400 items processed. Progress is 24%.
2022-10-02 08:34:42,578 - ProgressLogger [INFO] - 2500 items processed. Progress is 25%.
2022-10-02 08:35:05,253 - ProgressLogger [INFO] - 2600 items processed. Progress is 26%.
2022-10-02 08:35:28,943 - ProgressLogger [INFO] - 2700 items processed. Progress is 27%.
2022-10-02 08:35:57,595 - ProgressLogger [INFO] - 2800 items processed. Progress is 28%.
2022-10-02 08:36:12,488 - evmdasm.disassembler [ERROR] - invalid instruction: PUSH29
2022-10-02 08:36:22,670 - ProgressLogger [INFO] - 2900 items processed. Progress is 29%.
2022-10-02 08:36:36,582 - evmdasm.disassembler [ERROR] - invalid instruction: PUSH29
2022-10-02 08:36:47,463 - ProgressLogger [INFO] - 3000 items processed. Progress is 30%.
2022-10-02 08:36:51,637 - evmdasm.disassembler [ERROR] - invalid instruction: PUSH29
2022-10-02 08:37:11,383 - ProgressLogger [INFO] - 3100 items processed. Progress is 31%.
2022-10-02 08:37:35,272 - ProgressLogger [INFO] - 3200 items processed. Progress is 32%.
2022-10-02 08:37:59,814 - ProgressLogger [INFO] - 3300 items processed. Progress is 33%.
2022-10-02 08:38:20,508 - ProgressLogger [INFO] - 3400 items processed. Progress is 34%.
2022-10-02 08:38:46,376 - ProgressLogger [INFO] - 3500 items processed. Progress is 35%.
2022-10-02 08:38:52,634 - evmdasm.disassembler [ERROR] - invalid instruction: PUSH20
2022-10-02 08:39:13,087 - ProgressLogger [INFO] - 3600 items processed. Progress is 36%.
2022-10-02 08:39:40,595 - evmdasm.disassembler [ERROR] - invalid instruction: PUSH22

Support stream

Support stream option

ERC 1155 Support

Support ERC 1155

New feature suggestion

I took a quick look at what klaytn-etl provides for chaindata processing and one feature suggestion was popped up.

The Klaytn block header contains Governance and Vote which are RLP-encoded data. If this project can help to parse as human-readable for post-processing of their favors, it would be leveraged by third parties that do business with data viewer, processing, etc.
The API governance.itemsAt may be leveraged to implement this feature by iteration of a query with every block number parameter.

I'm not sure this suggestion is out of this project's scope. However, kindly, the extension of features that fit Klaytn would be attractable rather than the ethereum-etl which provides only general stuff like fetching blocks and transactions.

Create documentation

Create documentation that could explain schemas/cli flags