blockchain-etl / bitcoin-etl

ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

Bitcoin ETL

Join Telegram Group

Install Bitcoin ETL:

pip install bitcoin-etl

Export blocks and transactions (Schema, Reference):

> bitcoinetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
--provider-uri http://user:pass@localhost:8332 --chain bitcoin \
 --blocks-output blocks.json --transactions-output transactions.json

Supported chains:

• bitcoin
• bitcoin_cash
• bitcoin_gold
• dogecoin
• litecoin
• dash
• zcash

Stream blockchain data continually to console (Reference):

> pip install bitcoin-etl[streaming]
> bitcoinetl stream -p http://user:pass@localhost:8332 --start-block 500000

Stream blockchain data continually to Google Pub/Sub (Reference):

> export GOOGLE_APPLICATION_CREDENTIALS=/path_to_credentials_file.json
> bitcoinetl stream -p http://user:pass@localhost:8332 --start-block 500000 --output projects/your-project/topics/crypto_bitcoin

For the latest version, check out the repo and call

> pip install -e .[streaming]
> python bitcoinetl.py

Table of Contents

• Bitcoin ETL
  • Table of Contents
  • Schema
    • blocks.json
    • transactions.json
    • transaction_input
    • transaction_output
  • Exporting the Blockchain
    • Running in Docker
    • Command Reference
      • export_blocks_and_transactions
      • enrich_transactions
      • get_block_range_for_date
      • export_all
      • stream
    • Running Tests
    • Running Tox Tests
    • Public Datasets in BigQuery

Schema

blocks.json

Field	Type
hash	hex_string
size	bigint
stripped_size	bigint
weight	bigint
number	bigint
version	bigint
merkle_root	hex_string
timestamp	bigint
nonce	hex_string
bits	hex_string
coinbase_param	hex_string
transaction_count	bigint

transactions.json

Field	Type
hash	hex_string
size	bigint
virtual_size	bigint
version	bigint
lock_time	bigint
block_number	bigint
block_hash	hex_string
block_timestamp	bigint
is_coinbase	boolean
index	bigint
inputs	[]transaction_input
outputs	[]transaction_output
input_count	bigint
output_count	bigint
input_value	bigint
output_value	bigint
fee	bigint

transaction_input

Field	Type
index	bigint
spent_transaction_hash	hex_string
spent_output_index	bigint
script_asm	string
script_hex	hex_string
sequence	bigint
required_signatures	bigint
type	string
addresses	[]string
value	bigint

transaction_output

Field	Type
index	bigint
script_asm	string
script_hex	hex_string
required_signatures	bigint
type	string
addresses	[]string
value	bigint

You can find column descriptions in schemas

Notes:

1. Output values returned by Dogecoin API had precision loss in the clients prior to version 1.14. It's caused by this issue dogecoin/dogecoin#1558 The explorers that used older versions to export the data may show incorrect address balances and transaction amounts.

2. For Zcash, vjoinsplit and valueBalance fields are converted to inputs and outputs with type 'shielded' https://zcash-rpc.github.io/getrawtransaction.html, https://zcash.readthedocs.io/en/latest/rtd_pages/zips/zip-0243.html

Exporting the Blockchain

1. Install python 3.5.3+ https://www.python.org/downloads/

2. Install Bitcoin node https://hackernoon.com/a-complete-beginners-guide-to-installing-a-bitcoin-full-node-on-linux-2018-edition-cb8e384479ea

3. Start Bitcoin. Make sure it downloaded the blocks that you need by executing $ bitcoin-cli getblockchaininfo in the terminal. You can export blocks below blocks, there is no need to wait until the full sync

4. Install Bitcoin ETL:

   > pip install bitcoin-etl

5. Export blocks & transactions:

   > bitcoinetl export_all --start 0 --end 499999  \
   --partition-batch-size 100 \
   --provider-uri http://user:pass@localhost:8332 --chain bitcoin

   The result will be in the output subdirectory, partitioned in Hive style:

   output/blocks/start_block=00000000/end_block=00000099/blocks_00000000_00000099.csv
   output/blocks/start_block=00000100/end_block=00000199/blocks_00000100_=00000199.csv
   ...
   output/transactions/start_block=00000000/end_block=00000099/transactions_00000000_00000099.csv
   ...

   In case bitcoinetl command is not available in PATH, use python -m bitcoinetl instead.

Running in Docker

1. Install Docker https://docs.docker.com/install/

2. Build a docker image

   > docker build --platform linux/x86_64 -t bitcoin-etl:latest .
   > docker image ls

3. Run a container out of the image

   > docker run --platform linux/x86_64 -v $HOME/output:/bitcoin-etl/output bitcoin-etl:latest export_blocks_and_transactions --start-block 0 --end-block 500000 \
       --provider-uri http://user:pass@localhost:8332 --blocks-output output/blocks.json --transactions-output output/transactions.json

4. Run streaming to console or Pub/Sub

   > docker build --platform linux/x86_64 -t bitcoin-etl:latest-streaming -f Dockerfile_with_streaming .
   > echo "Stream to console"
   > docker run --platform linux/x86_64 bitcoin-etl:latest-streaming stream -p http://user:pass@localhost:8332 --start-block 500000
   > echo "Stream to Pub/Sub"
   > docker run --platform linux/x86_64 -v /path_to_credentials_file/:/bitcoin-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/bitcoin-etl/credentials_file.json bitcoin-etl:latest-streaming stream -p http://user:pass@localhost:8332 --start-block 500000 --output projects/your-project/topics/crypto_bitcoin

5. Refer to https://github.com/blockchain-etl/bitcoin-etl-streaming for deploying the streaming app to Google Kubernetes Engine.

Command Reference

• export_blocks_and_transactions
• enrich_transactions
• get_block_range_for_date
• export_all
• stream

All the commands accept -h parameter for help, e.g.:

> bitcoinetl export_blocks_and_transactions --help
Usage: bitcoinetl.py export_blocks_and_transactions [OPTIONS]

  Export blocks and transactions.

Options:
  -s, --start-block INTEGER   Start block
  -e, --end-block INTEGER     End block  [required]
  -b, --batch-size INTEGER    The number of blocks to export at a time.
  -p, --provider-uri TEXT     The URI of the remote Bitcoin node
  -w, --max-workers INTEGER   The maximum number of workers.
  --blocks-output TEXT        The output file for blocks. If not provided
                              blocks will not be exported. Use "-" for stdout
  --transactions-output TEXT  The output file for transactions. If not
                              provided transactions will not be exported. Use
                              "-" for stdout
  --help                      Show this message and exit.

For the --output parameters the supported type is json. The format type is inferred from the output file name.

export_blocks_and_transactions

> bitcoinetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
  --provider-uri http://user:pass@localhost:8332 \
  --blocks-output blocks.json --transactions-output transactions.json

Omit --blocks-output or --transactions-output options if you want to export only transactions/blocks.

You can tune --batch-size, --max-workers for performance.

Note that required_signatures, type, addresses, and value fields will be empty in transactions inputs. Use enrich_transactions to populate those fields.

enrich_transactions

You need to run bitcoin daemon with option txindex=1 for this command to work.

> bitcoinetl enrich_transactions  \
  --provider-uri http://user:pass@localhost:8332 \
  --transactions-input transactions.json --transactions-output enriched_transactions.json

You can tune --batch-size, --max-workers for performance.

get_block_range_for_date

> bitcoinetl get_block_range_for_date --provider-uri http://user:pass@localhost:8332 --date=2017-03-01

This command is guaranteed to return the block range that covers all blocks with block.time on the specified date. However the returned block range may also contain blocks outside the specified date, because block times are not monotonic https://twitter.com/EvgeMedvedev/status/1073844856009576448. You can filter blocks.json/transactions.json with the below command:

> bitcoinetl filter_items -i blocks.json -o blocks_filtered.json \
-p "datetime.datetime.fromtimestamp(item['timestamp']).astimezone(datetime.timezone.utc).strftime('%Y-%m-%d') == '2017-03-01'"

export_all

> bitcoinetl export_all --provider-uri http://user:pass@localhost:8332 --start 2018-01-01 --end 2018-01-02

You can tune --export-batch-size, --max-workers for performance.

stream

> bitcoinetl stream --provider-uri http://user:pass@localhost:8332 --start-block 500000

• This command outputs blocks and transactions to the console by default.
• Use --output option to specify the Google Pub/Sub topic where to publish blockchain data, e.g. projects/your-project/topics/crypto_bitcoin. Blocks and transactions will be pushed to projects/your-project/topics/crypto_bitcoin.blocks and projects/your-project/topics/crypto_bitcoin.transactions topics.
• The command saves its state to last_synced_block.txt file where the last synced block number is saved periodically.
• Specify either --start-block or --last-synced-block-file option. --last-synced-block-file should point to the file where the block number, from which to start streaming the blockchain data, is saved.
• Use the --lag option to specify how many blocks to lag behind the head of the blockchain. It's the simplest way to handle chain reorganizations - they are less likely the further a block from the head.
• Use the --chain option to specify the type of the chain, e.g. bitcoin, litecoin, dash, zcash, etc.
• You can tune --period-seconds, --batch-size, --max-workers for performance.

Running Tests

> pip install -e .[dev]
> echo "The below variables are optional"
> export BITCOINETL_BITCOIN_PROVIDER_URI=http://user:pass@localhost:8332
> export BITCOINETL_LITECOIN_PROVIDER_URI=http://user:pass@localhost:8331
> export BITCOINETL_DOGECOIN_PROVIDER_URI=http://user:pass@localhost:8330
> export BITCOINETL_BITCOIN_CASH_PROVIDER_URI=http://user:pass@localhost:8329
> export BITCOINETL_DASH_PROVIDER_URI=http://user:pass@localhost:8328
> export BITCOINETL_ZCASH_PROVIDER_URI=http://user:pass@localhost:8327
> pytest -vv

Running Tox Tests

> pip install tox
> tox

Public Datasets in BigQuery

https://cloud.google.com/blog/products/data-analytics/introducing-six-new-cryptocurrencies-in-bigquery-public-datasets-and-how-to-analyze-them

bitcoin-etl's Issues

oops

rpc error of bitcoinetl enrich_transactions

@medvedev1088 Hi,
I want to get the values of transactions inputs. so I use enrich_transactions.
An error occurred at runtime:
File "..../bitcoinetl/rpc/bitcoin_rpc.py", line 53, in batch
raise ValueError('"result" is None in the JSON RPC response {}. Request: {}', resp_item.get('error'), text)
ValueError: ('"result" is None in the JSON RPC response {}. Request: {}', {'code': -5, 'message': 'No such mempool transaction. Use -txindex or provide a block hash to enable blockchain transaction queries. Use gettransaction for wallet transactions.'},
'[{"jsonrpc": "2.0", "method": "getrawtransaction", "params": ["899652e48f685736a1b9af0d2db2e64a906d74064b79650c173b85073aac5d4f", 1], "id": "1"},
...
It dosen't seem to be a problem of bitcoin core.
I will appreciate it if you can give me some advice!
Thank you very much!!

Factor out bitcoin_rpc.py to a separate lib

Implement export_all command

Should partition by blocks or by date, same as Ethereum ETL's export_all https://github.com/blockchain-etl/ethereum-etl/blob/develop/ethereumetl/cli/export_all.py.

The information of input is lost

The JSON files are not supported to upload as an attachment. I captured part of a file. All the input arrays in the files downloaded are empty. What's the problem?

About nonstandard output

@medvedev1088
Hi,
I found that there ara two kinds of output in transactions.json.
One is nonstandard, and the other is pubkeyhash.
I thought the nonstandard output is the op_return output, but i found outputs of many (not all ) coinbase txs also are nonstandard. And address of miner is like“nonstandard3318537dfb3135df9f3d950dbdf8a7ae68dd7c7d”.
Here is an example:
transaction: {
"hash": "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b", "block_number": 0,
"block_hash": "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f", "is_coinbase": true,
"outputs": [{"index": 0, "script_asm": "04678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5f OP_CHECKSIG", "script_hex": "4104678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5fac", "required_signatures": null, "type": "nonstandard", "addresses": ["nonstandard3318537dfb3135df9f3d950dbdf8a7ae68dd7c7d"], "value": 5000000000}], "input_count": 0, "output_count": 1, "input_value": 0, "output_value": 5000000000, "fee": 0}
The address above is 1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa according to [https://www.blockchain.com/btc/tx/4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b]

I'm confused. Perhaps you know why?
Thank you very much!!!!

pip install bitcoin-etl error

This is the error:
$ pip install bitcoin-etl
Collecting bitcoin-etl
Could not find a version that satisfies the requirement bitcoin-etl (from versions: )
No matching distribution found for bitcoin-etl

Support of REST API "block" for the coins with old APIs.

As you have already implemented, getblock JSON PRC has 2 versions.
Old one doesn't support verbose option, you need to call getrawtransaction RPC to get transaction data.

As long as I know, REST API "blocks" with ".json" extension returns transaction data even if the Coins don't support verbose option on their JSON RPC.
https://github.com/bitcoin/bitcoin/blob/master/doc/REST-interface.md#blocks

How about supporting REST API "block"?
I know it is not elegant, but it has an advantage on speed.

To make REST API enable, you need add args "-rest" with your bitcoind.

Zcash dataset ignores Sapling pool when calculating fees

For transactions containing Sapling shielded inputs or outputs, a single fake input or output is added to the transaction representing the net migration of funds into or out of the Sapling pool.

bitcoin-etl/bitcoinetl/service/btc_service.py

Lines 177 to 187 in 23877a5

	if transaction.value_balance is not None and transaction.value_balance != 0:
	if transaction.value_balance > 0:
	input = BtcTransactionInput()
	input.type = ADDRESS_TYPE_SHIELDED
	input.value = transaction.value_balance
	transaction.add_input(input)
	if transaction.value_balance < 0:
	output = BtcTransactionOutput()
	output.type = ADDRESS_TYPE_SHIELDED
	output.value = transaction.value_balance
	transaction.add_output(output)

However, this fake input or output is not included in the fee calculation, causing fees to appear massively inflated in some cases. Looking at the transaction implementation, the fee should include the fake input or output:

bitcoin-etl/bitcoinetl/domain/transaction.py

Lines 45 to 71 in 23877a5

	def add_input(self, input):
	if len(self.inputs) > 0:
	input.index = self.inputs[len(self.inputs) - 1].index + 1
	self.inputs.append(input)
	else:
	input.index = 0
	self.inputs.append(input)
	def add_output(self, output):
	if len(self.outputs) > 0:
	output.index = self.outputs[len(self.outputs) - 1].index + 1
	self.outputs.append(output)
	else:
	output.index = 0
	self.outputs.append(output)
	def calculate_input_value(self):
	return sum([input.value for input in self.inputs if input.value is not None])
	def calculate_output_value(self):
	return sum([output.value for output in self.outputs if output.value is not None])
	def calculate_fee(self):
	if self.is_coinbase:
	return 0
	else:
	return self.calculate_input_value() - self.calculate_output_value()

My guess is that this is an ordering problem: the fee is being cached somewhere before the fake input or output is added. I can't see where that is happening though.

As an example, BigQuery says that transaction 13d78b6036aa5be6646c4fa96e3a779eb03f82fdc226e7cf82a9197d28d03a6b has a fee of 294206950000 zatoshis, when it was really 0.0001 ZEC (10,000 zatoshis) according to a blockchain explorer. However, it returns inputs.value = 147103480000 and outputs.value = -147103470000, and adding these together gives the real fee.

can't decode BTC address

Hello,
anyone can help to fix this:

x
Out[178]: <bitcoinetl.domain.block.BtcBlock at 0x7f71d337b780>
x.transactions[1].outputs[1].addresses
Out[179]: ['nonstandardafad2456e5fd6732199d77546c9526255fa66b77']

But bitcoin-cli returns everything correct

root@b:~# bitcoin-cli getrawtransaction edd9212daaeb9ac51ce6f5a8b7ff096c74d4aada71615beb9cd1e0858d5ebe60 1
{
  "txid": "edd9212daaeb9ac51ce6f5a8b7ff096c74d4aada71615beb9cd1e0858d5ebe60",
  "hash": "5361e2dd2475f86433fd7cf409ca96884a9f53bfcd20bda16cdcdeaf23d74ca0",
  "version": 1,
  "size": 382,
  "vsize": 192,
  "weight": 766,
  "locktime": 0,
  "vin": [
    {
      "txid": "e960fdabe8e301846d64b924766ffc4aba94a8fb056c096e7e6c9b631f5e344e",
      "vout": 0,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "txinwitness": [
        "",
        "304402200efe981c24a248b5ee2957a06d592220999a33eed783f93384f06b059e5613fd0220085b445a1638ded282aad9dd49120df2d954a4d307266967fbcad2e54f57328c01",
        "30440220452e0f0bfbba23e93cc265b04fd9c96cf88d9d1ccc49370444b46ef35f6a199802202507471dc999f2bd8bf4ea16d63a71c02153398002b685365228cf9e99d9c81501",
        "52210279d1f38c1c80d47cb00ddbbe2915a60d5706e1ef66056a169150f083b288eb952102cb7d02b654f8616bfc5ab017b7a3ec9092e466381af0f552b7efcd8d920453672103c96d495bfdd5ba4145e3e046fee45e84a8a48ad05bd8dbb395c011a32cf9f88053ae"
      ],
      "sequence": 4294967295
    }
  ],
  "vout": [
    {
      "value": 0.00913448,
      "n": 0,
      "scriptPubKey": {
        "asm": "OP_DUP OP_HASH160 1683db0ca249360942d1a9dabe5d0cced0c5a903 OP_EQUALVERIFY OP_CHECKSIG",
        "hex": "76a9141683db0ca249360942d1a9dabe5d0cced0c5a90388ac",
        "address": "1343piJzM5DB4N3gDE7VF17nnRD5FVj4pr",
        "type": "pubkeyhash"
      }
    },
    {
      "value": 0.00682552,
      "n": 1,
      "scriptPubKey": {
        "asm": "0 701a8d401c84fb13e6baf169d59684e17abd9fa216c8cc5b9fc63d622ff8c58d",
        "hex": "0020701a8d401c84fb13e6baf169d59684e17abd9fa216c8cc5b9fc63d622ff8c58d",
        "address": "bc1qwqdg6squsna38e46795at95yu9atm8azzmyvckulcc7kytlcckxswvvzej",
        "type": "witness_v0_scripthash"
      }
    }
  ],
  "hex": "010000000001014e345e1f639b6c7e6e096c05fba894ba4afc6f7624b9646d8401e3e8abfd60e90000000000ffffffff0228f00d00000000001976a9141683db0ca249360942d1a9dabe5d0cced0c5a90388ac386a0a0000000000220020701a8d401c84fb13e6baf169d59684e17abd9fa216c8cc5b9fc63d622ff8c58d040047304402200efe981c24a248b5ee2957a06d592220999a33eed783f93384f06b059e5613fd0220085b445a1638ded282aad9dd49120df2d954a4d307266967fbcad2e54f57328c014730440220452e0f0bfbba23e93cc265b04fd9c96cf88d9d1ccc49370444b46ef35f6a199802202507471dc999f2bd8bf4ea16d63a71c02153398002b685365228cf9e99d9c815016952210279d1f38c1c80d47cb00ddbbe2915a60d5706e1ef66056a169150f083b288eb952102cb7d02b654f8616bfc5ab017b7a3ec9092e466381af0f552b7efcd8d920453672103c96d495bfdd5ba4145e3e046fee45e84a8a48ad05bd8dbb395c011a32cf9f88053ae00000000",
  "blockhash": "000000000000000000086a2cff995563e22ed9ed046dc8b56690130893d6175c",
  "confirmations": 1,
  "time": 1637151616,
  "blocktime": 1637151616
}

so correct address for this transaction should be: 1343piJzM5DB4N3gDE7VF17nnRD5FVj4pr

Implement GCSExporter

GCSExporter outputs blocks with transactions to a file (one per block) and uploads it to a GCS bucket. It should have the same interface as https://github.com/blockchain-etl/bitcoin-etl/blob/master/blockchainetl/jobs/exporters/google_pubsub_item_exporter.py.

It will be used for streaming e.g.

bitcoinetl stream -p http://user:pass@localhost:8332 --start-block 500000 --output gs://bucket_name/folder

The bucket can then be configured to trigger events to Cloud Functions or Pub/Sub.

ETL process needs upgrade to work for Bitcoin Core 22.0

From https://github.com/bitcoin/bitcoin the bitcoin core has been making changes in 22.0,
rpc: remove deprecated addresses and reqSigs from rpc outputs via this commit bitcoin/bitcoin@8721638

This change in the core 22.0 require the bitcoin-etl to make the corresponding changes.

macOS installation ERROR

environment:
macOS | python3.8 | pip 21.0.1

Running installation:
pip install bitcoin-etl

Error:

ERROR: Could not find a version that satisfies the requirement bitcoin-etl 
ERROR: No matching distribution found for bitcoin-etl

Any suggestions? Thanks!!!

Support for Litecoin

Confirm support for Litecoin

Dataset data is not equal with bitcoinetl stream

Address 15pHcPCSw9DfkJTD8uQY4okzrayNm9LiA6 in transaction 0b1f0fee9ac088422531af0cf18f9b24024f783ff5426a6b683c2d5082fce652 is not correctly parsed in stream

bitcoinetl stream  --start-block 134916 --end-block 134916 -p http://node  -B 1

it is parsed as nonstandard897bc364defc7cb7b0e4382dc605c897c7707f35

but is correctly stated in the google cloud dataset.

How is this done?

Are these issues related?
#55 #43

Adjust dependency versions

Hi,

I noted that the current, fixed dependency versions of requests, dateutil and click clash with dependencies for e.g. pandas.

Would it be possible to adjust the Python dependency definitions in setup.py?

Locally I've successfully tested

'requests ~= 2.20',
'python-dateutil ~= 2.7',
'click ~= 7.0'

Implement ETL scripts for blocks and transactions

The scope of this task is:

1. Implement a job for exporting blocks and transactions using export_blocks_and_transactions.py as a basis.
2. Implement tests using test_export_blocks_job.py as a basis.
3. Implement Dockerfile
4. Write documentation in README using Exporting the Blockchain as a basis.

Don't worry about copy-pasting the code from Ethereum ETL for now. Common code will be factored out later.

dependency conflict, 21 april 2024

Error message:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
cloudinary 1.40.0 requires urllib3>=1.26.5, but you have urllib3 1.24.3 which is incompatible.

---

Steps to reproduce error:

Step 1:
pip install bitcoin-etl[streaming]

Error will get:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
cloudinary 1.40.0 requires urllib3>=1.26.5, but you have urllib3 1.24.3 which is incompatible.

Try to solve error:
i use: pip install urllib3 --upgrade,

then another dependency conflict arise

after so many tries to installing various dependecy version,

when finally i run pip install bitcoin-etl[streaming] i get no error, but after that

on running bitcoinetl stream -p http://user:pass@localhost:8332 --start-block 840000

i get error: ModuleNotFoundError: No module named 'urllib3.packages.six.moves'

this all about dependecny conflict,

i dont know much and dont have much experience if there is any mistake writing in post, i am sorry

transaction_input.type & transaction_input.value are null

On my environment, the field "type" & "value" at transaction_input are always null.
like,
"inputs": [{"index": 0, "spent_transaction_hash": "100a776847c80201ca6efa5605b806ec770c105af95972eb522e45af385b3892", "spent_output_index": 0, "script_asm": "3045022100b61b8132060e810387518dbd697f731eb1680debf3882ed04821a5d54ced2bca022017a5356096f7db93d9b989114e248476d37325aab7ebc2a8f5800a2e9e6a6618[ALL] 047bade8836f9ccd1923851351e70683606d86a22b6c78676ea3eae343aff5c8c3964e39c9edcf11807949bd325c9475b14de3dcd91b354a7e9d64a0e9b3e10f6b", "script_hex": "483045022100b61b8132060e810387518dbd697f731eb1680debf3882ed04821a5d54ced2bca022017a5356096f7db93d9b989114e248476d37325aab7ebc2a8f5800a2e9e6a66180141047bade8836f9ccd1923851351e70683606d86a22b6c78676ea3eae343aff5c8c3964e39c9edcf11807949bd325c9475b14de3dcd91b354a7e9d64a0e9b3e10f6b", "type": null, "sequence": 4294967295, "value": null},

I know, Getting these values requires another RPC "getrawtransaction" and bitcoin-cli with "getrawtransaction" returns these values on my environment.
Has "getrawtransaction" not been implemented?

I'm using bitcoinetl, version 1.1.0, installed from pip.

Provide item_id for streaming to avoid duplicates in BigQuery

See ethereum-etl for an example https://github.com/blockchain-etl/ethereum-etl/blob/develop/ethereumetl/streaming/eth_streamer_adapter.py#L96.

I got ValueError('"result" is None in the JSON RPC response {}. Request: {}', resp_item.get('error'), text)

Hi there,

Although I succeeded bittcoinetl get_block_range_for_date command, I got following errors and failed when I run bitcoinetl export_all command.

$ bitcoinetl export_all -s 0 -e 573154 -b 100 -p http://bitcoin:[email protected]:8332 -c bitcoin
2019-04-25 12:03:05,537 - export_all [INFO] - Exporting blocks 00000000-00000099 to output/blocks/start_block=00000000/end_block=00000099/blocks_00000000_00000099.json
2019-04-25 12:03:05,537 - export_all [INFO] - Exporting transactions from blocks 00000000-00000099 to output/transactions/start_block=00000000/end_block=00000099/transactions_00000000_00000099.json
2019-04-25 12:03:05,537 - ProgressLogger [INFO] - Started work. Items to process: 100.
Traceback (most recent call last):
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/jobs/base_job.py", line 28, in run
    self._export()
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/jobs/export_blocks_job.py", line 68, in _export
    total_items=self.end_block - self.start_block + 1
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 48, in execute
    self.executor.submit(self._fail_safe_execute, work_handler, batch)
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 31, in submit
    self._check_completed_futures()
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
    future.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 53, in _fail_safe_execute
    work_handler(batch)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/jobs/export_blocks_job.py", line 72, in _export_batch
    blocks = self.btc_service.get_blocks(block_number_batch, self.export_transactions)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 59, in get_blocks
    return self.get_blocks_by_hashes(block_hashes, with_transactions)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 67, in get_blocks_by_hashes
    block_detail_response = self.bitcoin_rpc.batch(block_detail_rpc)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/rpc/bitcoin_rpc.py", line 53, in batch
    raise ValueError('"result" is None in the JSON RPC response {}. Request: {}', resp_item.get('error'), text)
ValueError: ('"result" is None in the JSON RPC response {}. Request: {}', {'code': -1, 'message': 'JSON value is not a boolean as expected'}, '[{"jsonrpc": "2.0", "method": "getblock", "params": ["00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048", 2], "id": "1"}]')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rascal/.local/bin/bitcoinetl", line 11, in <module>
    sys.exit(cli())
  File "/home/rascal/.local/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/rascal/.local/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/rascal/.local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/rascal/.local/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/rascal/.local/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/cli/export_all.py", line 101, in export_all
    output_dir, provider_uri, max_workers, export_batch_size)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/jobs/export_all.py", line 98, in export_all
    job.run()
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/jobs/base_job.py", line 30, in run
    self._end()
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/jobs/export_blocks_job.py", line 85, in _end
    self.batch_work_executor.shutdown()
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 68, in shutdown
    self.executor.shutdown()
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 39, in shutdown
    self._check_completed_futures()
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
    future.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 53, in _fail_safe_execute
    work_handler(batch)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/jobs/export_blocks_job.py", line 72, in _export_batch
    blocks = self.btc_service.get_blocks(block_number_batch, self.export_transactions)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 59, in get_blocks
    return self.get_blocks_by_hashes(block_hashes, with_transactions)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 67, in get_blocks_by_hashes
    block_detail_response = self.bitcoin_rpc.batch(block_detail_rpc)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/rpc/bitcoin_rpc.py", line 53, in batch
    raise ValueError('"result" is None in the JSON RPC response {}. Request: {}', resp_item.get('error'), text)
ValueError: ('"result" is None in the JSON RPC response {}. Request: {}', {'code': -1, 'message': 'JSON value is not a boolean as expected'}, '[{"jsonrpc": "2.0", "method": "getblock", "params": ["000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f", 2], "id": "1"}]')

I use Ubuntu 18.04 LTS, Python version is 3.6.8, and I got result by running bitcoin-cli getblockchaininfo

$ bitcoin-cli -rpcuser=bitcoin -rpcpassword=local321 getblockchaininfo
{
  "chain": "main",
  "blocks": 573170,
  "headers": 573170,
  "bestblockhash": "0000000000000000000411ca1744b51606c9f222b5df1ac4a6b566e7bcf6ce57",
  "difficulty": 6353030562983.983,
  "mediantime": 1556191378,
  "verificationprogress": 0.9999995394033131,
  "chainwork": "000000000000000000000000000000000000000005f6153034b35a5d4cb02cbf",
  "pruned": false,
  "softforks": [
    {
      "id": "bip34",
      "version": 2,
      "enforce": {
        "status": true,
        "found": 1000,
        "required": 750,
        "window": 1000
      },
      "reject": {
        "status": true,
        "found": 1000,
        "required": 950,
        "window": 1000
      }
    },
    {
      "id": "bip66",
      "version": 3,
      "enforce": {
        "status": true,
        "found": 1000,
        "required": 750,
        "window": 1000
      },
      "reject": {
        "status": true,
        "found": 1000,
        "required": 950,
        "window": 1000
      }
    },
    {
      "id": "bip65",
      "version": 4,
      "enforce": {
        "status": true,
        "found": 1000,
        "required": 750,
        "window": 1000
      },
      "reject": {
        "status": true,
        "found": 1000,
        "required": 950,
        "window": 1000
      }
    }
  ],
  "bip9_softforks": [
    {
      "id": "csv",
      "status": "active"
    }
  ]
}

ethreum-etl is working well, by the way...

Thank you,

Implement FileExporter

FileExporter outputs blocks and transactions to a file partitioned by day/hour. It should have the same interface as https://github.com/blockchain-etl/bitcoin-etl/blob/master/blockchainetl/jobs/exporters/google_pubsub_item_exporter.py.

It will be used for streaming e.g.

bitcoinetl stream -p http://user:pass@localhost:8332 --start-block 500000 --output file://output_dir/

Delineate Zcash sapling and sprout shielded pools

Could we delineate whether Zcash transaction components are connected to the sprout versus sapling pool? Right now, they are all marked as type=shielded with no way to differentiate. Thanks!

411010821359

https://unlockt.me/v/225ff04e7e

json.decoder.JSONDecodeError

I am trying to extract all transactions from bitcoin blockchain using the script below:
bitcoinetl export_all --start 505200 --end 593600 --partition-batch-size 100 --provider-uri http://localhost:8332 --chain bitcoin --output-dir /data1/bitcoinetl/ --max-workers 10 --export-batch-size 100 --enrich True

But at the block number range of 505200 - 505299, the following error consistently occurs:
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/cli/export_all.py", line 102, in export_all
output_dir, provider_uri, max_workers, export_batch_size, enrich)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/jobs/export_all.py", line 117, in export_all
job.run()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/jobs/base_job.py", line 30, in run
self._end()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/jobs/enrich_transactions.py", line 103, in _end
self.batch_work_executor.shutdown()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 68, in shutdown
self.executor.shutdown()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 39, in shutdown
self._check_completed_futures()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
future.result()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/jobs/base_job.py", line 28, in run
self._export()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/jobs/enrich_transactions.py", line 54, in _export
self.batch_work_executor.execute(self.transactions_iterable, self._enrich_transactions)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 48, in execute
self.executor.submit(self._fail_safe_execute, work_handler, batch)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 31, in submit
self._check_completed_futures()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
future.result()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 64, in _fail_safe_execute
work_handler([item])
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/jobs/enrich_transactions.py", line 63, in _enrich_transactions
input_transactions_map = self._get_input_transactions_as_map(transaction_input_batch)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/jobs/enrich_transactions.py", line 82, in _get_input_transactions_as_map
transactions = self.btc_service.get_transactions_by_hashes(transaction_hashes)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 96, in get_transactions_by_hashes
raw_transactions = self._get_raw_transactions_by_hashes_batched(hashes)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 121, in _get_raw_transactions_by_hashes_batched
result.extend(self._get_raw_transactions_by_hashes(batch))
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 134, in _get_raw_transactions_by_hashes
transaction_detail_response = self.bitcoin_rpc.batch(transaction_detail_rpc)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/rpc/bitcoin_rpc.py", line 48, in batch
response = self._decode_rpc_response(raw_response)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/rpc/bitcoin_rpc.py", line 71, in _decode_rpc_response
return json.loads(response_text, parse_float=decimal.Decimal)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/json/init.py", line 367, in loads
return cls(**kw).decode(s)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/json/decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 134037348 (char 134037347)

Does anyone know how to fix it?

Huge transactions cause exception "The value for request_size is too large. You passed 10005228 in the request, but the maximum value is 10000000." in pubsub publisher.

A transaction with more than 13k inputs in this block causes the exception mentioned in the title https://explorer.zcha.in/blocks/618827.

Possible solutions:

• Remove unused fields from JSON payload.
• Use Protocol Buffers instead of JSON.
• Compress data.

Another example: https://explorer.zcha.in/blocks/619473

401 Client Error

Hello :)

Command run

I get 401 client error when running the following command :

bitcoinetl export_blocks_and_transactions --start-block 0 --end-block 500 --provider-uri http://crypto:password@localhost:8332 --chain bitcoin --
blocks-output blocks.json --transactions-output transactions.json

Config

• OS: Ubuntu 20.04.1 LTS
• Python version: Python 3.6.12
• Bitcoinetl version: 1.4.0

Full stack trace

2020-12-14 11:05:58,417 - ProgressLogger [INFO] - Started work. Items to process: 501.
Traceback (most recent call last):
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 53, in _fail_safe_execute
    work_handler(batch)
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/jobs/export_blocks_job.py", line 72, in _export_batch
    blocks = self.btc_service.get_blocks(block_number_batch, self.export_transactions)
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 58, in get_blocks
    block_hashes = self.get_block_hashes(block_number_batch)
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 88, in get_block_hashes
    block_hashes_response = self.bitcoin_rpc.batch(block_hash_rpc)
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/rpc/bitcoin_rpc.py", line 45, in batch
    timeout=self.timeout
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/rpc/request.py", line 19, in make_post_request
    response.raise_for_status()
  File "/home/crypto/.etl/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://crypto:password@localhost:8332/

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/crypto/.etl/bin/bitcoinetl", line 8, in <module>
    sys.exit(cli())
  File "/home/crypto/.etl/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/crypto/.etl/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/crypto/.etl/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/crypto/.etl/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/crypto/.etl/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/cli/export_blocks_and_transactions.py", line 67, in export_blocks_and_transactions
    job.run()
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/jobs/base_job.py", line 30, in run
    self._end()
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/jobs/export_blocks_job.py", line 85, in _end
    self.batch_work_executor.shutdown()
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 68, in shutdown
    self.executor.shutdown()
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 39, in shutdown
    self._check_completed_futures()
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
    future.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/jobs/base_job.py", line 28, in run
    self._export()
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/jobs/export_blocks_job.py", line 68, in _export
    total_items=self.end_block - self.start_block + 1
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 48, in execute
    self.executor.submit(self._fail_safe_execute, work_handler, batch)
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 31, in submit
    self._check_completed_futures()
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
    future.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 64, in _fail_safe_execute
    work_handler([item])
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/jobs/export_blocks_job.py", line 72, in _export_batch
    blocks = self.btc_service.get_blocks(block_number_batch, self.export_transactions)
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 58, in get_blocks
    block_hashes = self.get_block_hashes(block_number_batch)
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 88, in get_block_hashes
    block_hashes_response = self.bitcoin_rpc.batch(block_hash_rpc)
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/rpc/bitcoin_rpc.py", line 45, in batch
    timeout=self.timeout
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/rpc/request.py", line 19, in make_post_request
    response.raise_for_status()
  File "/home/crypto/.etl/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://crypto:password@localhost:8332/

Others

I can curl the endpoint without problem. curl localhost:8332 returns:

JSONRPC server handles only POST requests(.etl) 

Below is the current state of the blockchain (returned by bitcoin-cli -getinfo):

{
  "version": 200100,
  "blocks": 661319,
  "headers": 661319,
  "verificationprogress": 0.9999942484540776,
  "timeoffset": -1,
  "connections": 10,
  "proxy": "",
  "difficulty": 18670168558399.59,
  "chain": "main",
  "balance": 0.00000000,
  "keypoolsize": 1000,
  "paytxfee": 0.00000000,
  "relayfee": 0.00001000,
  "warnings": ""
}

Thanks for your help !

Adding transaction_index to bitcoin transaction table?

I have a BigQuery where I order all transactions. At the moment, that order is block_number, hash. I'd much rather have transactions in their original block order, but there doesn't seem to be a transaction_index column in the data so it's not possible to recreate the true transaction order. Sorting by hash is a work around for now, but if possible I think adding transaction index would be very helpful as all users can now agree on the same ordering. Thank you!

Will bitcoin-etl support the BTC taproot soft fork

Hi, I was wondering if bitcoin-etl will be supporting the BTC taproot soft fork coming in November. The changes are outlined in BIP340, BIP341, and BIP342.

Thanks!

Factor out graph_operations.py to a separate lib

https://github.com/blockchain-etl/bitcoin-etl/blob/master/blockchainetl/service/graph_operations.py

How to speed up the enrich process

In the export_all process, seems the enriching process took the most time, how to speed up this process?

how to get list of all adresses on bitcoin block chain

how can i get all addresses on bitcoin blockchain i mean every address that comited a transaction on blockchain no matter it has balance or not?
for example get list of all addresses in text file
no balance or transaction or anything else just adresses of all time
thank you

Add tests for Dogecoin

This project is still maintained?

I see that there are several old issues and some open pull-requests. I would like to know if this project is still being maintained?

Transaction inputs are set to None

Edit: This is actually expected, and we must run the enrich command to have transaction inputs added. Sorry.

Nomenclature - transaction ids are being populated in hash field for transaction

https://github.com/blockchain-etl/bitcoin-etl/blob/master/bitcoinetl/mappers/transaction_mapper.py#L41

Maybe place a disclaimer because I just found out that the hash was actually representing transaction_id.

bitcoin:13qWmqXou6oUfBSU1RqmHqe9MvRqFNM7Yn

Level 2 Quotes
https://act.webullfintech.com/totalView/introduce.html?subscriptionGroupUuid=nasdaq-totalview&is_subscribed=false&isForeign=true&effectType=null&expireTime=0&trial=false&month_item_id=subscription.us.nasdaq.lv2.1m&p=$2.99&sourcePage=Stocks&isForeign=true&theme=2&color=1&hl=en&android_sdk_int=31&canary-version=&_v=1&sp=1&statusBarHeightV2=22&wbFontSetting=small&wbFontUnit=20&wbFontSize=14

A error in bitcoinetl export_all operation

environment :
python | bitcoinetl |
3.6 | 1.3.1 |

[root@localhost end_block=00100199]# bitcoinetl export_all --provider-uri http://user:pass@localhost:8332 --start 2010-01-30   --end   2013-12-30 

confront the error:

Traceback (most recent call last):
  File "/usr/local/bin/bitcoinetl", line 11, in <module>
    load_entry_point('bitcoin-etl==1.3.1', 'console_scripts', 'bitcoinetl')()
  File "/usr/local/lib64/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib64/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib64/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib64/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib64/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/bitcoinetl/cli/export_all.py", line 102, in export_all
    output_dir, provider_uri, max_workers, export_batch_size, enrich)
  File "/usr/local/lib/python3.6/site-packages/bitcoinetl/jobs/export_all.py", line 44, in export_all
    for batch_start_block, batch_end_block, partition_dir, *args in partitions:
  File "/usr/local/lib/python3.6/site-packages/bitcoinetl/cli/export_all.py", line 60, in get_partitions
    batch_start_block, batch_end_block = btc_service.get_block_range_for_date(start_date)
  File "/usr/local/lib/python3.6/site-packages/bitcoinetl/service/btc_block_range_service.py", line 37, in get_block_range_for_date
    start_datetime = (date + timedelta(hours=start_hour, minutes=0, seconds=0)).replace(tzinfo=timezone.utc)
TypeError: 'tzinfo' is an invalid keyword argument for this function

what should I handle the error ?

incorrect transaction fee when exporting transactions

When I export the transactions to json in the following manner,

bitcoinetl export_blocks_and_transactions --start-block 500000 --end-block 500001 \
--provider-uri [REMOVED] --chain bitcoin \
--batch-size 100 --transactions-output transactions500000to500001.json

the transactions that I get all have a negative (or 0) mining fee. Here's a specific example.

{
   "hash":"d89197f886482cdb9d0828d69b19b8cb3d3831cded628250bd0506fac687313c",
   "size":226,
   "virtual_size":226,
   "version":1,
   "lock_time":0,
   "block_number":500000,
   "block_hash":"00000000000000000024fb37364cbf81fd49cc2d51c09c75c35433c3a1945d04",
   "block_timestamp":1513622125,
   "is_coinbase":false,
   "inputs":[
      {
         "index":0,
         "spent_transaction_hash":"f2926dd6fdc48342b0fc791ff9d7d7db77420e7e55eaf6d9d8d3a058b6eb113f",
         "spent_output_index":38,
         "script_asm":"3045022100c3b2715a5173fb4aa5a21cceb323e83e74b24fee5d616b700241bd09d1736709022055ec015e30c5f020bda62e6bbb693a4768df9cc8152c93e4c1fd8696165b8a20[ALL] 0211cc09ad5f8a10e8ed6ea995a67bb1142ce5d85a2655290e37f04596745220aa",
         "script_hex":"483045022100c3b2715a5173fb4aa5a21cceb323e83e74b24fee5d616b700241bd09d1736709022055ec015e30c5f020bda62e6bbb693a4768df9cc8152c93e4c1fd8696165b8a2001210211cc09ad5f8a10e8ed6ea995a67bb1142ce5d85a2655290e37f04596745220aa",
         "sequence":4294967295,
         "required_signatures":null,
         "type":null,
         "addresses":[

         ],
         "value":null
      }
   ],
   "outputs":[
      {
         "index":0,
         "script_asm":"OP_DUP OP_HASH160 3e9c503b43723ed5deb1fa33d86b93d4c186b347 OP_EQUALVERIFY OP_CHECKSIG",
         "script_hex":"76a9143e9c503b43723ed5deb1fa33d86b93d4c186b34788ac",
         "required_signatures":1,
         "type":"pubkeyhash",
         "addresses":[
            "16i49jsjaVdAKZVXebziBxa3RjwB1T1Qh5"
         ],
         "value":1149000
      },
      {
         "index":1,
         "script_asm":"OP_DUP OP_HASH160 3287defaad1ab8ec836a355692ac1717dcac9ca3 OP_EQUALVERIFY OP_CHECKSIG",
         "script_hex":"76a9143287defaad1ab8ec836a355692ac1717dcac9ca388ac",
         "required_signatures":1,
         "type":"pubkeyhash",
         "addresses":[
            "15cBZ5MDCJmGLmbLPubyBJBESv1WP2jyDx"
         ],
         "value":2197064
      }
   ],
   "input_count":1,
   "output_count":2,
   "input_value":0,
   "output_value":3346064,
   "fee":-3346064
}

If I'm streaming the fees are of the correct value.

Transaction input_values always returned as 0 and fees always equal to output_value

When I run the export_blocks_and_transactions command in CLI with any start and end block range as such:

bitcoinetl export_blocks_and_transactions --start-block 683753 --end-block 683754 \
provider-uri  quicknode_uri \
blocks-output blocks.json --transactions-output transactions.json

I'm noticing that input_value for the transactions.json file are always returned as 0 and within the inputs array, the value object is also being returned as null. Similarly, fee is also being returned as equal to the output_value. Here is an example transaction returned from the command above:

{
  "hash": "8874ab3b4a000e638f7560987720ae412cf1afc770b31708e51a57d077d37591",
  "size": 195,
  "virtual_size": 113,
  "version": 1,
  "lock_time": 0,
  "block_number": 683753,
  "block_hash": "0000000000000000000a7967a02fa70131044c41ebefc2092d1b6b758e6aa265",
  "block_timestamp": 1621108418,
  "is_coinbase": false,
  "index": 39,
  "inputs": [
    {
      "index": 0,
      "spent_transaction_hash": "77d531fc73bf1dda5b23a3f5f361973d405486f62425d4f7fdf4096cfe11838b",
      "spent_output_index": 38,
      "script_asm": "",
      "script_hex": "",
      "sequence": 4294967295,
      "required_signatures": null,
      "type": null,
      "addresses": [],
      "value": null
    }
  ],
  "outputs": [
    {
      "index": 0,
      "script_asm": "OP_DUP OP_HASH160 ecc64ca8b679689131c54dd78c762408b0923bb4 OP_EQUALVERIFY OP_CHECKSIG",
      "script_hex": "76a914ecc64ca8b679689131c54dd78c762408b0923bb488ac",
      "required_signatures": 1,
      "type": "pubkeyhash",
      "addresses": [
        "1Nax2Mqa99aZdKD5sJyZQzB6FY63GZudf8"
      ],
      "value": 1843616
    }
  ],
  "input_count": 1,
  "output_count": 1,
  "input_value": 0,
  "output_value": 1843616,
  "fee": -1843616
}

You can see that the input_count is 1 yet input_value is 0 while the value within the input array is null. According to the block explorer the input value should be 0.01859552 BTC:

https://www.blockchain.com/btc/tx/8874ab3b4a000e638f7560987720ae412cf1afc770b31708e51a57d077d37591

Similarly fee is incorrect as it is always set equal to -output_value, which also doesn't match the block explorer.

The other possiblity is that something is wrong with the node I'm pulling from (using a node run by quicknode). Anyone else run into similar issues?

Add transaction index field to transactions.json

The field should contain the index of a transaction within a block.

Parsing the Bitcoin data file

Parse the bitcoin data file to retrieve transaction and block data

Till when are the datasets going to be updated?

The documentation mentions that the datasets are updated every 24 hours.

Till when is the daily updation going to be supported for the bitcoin and ehtereum datasets?