streamingfast / firehose-ethereum Goto Github PK

View Code? Open in Web Editor NEW

76.0 10.0 24.0 24.14 MB

Ethereum on StreamingFast

License: Apache License 2.0

Go 87.18% Shell 12.49% Dockerfile 0.34%

firehose-ethereum's Introduction

Firehose for Ethereum

Quick start with Firehose for Ethereum can be found in the official Firehose docs. Here some quick links to it:

Release

Use the ./bin/release.sh Bash script to perform a new release. It will ask you questions as well as driving all the required commands, performing the necessary operation automatically. The Bash script runs in dry-mode by default, so you can check first that everything is all right.

Releases are performed using goreleaser and specifically goreleaser-cross.

Docker Bundle Image Building

New version of Ethereum clients means releasing a new version of the full bundled image of firehose-ethereum that contains fireeth binary as well as node instrumented binary to sync with the chain. Doing this is really simple as we will simply ask GitHub to launch an action that will build for us the bundled image with the current up to date version of the Ethereum client.

First, install the GitHub CLI and configure it to be connected with your account.

Run the following commands:

Release latest official release with Firehose V2 Instrumentation: gh workflow run docker.yml -f geth_version=fh2 --ref v1.2.2
Release trunk develop with Firehose V2 Instrumentation (development builds): gh workflow run docker.yml -f geth_version=fh2 --ref develop

Contributing

Issues and PR in this repo related strictly to the Ethereum on StreamingFast.

Report any protocol-specific issues in their respective repositories

Please first refer to the general StreamingFast contribution guide, if you wish to contribute to this code base.

This codebase uses unit tests extensively, please write and run tests.

License

Apache 2.0

firehose-ethereum's People

Contributors

Stargazers

Watchers

firehose-ethereum's Issues

merger: publish number of files into prometheus (too_old_files_count)

2022-01-10T22:25:23.030Z (merger) retrieved list of files (merger/merger.go:179){"too_old_files_count": 0, "too_old_files_count": 884}

"not pushing block to a closed subscription" infinite spam in logs

log fragment:
https://pastebin.com/FfiP59EV

After I stopped and started firehose several times I get this(thousands upon thousands of such messages)
fireeth version dev (Commit d755a7e, Built 2023-02-01T19:56:53Z) 1.3.3 built from sources

geth:

INFO [02-07|06:40:33.649] Initializing firehose 
INFO [02-07|06:40:33.654] Firehose initialized                     enabled=false sync_instrumentation_enabled=true mining_enabled=false block_progress_enabled=false compaction_disabled=false archive_blocks_to_keep=0 genesis_provenance="Geth Default" firehose_version=2.1 geth_version=1.1.18-fh2 chain_variant=bsc
Geth
Version: 1.1.18-fh2
Git Commit: 7d87cc8c32c5654fddbd4a20ff4cce6476b41ce2
Architecture: amd64
Go Version: go1.17.13
Operating System: linux
GOPATH=
GOROOT=go

trying to sync bsc mainnet

any advice?

Error while running reader-node-stdin component

2023-03-31T15:57:42.161Z INFO (<n/a>) registering development exporters from environment variables
2023-03-31T15:57:42.161Z INFO (fireeth) starting with config file '/home/ubuntu/firehose_script/firehose/ethereum/jobs/eth/sub-jobs/reader-node-stdin.yaml'
2023-03-31T15:57:42.161Z INFO (fireeth) launching applications: reader-node-stdin
2023-03-31T15:57:42.162Z INFO (reader-node-stdin) launching nodeos mindreader-stdin {"config": {"GRPCAddr":":13010","OneBlocksStoreURL":"file:///home/ubuntu/zeeve/ethereum/sf-data/storage/one-blocks","OneBlockSuffix":"default","MindReadBlocksChanCapacity":100,"StartBlockNum":0,"StopBlockNum":0,"WorkingDir":"/home/ubuntu/zeeve/ethereum/sf-data/reader/work","LogToZap":false,"DebugDeepMind":false}}
2023-03-31T15:57:42.162Z INFO (reader-node-stdin) launching mindreader plugin
2023-03-31T15:57:42.162Z INFO (reader-node-stdin) creating mindreader plugin {"one_blocks_store_url": "file:///home/ubuntu/zeeve/ethereum/sf-data/storage/one-blocks", "one_block_suffix": "default", "working_directory": "/home/ubuntu/zeeve/ethereum/sf-data/reader/work", "start_block_num": 0, "stop_block_num": 0, "channel_capacity": 100, "with_head_block_updater": true, "with_shutdown_func": true}
2023-03-31T15:57:42.162Z INFO (dstore) sanitized base path {"original_base_path": "/home/ubuntu/zeeve/ethereum/sf-data/reader/work/uploadable-oneblock", "sanitized_base_path": "/home/ubuntu/zeeve/ethereum/sf-data/reader/work/uploadable-oneblock"}
2023-03-31T15:57:42.162Z INFO (dstore) sanitized base path {"original_base_path": "/home/ubuntu/zeeve/ethereum/sf-data/storage/one-blocks", "sanitized_base_path": "/home/ubuntu/zeeve/ethereum/sf-data/storage/one-blocks"}
2023-03-31T15:57:42.162Z INFO (reader-node-stdin) creating new mindreader plugin
2023-03-31T15:57:42.162Z INFO (reader-node-stdin) starting grpc listener {"listen_addr": ":13010"}
2023-03-31T15:57:43.163Z INFO (reader-node-stdin) grpc server listener ready
2023-03-31T15:57:43.163Z INFO (reader-node-stdin) starting mindreader
2023-03-31T15:57:43.163Z INFO (reader-node-stdin) starting consume flow
2023-03-31T15:57:43.164Z INFO (reader-node-stdin) starting stdin reader
2023-03-31T15:57:48.163Z INFO (reader-node-stdin) reader read statistics {"block_rate": "0 blocks/min (0 total)", "trx_rate": "0 trxs/min (0 total)", "last_block": "Block ", "block_average_parse_time": "0ms/block (over 1min)"}
2023-03-31T15:57:53.164Z INFO (reader-node-stdin) reader read statistics {"block_rate": "0 blocks/min (0 total)", "trx_rate": "0 trxs/min (0 total)", "last_block": "Block ", "block_average_parse_time": "0ms/block (over 1min)"}
2023-03-31T15:57:58.163Z INFO (reader-node-stdin) reader read statistics {"block_rate": "0 blocks/min (0 total)", "trx_rate": "0 trxs/min (0 total)", "last_block": "Block ", "block_average_parse_time": "0ms/block (over 1min)"}
2023-03-31T15:58:03.163Z INFO (reader-node-stdin) reader read statistics {"block_rate": "0 blocks/min (0 total)", "trx_rate": "0 trxs/min (0 total)", "last_block": "Block ", "block_average_parse_time": "0ms/block (over 1min)"}
2023-03-31T15:58:08.163Z INFO (reader-node-stdin) reader read statistics {"block_rate": "0 blocks/min (0 total)", "trx_rate": "0 trxs/min (0 total)", "last_block": "Block ", "block_average_parse_time": "0ms/block (over 1min)"}
2023-03-31T15:58:13.163Z INFO (reader-node-stdin) reader read statistics {"block_rate": "0 blocks/min (0 total)", "trx_rate": "0 trxs/min (0 total)", "last_block": "Block ", "block_average_parse_time": "0ms/block (over 1min)"}
2023-03-31T15:58:18.164Z INFO (reader-node-stdin) reader read statistics {"block_rate": "0 blocks/min (0 total)", "trx_rate": "0 trxs/min (0 total)", "last_block": "Block ", "block_average_parse_time": "0ms/block (over 1min)"}
2023-03-31T15:58:23.163Z INFO (reader-node-stdin) reader read statistics {"block_rate": "0 blocks/min (0 total)", "trx_rate": "0 trxs/min (0 total)", "last_block": "Block ", "block_average_parse_time": "0ms/block (over 1min)"}
2023-03-31T15:58:28.163Z INFO (reader-node-stdin) reader read statistics {"block_rate": "0 blocks/min (0 total)", "trx_rate": "0 trxs/min (0 total)", "last_block": "Block ", "block_average_parse_time": "0ms/block (over 1min)"}
2023-03-31T15:58:33.164Z INFO (reader-node-stdin) reader read statistics {"block_rate": "0 blocks/min (0 total)", "trx_rate": "0 trxs/min (0 total)", "last_block": "Block ", "block_average_parse_time": "0ms/block (over 1min)"}
2023-03-31T15:58:38.164Z INFO (reader-node-stdin) reader read statistics {"block_rate": "0 blocks/min (0 total)", "trx_rate": "0 trxs/min (0 total)", "last_block": "Block ", "block_average_parse_time": "0ms/block (over 1min)"}
2023-03-31T15:58:43.163Z INFO (reader-node-stdin) reader read statistics {"block_rate": "0 blocks/min (0 total)", "trx_rate": "0 trxs/min (0 total)", "last_block": "Block ", "block_average_parse_time": "0ms/block (over 1min)"}
2023-03-31T15:58:48.164Z INFO (reader-node-stdin) reader read statistics {"block_rate": "0 blocks/min (0 total)", "trx_rate": "0 trxs/min (0 total)", "last_block": "Block ", "block_average_parse_time": "0ms/block (over 1min)"}
2023-03-31T15:58:53.163Z INFO (reader-node-stdin) reader read statistics {"block_rate": "0 blocks/min (0 total)", "trx_rate": "0 trxs/min (0 total)", "last_block": "Block ", "block_average_parse_time": "0ms/block (over 1min)"}
2023-03-31T15:58:58.164Z INFO (reader-node-stdin) reader read statistics {"block_rate": "0 blocks/min (0 total)", "trx_rate": "0 trxs/min (0 total)", "last_block": "Block ", "block_average_parse_time": "0ms/block (over 1min)"}

=======
How can i pass stdout of geth to stdin of fireeth start reader-node-stdin
i tried pipe to pass the output bu the result is like in the above logs not getting any data .
please help me to solve the issue.

allow passing a string to the ready configuration block in prometheus exporter

We have different "firehose block versions" of an app running at the same time. I would be great if the ready status in the prometheus exporter could say which version was running.

# HELP ready readiness of an app. 1 if ready, 0 otherwise
# TYPE ready gauge
ready{app="firehose", block-version="v3"} 0

Where this might be configured as common-reporting-block-version: v3.

This is just for reporting, firehose doesn't do anything with it otherwise.

reader libNumber never moves forward when replaying old chain data

libNumber never moves forward

$ grpcurl -plaintext -d '{}' 'localhost:9000' 'sf.headinfo.v1.HeadInfo.GetHeadInfo'
{
  "libNum": "12660716",
  "headNum": "12715300",
  "headID": "1d39f0036deb2770748da57717bd7dc52d2244b524bc0881fb57cb5f0dcea37c",
  "headTime": "2022-08-03T14:50:00Z"
}

check later:

$ grpcurl -plaintext -d '{}' 'localhost:9000' 'sf.headinfo.v1.HeadInfo.GetHeadInfo'
{
  "libNum": "12660716",
  "headNum": "12747578",
  "headID": "b869c6f610588ed5618f24cb1e9d8beff6451d85bfec48950700f509092046e5",
  "headTime": "2022-08-08T21:45:12Z"
}

How to consume the firehose block stream?

I've read through the firehose docs and it's not clear how the block stream produced by firehouse should be consumed. Does the gRPC server provide an interface that is compatible with the eth JSON-RPC interface? Does Firehose integrate tightly with graph node (considering it's mentioned in the graph node docs)? How should the grpc server be started, in the startup YAML or as a subcommand?

test

fireeth tools check merged-blocks doesn't print holes

From indexer-firehose discussion:

✅ Range #0 - #999 999
✅ Range #1 000 000 - #1 999 999
✅ Range #2 000 000 - #2 999 999
✅ Range #3 000 000 - #3 999 999
✅ Range #4 000 000 - #4 999 999
✅ Range #5 000 000 - #5 999 999
✅ Range #6 000 000 - #6 999 999
✅ Range #7 500 100 - #8 000 099
✅ Range #8 000 100 - #9 000 099
✅ Range #9 000 100 - #10 000 099
✅ Range #10 000 100 - #11 000 099
✅ Range #11 500 100 - #12 000 199
✅ Range #13 000 100 - #13 000 399
✅ Range #13 000 400 - #14 000 399
✅ Range #15 000 100 - #15 071 299
✅ Range #15 750 100 - #16 071 599
✅ Range #17 000 100 - #17 071 699

Summary:
> 🆘 Holes found!

Indexing: Read-in a LogAddressIndex from dstore

We want to handle indexes for Log Addresses

Acceptance Criteria:

The LogAddressIndexer should:

accept an logaddr index file from dstore
unmarshal the index to a LogAddressIndex

Initialize LogAddressIndexProvider from the Transform

BasicLogFilter (implementing interface Transform) should implement a new method, ~GetIndexProvider() that will utlimately be called from the firehose (returns the provider, or nil if anything goes wrong)
in cmd/sfeth/cli/firehose.go, when registering the BasicLogFilterFactory, it should be pre-loaded (using a FactoryFactory ™️ ) with the index store and the indexSizes array.
in bstream and maybe the firehose package, you need to check if GetIndexProvider is implemented via a new interface, and call it on your transforms (see the transformReg.BuildFromTransforms method, that generates a preprocessor, now you need the same mechanism to generate an index)
What to do if more than one tranform generates an index ? panic for now..... or maybe just use one of them.

Add support for "withdrawal" balance change reason

On sepolia

panic: receive unknown balance change reason, received reason string is "withdrawal"
goroutine 102 [running]:
github.com/streamingfast/firehose-ethereum/types/pb/sf/ethereum/type/v2.MustBalanceChangeReasonFromString({0xc007fee0e9, 0xa})

Panic in firehose while processing block

Network: BSC Test net
Firehose version: v1.3.1
Geth: bsc-v1.1.18-fh2.1-2

Steps to reproduce - start firehose sync, try doing an RPC request from substream code which results in an error like this
WARN (rpc-cache) retrying RPCCall on non-deterministic RPC call error {"error": "rpc error (code -32000): missing trie node cba260375b8a093ff89fa569a257fddee8ecec5178b4590a9238b64d71950030 (path )", "at_block": "2720c990fc3b507cd22f08ed2804f1e6c8c324e04b32e2ce70819ca40795dfe6", "endpoint": "http://localhost:8547"} and then stop substreams process(either ctrl-c or sigkill)

Firehose log during the panic: https://pastebin.com/HLzRjZ49
Probably this error should be handled more gracefully

Default command line uses obsolete arguments to geth

(mindreader-node) WARN [11-26|15:17:46.583] The flag --rpc is deprecated and will be removed June 2021, please use --http  (log_plugin/to_zap_log_plugin.go:131)
(mindreader-node) WARN [11-26|15:17:46.583] The flag --rpcaddr is deprecated and will be removed June 2021, please use --http.addr  (log_plugin/to_zap_log_plugin.go:131)
(mindreader-node) WARN [11-26|15:17:46.583] The flag --rpcport is deprecated and will be removed June 2021, please use --http.port  (log_plugin/to_zap_log_plugin.go:131)
(mindreader-node) WARN [11-26|15:17:46.583] The flag --rpcapi is deprecated and will be removed June 2021, please use --http.api  (log_plugin/to_zap_log_plugin.go:131)
(mindreader-node) WARN [11-26|15:17:46.583] The flag --rpcvhosts is deprecated and will be removed June 2021, please use --http.vhosts  (log_plugin/to_zap_log_plugin.go:131)
(mindreader-node) WARN [11-26|15:17:46.583] Option nousb is deprecated and USB is deactivated by default. Use --usb to enable  (log_plugin/to_zap_log_plugin.go:131)

mindreader provides --port 30305 is provided even if overridden

I have mindreader-node-arguments: "+--port=30301 --cache=2048"

ps shows me

/usr/bin/bsc-geth --networkid=56 --datadir=/var/lib/geth --ipcpath=/var/lib/dfuse/sf/ipc --port=30305 --rpc --rpcapi=admin,debug,eth,net,web3 --rpcport=8547 --rpcaddr=0.0.0.0 --rpcvhosts=* --nousb --firehose-deep-mind --port=30301 --cache=2048

Notice there are two --port arguments. It seems to work by taking the last one, but would like to avoid passing the first one anyway.

Indexing: Write-out logAddressIndex to dstore

We want to create indexes for Log Addresses

Acceptance Criteria:

When processing an Ethereum block, the LogAddressIndexer should:

initialize an index
check if the index is on a boundary, defined by indexSize
handle a block's offset, if it's in between bounds of an indexSize
wrire-out an index file using dstore when the upper bound of an indexSize is reached
spawn a new index to handle the next bounds

Chache glitch and log spam

After upgrade to v1.4.1 storm of calling updateCache can be observed from unknown source.

There is additional repetative messages:

INFO (rpc-cache) rpc cache not found {"filename": "cache-0013142000-0013143000.cache", "read_store_url": "file:///firehose/sf-data/rpc-cache", "error": "not found"}
INFO (firehose.tier2) not saving cache, because empty {"parent_trace_id": "ce1298b5ee1e9df69170ea2201d7fc79", "trace_id": "f7a89284e5c91d73982d4caa0dc4d4b2", "module": "store_supply", "block_range": "[13141000, 13142000)"}

Reprocess Polygon Mainnet

OKR: Build a raving community of developers around Substreams / Publish 40 pieces of contents on Medium or YouTube

Goal: Enable rewards for polygon for SbS on the network
Airstack needs it?

Need to reprocess due to fixes here: streamingfast/firehose#25

rpcapi is bound to 0.0.0.0:8547 by default. Why is this needed?

When starting mindreader, it forks off geth like this:

Why is it listening on 0.0.0.0:8547 for admin requests by default? Seems insecure?

substreams tier1 http://localhost:9000/healthz returns text/plain instead of application/json

tier1 http://localhost:9000/healthz returns text/plain instead of application/json.

tier2 properly returns application/json.

(port 9000 being wherever substreams-tier1-grpc-listen-addr is listening.)

cannot get lib num from blockstream

When I'm trying to connect to GRPC server using method Blocks(ctx, req, opts...) there is an error:

(firehose) cannot get lib num from blockstream, retrying {"retries": 4, "error": "tracking fetch errors in order: tracker block not found"}

sf.yaml config:

  args:
    #    - merger
    #    - mindreader-node
    #    - relayer
  - firehose
  flags:
    common-blockstream-addr: :9002
    firehose-grpc-listen-addr: :9000
    common-chain-id: "56"
    common-network-id: "56"
    mindreader-node-bootstrap-data-url: ./mindreader/genesis.json
    mindreader-node-log-to-zap: false
    mindreader-node-arguments: *skipped*

(mindreader-node) WARN [11-18|20:48:51.723] Failed to load old bad blocks error="leveldb: not found" (log_plugin/to_zap_log_plugin.go:131)

geth with mindreader crashed

$ sfeth --version
sfeth version -dev

Firehose dies with cannot link new blocks to chain after a reconnection

Network: BSC Test net
Firehose version: v1.3.1
Geth: bsc-v1.1.18-fh2.1-2

Steps to repeat: start syncing, sync some blocks and press ctrl-c or terminate process abruptly(I'm running merger, firehose, reader-node, relayer, combined-index-builder) all in one process.
Restart ./fireeth -c config.yaml start(maybe several times) and watch it die with cannot link new blocks to chain after a reconnection
sometimes it also enters infinite loop spamming console with millions of 2023-01-18T09:41:07.186+0400 INFO (reader.sub.relayer) not pushing block to a closed subscription

I think firehose should be able to restart gracefully even after abrupt termination(and it fails to do so even when doing graceful shutdown via ctrl+c)

edit: on yet another restart with the same state it started to complain on this 2023-01-18T09:43:46.926+0400 WARN (bstream) too many consecutive unlinkable blocks {"block": "#136834 (06ce4bb2b6c54d9c)", "consecutive_unlinkable_blocks": 24300, "last_block_sent": "#112408 (444c547f0115596d)"}

in the result it seems that local state is corrupted and I cannot continue syncing data... is there any way to repair it?

config file

start:
  args:
  # Define Firehose components that will be used for this setup
  - merger
  - firehose
  - reader-node
  - relayer
  - combined-index-builder
  flags:
    # Prevent creation of file for logging (logs from previous sessions are not visible with this setting)
    log-to-file: false

    # Ethereum is the default chain, its id is 1. Update these flags for different chains such as Binance
    common-chain-id: "97"
    common-network-id: "97"
    # Update the reader-node-path to reference the geth binary for the chain and OS being targeted
    reader-node-path: ../geth/geth_linux
    # Update the reader-code-arguments for the chain being targeted
    # Find specific values in the Firehose Ethereum Setup Documentation
    reader-node-arguments: "+--maxpeers 500 --cache 4096 --snapshot=false  --bootnodes=enode://69a90b35164ef862185d9f4d2c5eff79b92acd1360574c0edf36044055dc766d87285a820233ae5700e11c9ba06ce1cf23c1c68a4556121109776ce2a3990bba@52.199.214.252:30311,enode://330d768f6de90e7825f0ea6fe59611ce9d50712e73547306846a9304663f9912bf1611037f7f90f21606242ded7fb476c7285cb7cd792836b8c0c5ef0365855c@18.181.52.189:30311,enode://df1e8eb59e42cad3c4551b2a53e31a7e55a2fdde1287babd1e94b0836550b489ba16c40932e4dacb16cba346bd442c432265a299c4aca63ee7bb0f832b9f45eb@52.51.80.128:30311,enode://0bd566a7fd136ecd19414a601bfdc530d5de161e3014033951dd603e72b1a8959eb5b70b06c87a5a75cbf45e4055c387d2a842bd6b1bd8b5041b3a61bab615cf@34.242.33.165:30311,enode://604ed87d813c2b884ff1dc3095afeab18331f3cc361e8fb604159e844905dfa6e4c627306231d48f46a2edceffcd069264a89d99cdbf861a04e8d3d8d7282e8a@3.209.122.123:30311,enode://4d358eca87c44230a49ceaca123c89c7e77232aeae745c3a4917e607b909c4a14034b3a742960a378c3f250e0e67391276e80c7beee7770071e13f33a5b0606a@52.72.123.113:30311"
    reader-node-log-to-zap: false
    reader-node-bootstrap-data-url: ../geth/testnet/genesis.json

    firehose-grpc-listen-addr: ":9000"
    substreams-enabled: true
    substreams-sub-request-parallel-jobs: 10
    substreams-partial-mode-enabled: true
    substreams-rpc-endpoints: "$BSC_SUBSTREAMS_RPC_ENDPOINT"
    substreams-sub-request-block-range-size: 100
    substreams-stores-save-interval: 100

example log:https://pastebin.com/w3Rn2k8r

Reprocess Polygon Mumbai

Need to reprocess due to fixes here: streamingfast/firehose#25

Unable to sync ethereum mainnet

Following docs at
My config is as recommended is: https://firehose.streamingfast.io/firehose-setup/ethereum/installation-1

start:
  args:
  # Define Firehose components that will be used for this setup
  - merger
  - firehose
  - reader-node
  - relayer
  - combined-index-builder
  flags:
    # Prevent creation of file for logging (logs from previous sessions are not visible with this setting)
    log-to-file: true

    # Ethereum is the default chain, its id is 1. Update these flags for different chains such as Binance
    common-chain-id: "1"
    common-network-id: "1"
    # Update the reader-node-path to reference the geth binary for the chain and OS being targeted
    reader-node-path: ../geth/eth_geth
    # Update the reader-code-arguments for the chain being targeted
    # Find specific values in the Firehose Ethereum Setup Documentation
    reader-node-arguments: "+--cache 8192 --maxpeers 100 --metrics --metrics.port 6061 --port=30303 --http.port=8545 --snapshot=true --txlookuplimit=1000"
    reader-node-log-to-zap: false

    firehose-grpc-listen-addr: ":9000"
    substreams-enabled: true
    #substreams-sub-request-parallel-jobs: 10
    #substreams-partial-mode-enabled: true
    substreams-rpc-endpoints: "$BSC_SUBSTREAMS_RPC_ENDPOINT"
    #substreams-sub-request-block-range-size: 100
    #substreams-stores-save-interval: 100

geth:

./eth_geth version
INFO [01-24|09:51:57.922] Initializing firehose 
INFO [01-24|09:51:57.926] Firehose initialized                     enabled=false sync_instrumentation_enabled=true mining_enabled=false block_progress_enabled=false compaction_disabled=false archive_blocks_to_keep=0 genesis_provenance="Geth Default" firehose_version=2.1 geth_version=1.10.26-fh2 chain_variant=geth
Geth
Version: 1.10.26-fh2
Git Commit: b47e39e956b6646f215f11b9a8a70ad03f733ab6
Architecture: amd64
Go Version: go1.17.13
Operating System: linux
GOPATH=
GOROOT=go

lots of such messages in logs:

WARN [01-24|09:47:44.328] Snapshot extension registration failed   peer=aab3bbde err="peer connected on snap without compatible eth support"
INFO [01-24|09:48:04.915] Looking for peers                        peercount=2 tried=37 static=0
WARN [01-24|09:50:01.325] Snapshot extension registration failed   peer=c5fb898d err="peer connected on snap without compatible eth support"

There's definitely something fishy is going on with node startup, I had the same intermittent issue with BSC mainnet
Any help welcomed

Firehose-Ethereum migrate to Firehose Core

Devel sync-mainnet stuck on startup (can't find peers)

Hi,

i'm having an issue with running the example from the devel/sync-mainnet/ folder and all links to your discord seem to be expired in docs so I can't find/join it.

Lighthouse release used: https://github.com/sigp/lighthouse/releases/tag/v3.3.0
Config used(no changes, just for testing): https://github.com/streamingfast/firehose-ethereum/tree/develop/devel/sync-mainnet

streamingfast/go-ethereum release used: release/geth-v1.10.26-fh2.1

fireeth --version
fireeth version dev

Issue: Basically the geth node can't find peers. It gets stuck in this state for >30minutes and then I terminate it. I checked the security groups in AWS and they all seem reasonably open for the ports it needs. I tried opening the ports completely too to 0.0.0.0/0. Other aws compute with rocketpool and erigon nodes running in the same security group work fine so I don't think it's the network.

For testing:

export CHECKPOINT_SYNC_URL=https://sync.invis.tools

Consensys output: https://pastecode.io/s/djzjdw8i

Firehose output: https://pastecode.io/s/ym9pfdkk

file does not exist error while running start.sh

source shutting down {"endpoint_url": ":13010", "error": "rpc error: code = Unimplemented desc = unknown service sf.bstream.v1.BlockStream"}
2023-03-31T05:15:15.974Z INFO (index-builder) reading from blocks store: file does not (yet?) exist, retrying in {"filename": "/home/ubuntu/ethereum/sf-data/storage/merged-blocks/0000000000.dbin.zst", "base_filename": "0000000000", "retry_delay": "4s"}

Merger stuck at block 31000

It seems that my merger is now stuck at block 31000. I see the following log line emitted over and over

firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"WARNING","timestamp":"2022-10-21T15:44:42.420585258-04:00","logger":"bstream","message":"too many consecutive unlinkable blocks","block":"#109903 (b4a118481e568b1c)","consecutive_unlinkable_blocks":77100,"last_block_sent":"#31272 (00689cb504900f40)","logging.googleapis.com/labels":{}}

and here's a more complete sample of log lines:

firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"INFO","timestamp":"2022-10-21T15:44:36.322716469-04:00","logger":"index-builder","message":"reading from blocks store: file does not (yet?) exist, retrying in","filename":"/data/firehose/storage/merged-blocks/0000031000.dbin.zst","base_filename":"0000031000","retry_delay":4,"logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"INFO","timestamp":"2022-10-21T15:44:37.068411556-04:00","logger":"reader.geth","message":"forkchoice requested sync to new head    number=15,798,650 hash=d179b5..c50f7d","logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"INFO","timestamp":"2022-10-21T15:44:40.322901399-04:00","logger":"index-builder","message":"reading from blocks store: file does not (yet?) exist, retrying in","filename":"/data/firehose/storage/merged-blocks/0000031000.dbin.zst","base_filename":"0000031000","retry_delay":4,"logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"INFO","timestamp":"2022-10-21T15:44:40.834577546-04:00","logger":"reader.geth","message":"imported new chain segment               blocks=71   txs=389  mgas=8.229  elapsed=8.235s    mgasps=0.999  number=369,258    hash=f8be77..e6c618 age=7y1mo2w  dirty=5.43MiB","logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"WARNING","timestamp":"2022-10-21T15:44:42.420585258-04:00","logger":"bstream","message":"too many consecutive unlinkable blocks","block":"#109903 (b4a118481e568b1c)","consecutive_unlinkable_blocks":77100,"last_block_sent":"#31272 (00689cb504900f40)","logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"INFO","timestamp":"2022-10-21T15:44:44.323103322-04:00","logger":"index-builder","message":"reading from blocks store: file does not (yet?) exist, retrying in","filename":"/data/firehose/storage/merged-blocks/0000031000.dbin.zst","base_filename":"0000031000","retry_delay":4,"logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"INFO","timestamp":"2022-10-21T15:44:48.323232598-04:00","logger":"index-builder","message":"reading from blocks store: file does not (yet?) exist, retrying in","filename":"/data/firehose/storage/merged-blocks/0000031000.dbin.zst","base_filename":"0000031000","retry_delay":4,"logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"INFO","timestamp":"2022-10-21T15:44:48.996961958-04:00","logger":"reader.geth","message":"imported new chain segment               blocks=804  txs=708  mgas=15.726 elapsed=8.115s    mgasps=1.938  number=370,062    hash=266bdd..3b7689 age=7y1mo2w  dirty=4.67MiB","logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"INFO","timestamp":"2022-10-21T15:44:49.077330751-04:00","logger":"reader.geth","message":"forkchoice requested sync to new head    number=15,798,651 hash=31b649..2c1d7c","logging.googleapis.com/labels":{}}
firehose-mainnet-6f8bbfbd75-gcwpc firehose {"severity":"WARNING","timestamp":"2022-10-21T15:44:50.162217455-04:00","logger":"bstream","message":"too many consecutive unlinkable blocks","block":"#110003 (7b770ad57aa913a8)","consecutive_unlinkable_blocks":77200,"last_block_sent":"#31272 (00689cb504900f40)","logging.googleapis.com/labels":{}}

Are the unlinkable blocks an issue? If so, how can I resolve them?

example config needs update to v2

https://github.com/streamingfast/firehose-ethereum/blob/develop/devel/sync-mainnet/sync-mainnet.yaml needs update to v2.

Ethereum Block Model Known Issues

This issue lists the currently know data issue with our Ethereum Block model.

Genesis block storage changes determinism issue

In the previous Firehose patch, we were not sorting genesis storage keys for each account leading to determinism problem on chain for which the genesis contains such definitions.

This is because the storage key/value is stored in a map[common.Hash]common.Hash map container which is unordered an may yield different results on each invocation.

We did not review each genesis config yet to see if there was such problem active today.

It's important to note that there is no problem with the data in itself, everything is there. It's just that two syncs of the genesis block might leads to different order of the StorageChanges kept in the root call.

Transaction root call `BeginOrdinal` is always 0

The root call BeginOrdinal value of all transactions in a block is always 0 today. This is a problem if you sort by call.BeginOrdinal, in that case relies on the fact that Calls are already sorted by execution order within a TransactionTrace.

You can "fix" the issue by doing trx.Calls[0].BeginOrdinal = trx.BeginOrdinal.

Genesis block root call missing `BeginOrdinal/EndOrdinal`

The root call of the genesis block didn't properly record an ordinal increment for BeginOrdinal and for EndOrdinal they will always be both 0.

There is a single call in a genesis block so this shouldn't pose any problem, the trx.Calls[0].BeginOrdinal = trx.BeginOrdinal and trx.Calls[0].EndOrdinal = max.Uint64 change can be applied to fix the issue.

`Transaction#return_data` is not populated

The field TransactionTrace#return_data is not populated by our patch currently.

Some `Call#GasChanges` are missing

While doing the Geth native tracer patch, I noticed we missed at least one place where gas change would have been meaningful. The gas refunded because of data returned to the chain is not tracked properly, but it's an important gas change, those happen at the very end of the transaction, just before the buy back.

Also, the new Geth tracer is also tracing the change from 0 -> <GasLimit> (initial balance) and the change from <LeftOver> -> 0 (buy back) which happens at the very beginning and very end of each call (buy back possibly not present if there is no gas left).

`Call#input` not populate on contract creation

On CREATE/CREATE2 call, we do not populate the call.input field. The contract's code is available in the repeated CodeChange code_changes field so we though we could reduce the block's size by omitting the call.input for those.

But this was a big oversight for contracts with a constructor. Indeed, in those case the constructor's parameter input is found in the call.input at the end of the EVM code, so we are not having it.

Note that for the root call CREATE operation, the input can be taken from the TransactionTrace object containing the call, this one is always properly populated. So it's a problem for smart contracts that deploys other smart contract(s).

`Call#executed_code` not `true` on some situation where it should

The Call#executed_code is instrumented https://github.com/streamingfast/go-ethereum/blob/51624bc5371590de0414116735634705fcbe2a45/core/vm/interpreter.go#L159-L165 which emits if no code was executed at all by the interpreter.

In the call start with do:

call.executed_code = call.type != CREATE && len(call.input) > 0

As the default value and then set call.executed_code = false when the account without code event is emitted by the tracer.

This is problematic for account that have code to "deal" with native transfer of value which is possible with Solidity. In those situations, we are not properly setting executed_code correctly.

The same would be true for a pre-compile address that accepts no input, but I'm unaware of such pre-compiles so I doubt this happen in practices.

`TransactionTrace` enforced state changes

This is a more quirk of the model than a bug in itself, but I'll list here.

The problem here is the fact that even for failed transactions there is some state changes that are still record to the chain namely: the payment of the gas and the nonce change are recorded on the chain's state even though the transaction failed.

Now the modeling problem is that we use call.state_reverted to determine if a call should be inspected but the logic does not apply to every use cases, especially not those dealing with balance changes or nonce changes.

We should maybe have moved those "fixed" changes in the TransactionTrace model.

Bad block breaks geth

I've started running firehose and I just ran into the following log lines which seems to have broken geth - it's now reporting that a Disk Quota has been been exceeded:

firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:24.478946191-04:00","logger":"reader.geth","message":"beacon chain gapped                      head=15,869,849 newHead=15,869,855","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:24.478990291-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:24.477] Failed to clean stale beacon headers     err=\"filled header below beacon header tail: 6158489 < 15869849\"","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:24.693057822-04:00","logger":"reader.geth","message":"unindexed transactions                   blocks=439 txs=55767 tail=6,158,362 elapsed=227.838ms","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:24.880649279-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:25.332971157-04:00","logger":"reader.geth","message":"syncing beacon headers                   downloaded=1540 left=6,158,488 eta=550h18m24.979s","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:25.382861547-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:25.518788928-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:25.518] Expired request does not exist           peer=e053914017a506772cce70913eac44cc9b0e4378adc138f6f11b7c943fe5acbf","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:25.884820224-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:25.955989251-04:00","logger":"reader.geth","message":"downloader queue stats                   receiptTasks=0 blockTasks=34168 itemSize=27.24KiB throttle=8192","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.036932293-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.036] Unhandled trie error: missing trie node 4cdf3958a41d42a048cb37085d0e2450f3a92074a99d446ab4dd7b48cfd190ab (path 03070b030e0c04) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.212740567-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.212] Unhandled trie error: missing trie node 164b4f0699da01839785857cce9e614559afc9a569c10b3e82254db1abb7d16b (path 09090b0f0d04) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.259208341-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.259] Unhandled trie error: missing trie node 164b4f0699da01839785857cce9e614559afc9a569c10b3e82254db1abb7d16b (path 09090b0f0d04) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.268408351-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.268] Unhandled trie error: missing trie node 7217e48235c2c5bed47a4bc25418e952db40c64433424ea1f99b44669000b375 (path 09090e00010900) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.28229091-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.282] Unhandled trie error: missing trie node 7217e48235c2c5bed47a4bc25418e952db40c64433424ea1f99b44669000b375 (path 09090e00010900) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.287387822-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.283] Unhandled trie error: missing trie node ca4cbcd5f6e09f53c448cc68a9b452ce669a25a9eed0f22a79a4e07d4a07bae3 (path 07010e090b0f08) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.287431338-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.284] Unhandled trie error: missing trie node ca4cbcd5f6e09f53c448cc68a9b452ce669a25a9eed0f22a79a4e07d4a07bae3 (path 07010e090b0f08) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.30427221-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.304] Unhandled trie error: missing trie node b60717f155b92939dc3387492394afd09912678a26923af5d2c4e5e315c913f2 (path 04070100050e03) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.30760353-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.306] Unhandled trie error: missing trie node b60717f155b92939dc3387492394afd09912678a26923af5d2c4e5e315c913f2 (path 04070100050e03) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.30780068-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.307] Unhandled trie error: missing trie node 0936e560feba21cfd5ca6686193ede797b4c4a1818436703888696343c38b086 (path 0c09040909020b) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.30991422-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.309] Unhandled trie error: missing trie node 0936e560feba21cfd5ca6686193ede797b4c4a1818436703888696343c38b086 (path 0c09040909020b) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.311182544-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.311] Unhandled trie error: missing trie node da95e202db5dadb272aba88d218c26038af5e437514fa80e3f116f60d50cc4f5 (path 0c0a00030108) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.312121839-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.312] Unhandled trie error: missing trie node da95e202db5dadb272aba88d218c26038af5e437514fa80e3f116f60d50cc4f5 (path 0c0a00030108) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:26.393611317-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.40731312-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.407] Unhandled trie error: missing trie node d460e787b2bfc6b3683c49698b51115ee39bc01ed3211d86d2e2972c0237d6ad (path 06040e090b050b00) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.408046113-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.407] Unhandled trie error: missing trie node d460e787b2bfc6b3683c49698b51115ee39bc01ed3211d86d2e2972c0237d6ad (path 06040e090b050b00) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.408799923-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.408] Unhandled trie error: missing trie node d460e787b2bfc6b3683c49698b51115ee39bc01ed3211d86d2e2972c0237d6ad (path 06040e090b050b00) <nil> ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose FAILED trx "a9cb5b1c828218ee6a334dc9d48ed2c9367f6f35fbb644a1d66c4178056b1927" at block 6158490 (hash unavailable, probably forked): insufficient funds for gas * price + value: address 0x80f2b4ff20c8Ff8835cb815094E91E5b0120CD0B have 0 want 14382095304347515 2472
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410209421-04:00","logger":"reader.geth","message":"skip duplicated bad block                number=6,158,490  hash=b6e729..3720d4","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.410320327-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.410] ","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410359297-04:00","logger":"reader.geth","message":"########## BAD BLOCK #########","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410375414-04:00","logger":"reader.geth","message":"chain config: Chain ID:  1 (mainnet)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410387081-04:00","logger":"reader.geth","message":"consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410399144-04:00","logger":"reader.geth","message":"pre-Merge hard forks:","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41041168-04:00","logger":"reader.geth","message":" - Homestead:                   1150000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410424045-04:00","logger":"reader.geth","message":" - DAO Fork:                    1920000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410451966-04:00","logger":"reader.geth","message":" - Tangerine Whistle (EIP 150): 2463000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410467232-04:00","logger":"reader.geth","message":" - Spurious Dragon/1 (EIP 155): 2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41047902-04:00","logger":"reader.geth","message":" - Spurious Dragon/2 (EIP 158): 2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410490538-04:00","logger":"reader.geth","message":" - Byzantium:                   4370000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410502395-04:00","logger":"reader.geth","message":" - Constantinople:              7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410515128-04:00","logger":"reader.geth","message":" - Petersburg:                  7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410528678-04:00","logger":"reader.geth","message":" - Istanbul:                    9069000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410545419-04:00","logger":"reader.geth","message":" - Muir Glacier:                9200000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41057709-04:00","logger":"reader.geth","message":" - Berlin:                      12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410593931-04:00","logger":"reader.geth","message":" - London:                      12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410607918-04:00","logger":"reader.geth","message":" - Arrow Glacier:               13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.4106357-04:00","logger":"reader.geth","message":" - Gray Glacier:                15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41065043-04:00","logger":"reader.geth","message":"merge configured:","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410661111-04:00","logger":"reader.geth","message":" - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41067195-04:00","logger":"reader.geth","message":" - Network known to be merged: true","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410682275-04:00","logger":"reader.geth","message":" - Total terminal difficulty:  58750000000000000000000","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410693105-04:00","logger":"reader.geth","message":" - Merge netsplit block:       <nil>","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410704011-04:00","logger":"reader.geth","message":"number: 6158490","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.41072913-04:00","logger":"reader.geth","message":"hash: 0xb6e729d033a32968a21450a3408c8b5ab4566e0c5e1bd112017519d21b3720d4","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410747068-04:00","logger":"reader.geth","message":"error: could not apply tx 113 [0xa9cb5b1c828218ee6a334dc9d48ed2c9367f6f35fbb644a1d66c4178056b1927]: insufficient funds for gas * price + value: address 0x80f2b4ff20c8Ff8835cb815094E91E5b0120CD0B have 0 want 14382095304347515","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410762094-04:00","logger":"reader.geth","message":"##############################","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.410772044-04:00","logger":"reader.geth","message":" ","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"ERROR","timestamp":"2022-10-31T15:23:26.410808153-04:00","logger":"reader.geth","message":"eRROR[10-31|15:23:26.410] Beacon backfilling failed                err=\"retrieved hash chain is invalid: could not apply tx 113 [0xa9cb5b1c828218ee6a334dc9d48ed2c9367f6f35fbb644a1d66c4178056b1927]: insufficient funds for gas * price + value: address 0x80f2b4ff20c8Ff8835cb815094E91E5b0120CD0B have 0 want 14382095304347515\"","logging.googleapis.com/labels":{},"serviceContext":{"service":"unknown"}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"INFO","timestamp":"2022-10-31T15:23:26.874415245-04:00","logger":"index-builder","message":"reading from blocks store: file does not (yet?) exist, retrying in","filename":"merged-blocks/0006158300.dbin.zst","base_filename":"0006158300","retry_delay":4,"logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:26.896155225-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:27.399447259-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}
firehose-mainnet-7457c6d7f7-5w7pk firehose {"severity":"WARNING","timestamp":"2022-10-31T15:23:27.901626774-04:00","logger":"reader","message":"failed to upload file","error":"moving file \"0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default\" to storage: open file: open /data/firehose/reader/work/uploadable-oneblock/0006158489-8057bdc930322e92-016b7dd2cfb0000c-6158289-default.dbin.zst: disk quota exceeded","logging.googleapis.com/labels":{}}

Is this a known issue? Could it be that my geth instance isn't syncing properly? Might the bad block be a red herring for the disk quota error?

Indexing: Handle LogAddressIndex queries via an IndexProvider

We want to be able to find and serve LogAddressIndex indexes

Acceptance Criteria:

create a LogAddressIndexProvider conforming to the bstream.IndexProvider interface
determine if a given block number is within an index
determine if a given block number matches an element in an index
fetch and return all next matching indexes for a given block number

Mindreader error on startup "unsupported log line"

I am trying to start the sfeth process but it shutdown immediately due to a mindreader error.

reading from console logs {"error": "unsupported log line: \"INIT 2.0 geth 1.10.21-fh2-86d99626\""}

2022-08-07T00:29:19.460Z (sfeth) starting with config file './sf.yaml'
2022-08-07T00:29:19.460Z (sfeth) starting atomic level switcher {"listen_addr": "localhost:1065"}
2022-08-07T00:29:19.460Z (sfeth) launching applications: firehose,merger,mindreader-node,relayer
2022-08-07T00:29:20.503Z (mindreader.geth) initializing deep mind 
2022-08-07T00:29:20.511Z (mindreader.geth) deep mind initialized                    enabled=true sync_instrumentation_enabled=true mining_enabled=false block_progress_enabled=false compaction_disabled=false archive_blocks_to_keep=0 genesis_provenance="Geth Default"
2022-08-07T00:29:20.511Z (mindreader) reading from console logs {"error": "unsupported log line: \"INIT 2.0 geth 1.10.21-fh2-86d99626\""}
2022-08-07T00:29:20.512Z (sfeth) app mindreader-node triggered clean shutdown
2022-08-07T00:29:20.512Z (mindreader) {"status": {"Cmd":"./geth","PID":3901,"Exit":-1,"Error":{},"StartTs":1659832160470126042,"StopTs":1659832160512767430,"Runtime":0.042641391,"Stdout":null,"Stderr":null}} command terminated with non-zero status, last log lines:
<None>
2022-08-07T00:29:20.512Z (sfeth) application mindreader-node triggered a clean shutdown, quitting
2022-08-07T00:29:20.512Z (sfeth) waiting for all apps termination...
2022-08-07T00:29:20.513Z (sfeth) all apps terminated gracefully
2022-08-07T00:29:20.513Z (sfeth) goodbye

Sometimes, firehose sends block below the actual cursor block

From this graph-node log:

2023-04-07 17:54:49.380	DEBG 0 candidate triggers in this block, block_hash: 0xbdf42b9d117c0c0ac8d0882a957d703957f08fda98977ef2d902f9f3c5219bdc, block_number: 8791730, sgd: 536172, subgraph_id: QmS2GCuAkzH2kNDYe2pA9HkRTPLpC5DpbXRqhQW93exZEM, component: SubgraphInstanceManager
2023-04-07 18:35:13.918	ERRO An error occurred while streaming blocks: status: Internal, message: "unexpected stream termination", details: [], metadata: MetadataMap { headers: {} }, provider: goerli-firehose-pinax, deployment: QmS2GCuAkzH2kNDYe2pA9HkRTPLpC5DpbXRqhQW93exZEM, sgd: 536172, subgraph_id: QmS2GCuAkzH2kNDYe2pA9HkRTPLpC5DpbXRqhQW93exZEM, component: FirehoseBlockStream
2023-04-07 18:35:14.418	INFO Blockstream disconnected, connecting, provider_err_count: 0, cursor: iuBX60B1d2impcutsaAcFKWwLpcyDl1mVAzmKhsT0d3y8iDMiZynBzJ1Ox2Bw6Gk3R3jSQul29iZE354-8VX6tHilew15CkxQXx5xYu68rTtLvqkOlkec7prDb7daNDcUj3RZQ3xfrBT4tXgMvHYZkAwMMV1KjK2jWsCpNBcIvUX7CY1w2n_esqH0vjFpIBJ-bcjEOPylCKiVj0pJRwPPMWDbvXNvA==, subgraph: QmS2GCuAkzH2kNDYe2pA9HkRTPLpC5DpbXRqhQW93exZEM, start_block: 8784178, endpoint_uri: goerli-firehose-sf, provider: goerli-firehose-sf, deployment: QmS2GCuAkzH2kNDYe2pA9HkRTPLpC5DpbXRqhQW93exZEM, sgd: 536172, subgraph_id: QmS2GCuAkzH2kNDYe2pA9HkRTPLpC5DpbXRqhQW93exZEM, component: FirehoseBlockStream
2023-04-07 18:35:27.453	INFO Blockstream connected, provider: goerli-firehose-sf
2023-04-07 18:35:27.453	DEBG 0 candidate triggers in this block, block_hash: 0x787ac44d2913367a0c6286411f204be77175789888b430fc18757bfbf5520d5d, block_number: 8791700
2023-04-07 18:35:27.453	ERRO Subgraph writer failed, error: subgraph `QmS2GCuAkzH2kNDYe2pA9HkRTPLpC5DpbXRqhQW93exZEM` has already processed block `8791700`

What happened frmo the graph-node perspective:

At 17:54, it received block 8791730 (it was connected to Pinax)
Then no more logs until 18:35, so it seems to have been stuck.
At 18:35, it got an unexpected termination (maybe caused by timeout or a service restart from pinax, hence the 'getting stuck')
it reconnected with: start_block=8784178 and a cursor that points to 8791730 with the correct canonical hash (I decoded it here with opaque tool)
It seems to have then received blocks from streamingfast firehose starting from 8791700. (I also saw block 8791701 come in in truncated log before the error was triggered)

I tried reproducing the behavior here without success.

The INFO level of logs does not allow me to confirm if the block 8791700 was actually sent from firehose, because it only prints at every "modulo 200".

It happened to at least 2 different subgraphs connecting to firehose, at exactly the same block.

Hypothesis:

some kind of race condition that depends on where the block is (in reversible segment, written to file, maybe in the buffer in between). It is possible that the first (pinax) firehose endpoint got stuck for long enough for the head block to get written to disk, and both subgraphs connected at exactly the same moment where that file got written, so it was still in the buffer...

merger reads files 1 one thread and deletes them in a another thread?

race condition? merger reads files 1 one thread and deletes them in a another thread?

################################################################
Fatal error in app merger:
retreiving one block files: fetching one block files: processing object list: unable to read one block: 0014255900-20220110T225705.0-24f2e12c-ed760311-dom25:unable to read file header: EOF
################################################################
(launcher/launcher.go:174)

Polygon Deposit Transactions are not part of block

Polygon has special transaction that Deposit assets from mainnet to polygon chain.
These transactions are not part of Firehose blocks.

Example Transaction: https://polygonscan.com/tx/0x8f3e05c5af7d601a6015c4fdbb68a04cbf0305fe2740920ce70f81ed1943a194

As per the release notes V1.3.6. State sync transactions should be tracked. But firehose node is not tracking it.
https://github.com/streamingfast/firehose-ethereum/releases/tag/v1.3.6

This release fixes polygon "StateSync" transactions by grouping the calls inside an artificial transaction.

Discord Thread: https://discord.com/channels/666749063386890256/982135810742697984/1142136382853234719

merger should delete old files upon detection

Merger keeps finding new files that are old, but it doesn't delete them for some reason.

Re: #10

It seems to wait until it can merge a block, then delete the old files. But it might never merge a block because there are too many old files.

Cannot start syncing with BSC mainnet

Firehose cannot start syncing spewing WARNs like this

WARN [01-23|07:00:25.852] Snapshot extension registration failed   peer=5a1afad4 err="peer connected on snap without compatible eth support"
WARN [01-23|07:00:26.322] Synchronisation failed, dropping peer    peer=ac4488f11a86a97c2c1331743f92339e9764681ae484e1a452aee69caef8a4b3 err=timeout
WARN [01-23|07:00:26.322] Synchronisation failed, retrying         err="peer is unknown or unhealthy"
WARN [01-23|07:00:26.322] Synchronisation failed, retrying         err="peer is unknown or unhealthy"

config:

start:
  args:
  # Define Firehose components that will be used for this setup
  - merger
  - firehose
  - reader-node
  - relayer
  - combined-index-builder
  flags:
    # Prevent creation of file for logging (logs from previous sessions are not visible with this setting)
    log-to-file: true

    # Ethereum is the default chain, its id is 1. Update these flags for different chains such as Binance
    common-chain-id: "56"
    common-network-id: "56"
    # Update the reader-node-path to reference the geth binary for the chain and OS being targeted
    reader-node-path: ../geth/geth_linux
    # Update the reader-code-arguments for the chain being targeted
    # Find specific values in the Firehose Ethereum Setup Documentation
    reader-node-arguments: "+--maxpeers 500 --cache 4096 --snapshot=false --bootnodes=enode://1cc4534b14cfe351ab740a1418ab944a234ca2f702915eadb7e558a02010cb7c5a8c295a3b56bcefa7701c07752acd5539cb13df2aab8ae2d98934d712611443@52.71.43.172:30311,enode://28b1d16562dac280dacaaf45d54516b85bc6c994252a9825c5cc4e080d3e53446d05f63ba495ea7d44d6c316b54cd92b245c5c328c37da24605c4a93a0d099c4@34.246.65.14:30311,enode://5a7b996048d1b0a07683a949662c87c09b55247ce774aeee10bb886892e586e3c604564393292e38ef43c023ee9981e1f8b335766ec4f0f256e57f8640b079d5@35.73.137.11:30311"
    reader-node-log-to-zap: false
    reader-node-bootstrap-data-url: ../geth/mainnet/genesis.json

    firehose-grpc-listen-addr: ":9000"
    substreams-enabled: true
    substreams-sub-request-parallel-jobs: 10
    substreams-partial-mode-enabled: true
    substreams-rpc-endpoints: "$BSC_SUBSTREAMS_RPC_ENDPOINT"
    substreams-sub-request-block-range-size: 100
    substreams-stores-save-interval: 100

When substream client restart, firehose doesn't return block from the cursor, it returns older block

If there is restart from consumer, firehose server always return blocks from nearest block in multiple of batch size.

Sink code also throws error

block continuity on first block after restarting from cursor does not follow

https://github.com/streamingfast/substreams-consumer/blob/develop/main.go#L290

For example If batch size of 1000, if client restarts & cursor is at 10543445. Firehose will start sending block from 10543000.

    substreams-sub-request-block-range-size: 1000

Here is conversation in discord for more details:
https://discord.com/channels/666749063386890256/1041633976034533406/1060884898560364555

error: failed storing block in archiver, shutting down. You will need to reprocess over this range to get this block (mindreader/mindreader.go:275){"error": "blocks non contiguous, expectedBlock: 5573454, got block: 5573455"...

Ran into a problem:

error: failed storing block in archiver, shutting down. You will need to reprocess over this range to get this block (mindreader/mindreader.go:275){"error": "blocks non contiguous, expectedBlock: 5573454, got block: 5573455"...

2 issues here:

why does mindreader fail to read the correct block,
why does the entire process not just exit? Instead we just left with a partially working system. no firehose blocks are generated.

Other than reading logs, what is the correct way to detect this problem?

And how to recover? Restarting mindreader just gets stuck again at the same spot. I don't care about the hole in merged blocks. Another mindreader (running on another server) will fill it in eventually.

merger: how to increase added_files_count

The added_files_count never goes past 2000. How to increase it?

2022-01-10T23:01:24.787Z (merger) retrieved list of files (merger/merger.go:179){"too_old_files_count": 0, "added_files_count": 2000}

Question: Implement compaction into parquet files

Currently protobuf files reach terabytes of disk space.

Maybe, there can be a compaction process that packed the data into parquet files when the protobuf file reaches a certain treshold?

There is a mature ecosystem in rust to handle parquet with arrow and parquet crates.

Is this prossible to implement which the current setup?

it sometimes takes mindreader a while to figure out geth is complete on shutdown

This goes on until systemd reaches the programmed timeout and hard kills it. I've seen this many, many times.

Seen both on BSC and ETH.

At least it is a clean shutdown so geth will start properly next time.

Config for BSC as example:

start:
  args:
  - mindreader-node
  flags:
    verbose: 2
    log-to-file: false
    common-chain-id: "56"
    common-network-id: "56"
    common-blocks-store-url: s3://ceph/.....
    common-oneblock-store-url: s3://ceph/.....
    mindreader-node-grpc-listen-addr: :9000
    mindreader-node-oneblock-suffix: servername
    mindreader-node-path: /usr/bin/geth
    mindreader-node-data-dir: /var/lib/geth
    mindreader-node-arguments: "+--port=30300 --cache=16384 --snapshot=false"
    mindreader-node-merge-and-store-directly: false
    mindreader-node-bootstrap-data-url: /etc/dfuse/genesis.json

This is not reproducible on demand. I'll try to note when it happens going forward.

/healthz returns "is_ready: true" when shutting down

When using common-system-shutdown-signal-delay: 30s this gives time for the load-balancer to stop sending traffic to the endpoint before it shuts down. However /healthz URL returns is_ready: true during this time, so there is no way for the load balancer to understand it is in shutdown state, making the flag "useless".

confusing block numeration on names of merged-block files

i had a broken merged-block file. was not easy to spot as the numeration was not clear. here my observations: ( matic )

fireeth will crash with this error:
index builder exited {"error": "processing of file \"0004933300\" failed: error processing incoming file: failed reading next dbin message: corrupt stream, did not find end of stream"}

then i can scan:

fireeth tools check merged-blocks /merged-blocks -r 4933300:4933399    
Summary:
> 🆗 No hole found

but going further see:

fireeth tools check merged-blocks /merged-blocks -r 4933300:4933600 -s
...
Block #4933499 (0fab663c4a0ebf45e664e693ee6418d1e68cfdb5cc4540c3d4879fe627a37c51) 201 transactions, 1005 calls

2023-05-16T08:41:25.636+0200 INFO (dtracing) registering development exporters from environment variables
panic: unable to decode block #4933500 (91c4abc9fd9da55554dcd7d5e94e498835850b9b7c6d1f42bc07c21f8f884cca) payload (kind: ETH, version: 3, size: 1332588, sha256: c77cda5e09bbdacf61e19343710e15507674b899ae405db0c060e657488ee9c9): unable to decode payload: proto:<C2><A0>cannot parse invalid wire-format data

-> file 0004933500 was broken. restoring from backup helped

so somebody seeing the error "processing of file 0004933300 failed" has to make the mental transfer, to check the file TWO files after the file, here 0004933500

merger used to count how many one block files were available

merger used to count how many one block files were available. Now get messages like:

bundle not completed after retrieving one block file (merger/merger.go:112){"bundle": "bundle_size: 100, last_merge_block_num: 0, inclusive_lower_block_num: 14257400, exclusive_highest_block_limit: 14257500"}

Please put back the old behaviour where it showed how many one-block files have been received for the 100-block bundle.

`unable to decode payload: proto: cannot parse invalid wire-format data`

I tried to sync firehose-polygon but I got unable to decode payload: proto: cannot parse invalid wire-format data error and all clients are crashed.

fireeth version

$  fireeth --version
fireeth version 1.3.7 (Commit 4ffb127, Built 2023-03-20T13:59:22Z)

Polygon Execution Layer version

~$ geth version
Bor
Version: 0.3.7-stable-fh2
Git Commit: e2931fd6b192eac721f8550315168391dd6e4805
Architecture: amd64
Go Version: go1.19.3
Operating System: linux
GOPATH=/usr/local/go
GOROOT=
root@thegraph-mainnet-firehose1-hetzner-fsn1:~#

config

~$ cat firehose-config.yaml
start:
  args:
    - merger
    - firehose
    - reader-node
    - relayer
    - combined-index-builder
  flags:
    data-dir: /mnt/firehose
    log-to-file: false
    metrics-listen-addr: 0.0.0.0:9102

    common-chain-id: "137"
    common-network-id: "137"

    reader-node-path: /usr/local/bin/geth
    reader-node-ipc-path: /mnt/bor/bor.ipc
      # common-merged-blocks-store-url: /mnt/firehose/polygon/merged-blocks

    reader-node-arguments: "--bor-mainnet --ipcpath=/mnt/bor/bor.ipc --datadir=/mnt/bor/data --bor.heimdall=http://127.0.0.1:1317 --http --http.api=eth,net,web3 --http.port=8545 --http.addr=127.0.0.1 --http.vhosts=* --firehose-enabled --port=30303 --cache=2048"
    # reader-node-log-to-zap: false

Error log

2023-04-03T23:57:00.629Z INFO (merger) starting pruning of unused (old) one-block-files {"pruning_distance_to_lib": 100, "time_between_pruning": "1m0s"}
2023-04-03T23:57:00.629Z INFO (reader) creating new command instance and launch read loop {"binary": "/usr/local/bin/geth", "arguments": ["--bor-mainnet", "--ipcpath=/mnt/bor/bor.ipc", "--datadir=/mnt/bor/data", "--bor.heimdall=http://127.0.0.1:1317", "--http", "--http.api=eth,net,web3", "--http.port=8545", "--http.addr=127.0.0.1", "--http.vhosts=*", "--firehose-enabled", "--port=30303", "--cache=2048"]}
2023-04-03T23:57:00.629Z INFO (dgrpc) launching gRPC server {"listen_addr": ":13012"}
2023-04-03T23:57:00.630Z INFO (dgrpc) serving gRPC {"listen_addr": ":13012"}
2023-04-03T23:57:00.630Z INFO (merger) starting pruning of forked files {"pruning_distance_to_lib": 50000, "time_between_pruning": "1m0s"}
2023-04-03T23:57:00.630Z INFO (reader) successfully start service
2023-04-03T23:57:00.630Z INFO (reader) operator ready to receive commands
2023-04-03T23:57:00.630Z INFO (reader) starting consume flow
2023-04-03T23:57:00.630Z INFO (relayer.src.13010) starting block source consumption {"endpoint_url": ":13010"}
2023-04-03T23:57:00.630Z INFO (relayer.src.13010) block stream source reading messages {"endpoint_url": ":13010"}
2023-04-03T23:57:00.630Z INFO (reader.sub.relayer) receive block request {"trace_id": "4f9ecb873b4cfb3b31db2ae3cadf52d6", "request": {"burst":1,"requester":"relayer"}}
2023-04-03T23:57:00.631Z INFO (reader.sub.relayer) sending burst {"busrt_size": 0}
2023-04-03T23:57:00.631Z INFO (reader) subscribed {"new_length": 1, "subscriber": "relayer"}
2023-04-03T23:57:00.642Z INFO (index-builder) index builder running
2023-04-03T23:57:00.642Z INFO (index-builder) processing incoming blocks request {"req_start_block": 24800000, "req_cursor": "", "req_stop_block": 0, "final_blocks_only": true}
2023-04-03T23:57:00.642Z INFO (index-builder) creating new joining source {"cursor": "<nil>", "start_block_num": 24800000}
2023-04-03T23:57:00.663Z INFO (reader) read firehose instrumentation init line {"fh_version": "2.2", "node_variant": "polygon", "node_version": "0.3.7-stable-fh2-e2931fd6"}
2023-04-03T23:57:01.621Z INFO (merger) resetting bundler base block num {"previous_base_block_num": 0, "new_base_block_num": 40894100, "lib": "#40894099 (f6b1d2c12804bf25)"}
2023-04-03T23:57:02.629Z WARN (reader) geth Monitor cannot get info from IPC socket {"error": "dial unix /mnt/bor/bor.ipc: connect: no such file or directory"}
2023-04-03T23:57:02.774Z WARN (reader.geth) enabling snapshot recovery               chainhead=40,692,459 diskbase=40,693,149
2023-04-03T23:57:02.778Z WARN (reader.geth) loaded snapshot journal                  diskroot=5f84ff..9e06b4 diffs=unmatched
2023-04-03T23:57:02.778Z WARN (reader.geth) snapshot is not continuous with chain    snaproot=5f84ff..9e06b4 chainroot=f41ee7..3eaad4
2023-04-03T23:57:02.821Z WARN (reader.geth) old unclean shutdowns found              count=156
2023-04-03T23:57:02.821Z WARN (reader.geth) unclean shutdown detected                booted=2022-03-15T08:27:58+0000 age=1y3w3d
2023-04-03T23:57:02.821Z WARN (reader.geth) unclean shutdown detected                booted=2022-03-15T09:07:12+0000 age=1y3w3d
2023-04-03T23:57:02.821Z WARN (reader.geth) unclean shutdown detected                booted=2022-03-15T09:46:08+0000 age=1y3w3d
2023-04-03T23:57:02.821Z WARN (reader.geth) unclean shutdown detected                booted=2022-03-15T10:25:12+0000 age=1y3w3d
2023-04-03T23:57:02.821Z WARN (reader.geth) unclean shutdown detected                booted=2022-03-15T11:04:57+0000 age=1y3w3d
2023-04-03T23:57:02.821Z WARN (reader.geth) unclean shutdown detected                booted=2022-03-15T11:44:37+0000 age=1y3w3d
2023-04-03T23:57:02.821Z WARN (reader.geth) unclean shutdown detected                booted=2022-03-15T12:43:50+0000 age=1y3w3d
2023-04-03T23:57:02.821Z WARN (reader.geth) unclean shutdown detected                booted=2022-12-10T10:54:49+0000 age=3mo3w3d
2023-04-03T23:57:02.821Z WARN (reader.geth) unclean shutdown detected                booted=2022-12-10T17:02:34+0000 age=3mo3w3d
2023-04-03T23:57:02.821Z WARN (reader.geth) unclean shutdown detected                booted=2023-04-03T17:47:38+0000 age=6h9m24s
2023-04-03T23:57:02.887Z WARN (reader.geth) unable to whitelist checkpoint - first run err="missing checkpoint blocks"
2023-04-03T23:57:08.213Z INFO (index-builder) blocks archive streaming was asked to stop
2023-04-03T23:57:08.213Z INFO (index-builder) index builder exited {"error": "processing of file \"0024802400\" failed: error processing incoming file: failed reading next dbin message: unexpected EOF"}
2023-04-03T23:57:08.213Z ERRO (fireeth)
################################################################
Fatal error in app combined-index-builder:

processing of file "0024802400" failed: error processing incoming file: failed reading next dbin message: unexpected EOF
################################################################
2023-04-03T23:57:08.213Z INFO (fireeth) application combined-index-builder shutdown unexpectedly, quitting
2023-04-03T23:57:08.213Z INFO (fireeth) waiting for all apps termination...
2023-04-03T23:57:08.213Z INFO (firehose) gracefully stopping the gRPC server {"timeout": "1s"}
2023-04-03T23:57:08.213Z INFO (firehose) gRPC server teminated gracefully
2023-04-03T23:57:08.213Z INFO (reader) operator is terminating {"error": "processing of file \"0024802400\" failed: error processing incoming file: failed reading next dbin message: unexpected EOF"}
2023-04-03T23:57:08.213Z INFO (reader) superviser is terminating
2023-04-03T23:57:08.213Z INFO (reader) supervisor received a stop request, terminating node process
2023-04-03T23:57:08.213Z INFO (reader) stopping underlying process
2023-04-03T23:57:08.213Z INFO (dgrpc) forcing gRPC server to stop
2023-04-03T23:57:08.213Z ERRO (reader.geth) ERROR[04-03|23:57:08.213] Failed to journal state snapshot         err="snapshot [0xf41ee783cdbce53430a600c3d6fbaaba90b422785eda463eeb73bc73c83eaad4] missing"
panic: unable to decode block #24802499 (f36a8609a3818244d3ed85061676f0a336dfc8ec0214f5f0bd25d772003902f9) payload (kind: ETH, version: 3, size: 1445841, sha256: eaed3b16e23cc82dca75df9e09d748e66990605d7a4d0ec4c4baabfe03d98ea8): unable to decode payload: proto: cannot parse invalid wire-format data

Payload: 08031220f36a8609a3818244d3ed85061676f0a336dfc8ec0214f5f0bd25d772003902f918c3e9e90b20aeb0072a96050a20cf37e6d4d69d26a4f65199f7650f2415476d37b063c54e4b6ebd4ea346383c1e12201dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d493471a14000000000000000000000000000000000000000022208c2f83ff
...
...
...
goroutine 3034 [running]:
github.com/streamingfast/bstream.(*Block).ToProtocol(0xc00191a180)
        /go/pkg/mod/github.com/streamingfast/[email protected]/block.go:246 +0x6c9
github.com/streamingfast/firehose.glob..func1(0x120e286?)
        /go/pkg/mod/github.com/streamingfast/[email protected]/factory.go:24 +0x19
github.com/streamingfast/bstream.(*FileSource).preprocess(0xc0019c00c0, 0xc00191a180, 0xc00fcd5f80)
        /go/pkg/mod/github.com/streamingfast/[email protected]/filesource.go:496 +0x5b
created by github.com/streamingfast/bstream.(*FileSource).streamReader
        /go/pkg/mod/github.com/streamingfast/[email protected]/filesource.go:485 +0x68a

Error Unimplemented desc = unknown service sf.substreams.internal.v2.Substreams"

Indexing with substream is not working after update fireeth and graph-node.

Versions

fireeth: fireeth version 1.4.0 (Commit 8d0beff, Built 2023-05-05T14:54:49Z)
graph-node: graphprotocol/graph-node:v0.31.0-rc.0

Issued subgraph

QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw

Configs

fireeth config

start:
  args:
  - merger
  - firehose
  - reader-node
  - relayer
  - combined-index-builder
  flags:
    data-dir: /mnt/firehose
    firehose-grpc-listen-addr: 0.0.0.0:13042
    reader-node-ipc-path: /mnt/el/geth.ipc
    common-chain-id: "1"
    common-network-id: "1"
    reader-node-path: /usr/local/bin/geth
    substreams-rpc-endpoints: https://ethereum-mainnet-archive.allthatnode.com
    substreams-enabled: true
    substreams-request-stats-enabled: true
    reader-node-arguments: "--mainnet --port 30303 --cache 2048 --http --http.api=eth,net,web3 --http.addr 0.0.0.0 --http.port 8545 --http.vhosts=* --authrpc.addr 127.0.0.1 --authrpc.port 8551 --authrpc.jwtsecret /mnt/el/jwt.hex --gcmode full --datadir /mnt/el --ipcpath /mnt/el/geth.ipc --firehose-enabled --maxpeers=200 --metrics --metrics.addr 0.0.0.0 --metrics.port 7071"

graph-node config.


[chains.mainnet]
shard = "primary"
protocol = "ethereum"

provider = [
        { label = "eth-rpc-atn-archive", url = "https://ethereum-mainnet-archive.allthatnode.com/", features = [ "archive" ] },
        { label = "eth-firehose", details = { type = "substreams", url = "http://-.-.-.-:13042" } }
]

Logs

Error logs from fireeth.

2023-05-06T03:34:00.538Z INFO (firehose.tier1) job completed {"trace_id": "19df1b8bd94c148f72414f3a43d38a29", "job": {"module_name": "store_supply", "start_block": 12975000, "end_block": 12985000}, "error": "receiving stream resp: rpc error: code = Unimplemented desc = unknown service sf.substreams.internal.v2.Substreams"}
2023-05-06T03:34:00.538Z INFO (firehose.tier1) shutting down request stats {"trace_id": "19df1b8bd94c148f72414f3a43d38a29"}
2023-05-06T03:34:00.538Z INFO (firehose.tier1) unexpected termination of stream of blocks {"trace_id": "19df1b8bd94c148f72414f3a43d38a29", "error": "error building pipeline: failed setup request: parallel processing run: scheduler run: process job result for target \"store_supply\": worker ended in error: receiving stream resp: rpc error: code = Unimplemented desc = unknown service sf.substreams.internal.v2.Substreams"}
2023-05-06T03:34:00.538Z INFO (firehose.tier1) worker failed with retryable error {"trace_id": "19df1b8bd94c148f72414f3a43d38a29", "error": "receiving stream resp: rpc error: code = Unimplemented desc = unknown service sf.substreams.internal.v2.Substreams"}
2023-05-06T03:34:00.538Z ERRO (firehose) finished streaming call with code Internal {"grpc.start_time": "2023-05-06T03:33:54Z", "system": "grpc", "span.kind": "server", "grpc.service": "sf.substreams.rpc.v2.Stream", "grpc.method": "Blocks", "peer.address": "142.93.100.242:59116", "error": "rpc error: code = Internal desc = error building pipeline: failed setup request: parallel processing run: scheduler run: process job result for target \"store_supply\": worker ended in error: receiving stream resp: rpc error: code = Unimplemented desc = unknown service sf.substreams.internal.v2.Substreams", "grpc.code": "Internal", "grpc.time_ms": 6008.35693359375}
2023-05-06T03:34:00.538Z INFO (firehose.tier1) job completed {"trace_id": "19df1b8bd94c148f72414f3a43d38a29", "job": {"module_name": "store_supply", "start_block": 12985000, "end_block": 12995000}, "error": "receiving stream resp: rpc error: code = Unimplemented desc = unknown service sf.substreams.internal.v2.Substreams"}
2023-05-06T03:34:00.538Z INFO (firehose.tier1) worker failed with retryable error {"trace_id": "19df1b8bd94c148f72414f3a43d38a29", "error": "receiving stream resp: rpc error: code = Unimplemented desc = unknown service sf.substreams.internal.v2.Substreams"}
2023-05-06T03:34:00.538Z INFO (firehose.tier1) launching remote worker {"trace_id": "19df1b8bd94c148f72414f3a43d38a29", "start_block_num": 13015000, "stop_block_num": 13025000, "output_module": "store_supply"}
2023-05-06T03:34:00.538Z INFO (firehose.tier1) worker failed with a non-retryable error {"trace_id": "19df1b8bd94c148f72414f3a43d38a29", "error": "context canceled"}

Tried remove $FIREHOSE_DIR/localdata, but not working.

prometheus exporter does not include tier1 or tier2 in the "ready" status

The prometheus exporter does not include susbtreams tier1 or tier2 in the "ready" status.

Seen on substreams tier2, for example:

# HELP ready readiness of an app. 1 if ready, 0 otherwise
# TYPE ready gauge
ready{app="block-indexer"} 0
ready{app="firehose"} 0
ready{app="merger"} 0
ready{app="relayer"} 0

merger no loner publishes head_block_time_drift or head_block_number

In the latest sfeth, the merger no loner publishes head_block_time_drift or head_block_number. How to tell if the merger is healthy now?

# HELP head_block_number
# TYPE head_block_number gauge
head_block_number{app="merger"} 1.607925e+08
# HELP head_block_time_drift Number of seconds away from real-time
# TYPE head_block_time_drift gauge
head_block_time_drift{app="merger"} 4.245490579

streamingfast / firehose-ethereum Goto Github PK

firehose-ethereum's Introduction

Firehose for Ethereum

Release

Docker Bundle Image Building

Contributing

License

firehose-ethereum's People

Contributors

Stargazers

Watchers

Forkers

firehose-ethereum's Issues

Genesis block storage changes determinism issue

Transaction root call BeginOrdinal is always 0

Genesis block root call missing BeginOrdinal/EndOrdinal

Transaction#return_data is not populated

Some Call#GasChanges are missing

Call#input not populate on contract creation

Call#executed_code not true on some situation where it should

TransactionTrace enforced state changes

fireeth version

Polygon Execution Layer version

config

Error log

Versions

Issued subgraph

Configs

Logs

Recommend Projects

Recommend Topics

Recommend Org

Transaction root call `BeginOrdinal` is always 0

Genesis block root call missing `BeginOrdinal/EndOrdinal`

`Transaction#return_data` is not populated

Some `Call#GasChanges` are missing

`Call#input` not populate on contract creation

`Call#executed_code` not `true` on some situation where it should

`TransactionTrace` enforced state changes