mempool / electrs

An efficient re-implementation of Electrum Server in Rust

The Mempool Open Source Project®

mempool-github.mp4

Mempool is the fully-featured mempool visualizer, explorer, and API service running at mempool.space.

It is an open-source project developed and operated for the benefit of the Bitcoin community, with a focus on the emerging transaction fee market that is evolving Bitcoin into a multi-layer ecosystem.

Installation Methods

Mempool can be self-hosted on a wide variety of your own hardware, ranging from a simple one-click installation on a Raspberry Pi full-node distro all the way to a robust production instance on a powerful FreeBSD server.

Most people should use a one-click install method.

Other install methods are meant for developers and others with experience managing servers. If you want support for your own production instance of Mempool, or if you'd like to have your own instance of Mempool run by the mempool.space team on their own global ISP infrastructure—check out Mempool Enterprise®.

One-Click Installation

Mempool can be conveniently installed on the following full-node distros:

• Umbrel
• RaspiBlitz
• RoninDojo
• myNode
• StartOS
• nix-bitcoin

We highly recommend you deploy your own Mempool instance this way. No matter which option you pick, you'll be able to get your own fully-sovereign instance of Mempool up quickly without needing to fiddle with any settings.

Advanced Installation Methods

Mempool can be installed in other ways too, but we only recommend doing so if you're a developer, have experience managing servers, or otherwise know what you're doing.

• See the docker/ directory for instructions on deploying Mempool with Docker.
• See the backend/ and frontend/ directories for manual install instructions oriented for developers.
• See the production/ directory for guidance on setting up a more serious Mempool instance designed for high performance at scale.

electrs's Issues

crash due to WouldBlock / Resource temporarily unavailable

x-powered-by: mempool-electrs 3.0.0-dev-5ad6733

thread 'thread 'thread 'tokio-runtime-worker' panicked at 'called Result::unwrap() on an Err value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }', thread 'tokio-runtime-workertokio-runtime-worker' panicked at 'thread 'src/new_index/schema.rsthread '' panicked at ':1094:tokio-runtime-worker10
tokio-runtime-workercalled Result::unwrap() on an Err value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }', src/new_index/schema.rsnote: run with RUST_BACKTRACE=1 environment variable to display a backtrace
' panicked at 'called Result::unwrap() on an Err value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }', src/new_index/schema.rs:1094:10
' panicked at 'thread 'called Result::unwrap() on an Err value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }called Result::unwrap() on an Err value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }', src/new_index/schema.rstokio-runtime-worker', :' panicked at 'called Result::unwrap() on an Err value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }', 1094src/new_index/schema.rs:src/new_index/schema.rs1094::101094
:10
:10
:1094:10
tokio-runtime-worker' panicked at 'called Result::unwrap() on an Err value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }', src/new_index/schema.rs:1094:10

Add integration testing

See Blockstream PR 60

Add more logging

More logging for Electrum and REST servers would be nice.

Ideally we could switch over to tracing and make things a bit more extendable.

Liquid: missing transaction prevouts

When querying a transaction from electrs at /tx/:txId, it fails with an error if the transaction has an input with a missing prevout (like liquid peg ins):

> curl https://liquid.network/api/tx/2686128adb8d678b1c852bb15c0c6c98fdac4e9626798a52e5a9827d711d7c2e
Transaction missing prevouts

It looks like this behaviour was introduced by 173d613, since running electrs from the parent commit fixes the issue (mempool-electrs 3.0.0-dev-a21a958):

curl http://localhost:3000/tx/2686128adb8d678b1c852bb15c0c6c98fdac4e9626798a52e5a9827d711d7c2e
{"txid":"2686128adb8d678b1c852bb15c0c6c98fdac4e9626798a52e5a9827d711d7c2e","version":2,"locktime":0,"vin":[{"txid":"7b94f15dbfda5f24b3ff486425b02da915419a03f20793fa85f44de20b0c37e7","vout":0,"prevout":null,"scriptsig":"","scriptsig_asm":"","witness":["304402203e66c3c61690e06a6d8394f4cafccd5a842c81e942a0d970ccdc7179b7c1010302200bcb9e76d90ca0fb0ffbc9e6f9fb61fb892632de5ac81605deda495a306b143c01","032abcbb6f2836c2a701d0f0e92743a0a60a64b86c90156bbc99191bc749988377"],"is_coinbase":false,"sequence":4294967295,"is_pegin":true}],"vout":[{"scriptpubkey":"0014602353deacdcd14d4fb4756591958eb18fb5ebe5","scriptpubkey_asm":"OP_0 OP_PUSHBYTES_20 602353deacdcd14d4fb4756591958eb18fb5ebe5","scriptpubkey_type":"v0_p2wpkh","scriptpubkey_address":"ex1qvq348h4vmng56na5w4jer9vwkx8mt6l9um0h0a","value":809953,"asset":"6f0279e9ed041c3d710a9f57d0c02928416460c4b722ae3457a11eec381c526d"},{"scriptpubkey":"","scriptpubkey_asm":"","scriptpubkey_type":"fee","value":47,"asset":"6f0279e9ed041c3d710a9f57d0c02928416460c4b722ae3457a11eec381c526d"}],"size":1062,"weight":1548,"fee":47,"status":{"confirmed":true,"block_height":2666759,"block_hash":"9c8c4340a290a62043ee6eb91ebd5755d8b49bcfc1c1d8fa010cecf5674000e9","block_time":1704310628}}

It results in the Liquid explorer not being able to display peg in transactions:

Electrum RPC server does not shut down gracefully

When there are active connections with Electrum RPC users, the signal for SIGTERM etc. is not propagated properly, due to tight loops of TcpStreams waiting on peers.

• Convert Electrum RPC to use async (making it much easier to cancel)
• Insert proper cancellation

Fix re-org treatment

1. Add new index with block height + block hash + TxHistoryRow entries for the last 100 blocks.
2. When a reorg occurs, utilize that to clean up the TxHistoryRows and roll them back.
3. StatsCache and UtxoCache to only cache up to 6 blocks deep. This will help prevent incorrect stats information.
   a. Not sure what the performance consequences are for this... maybe 3 blocks deep is more acceptable.

Electrs exits unexpectedly with error code 137

While indexing testnet3, the electrs container will exit unexpectedly with code 137.

15:19:56 src/mempool-docker-compose % dc up electrs
[+] Running 2/0
 ✔ Container mempool-docker-compose-bitcoind-1  Running                                                                                                                                              0.0s
 ✔ Container mempool-docker-compose-electrs-1   Recreated                                                                                                                                            0.0s
Attaching to electrs-1
electrs-1  | Config { log: StdErrLog { verbosity: Warn, quiet: false, show_level: true, timestamp: Off, modules: [], writer: "stderr", color_choice: Never }, network_type: Testnet, db_path: "/electrs-db/testnet", daemon_dir: "/.bitcoin/testnet3", blocks_dir: "/.bitcoin/testnet3/blocks", daemon_rpc_addr: 172.20.0.4:18332, cookie: None, electrum_rpc_addr: 0.0.0.0:50001, http_addr: 127.0.0.1:3001, http_socket_file: None, rpc_socket_file: None, monitoring_addr: 127.0.0.1:14224, jsonrpc_import: false, light_mode: true, main_loop_delay: 500, address_search: false, index_unspendables: false, cors: None, precache_scripts: None, precache_threads: 40, utxos_limit: 500, electrum_txs_limit: 500, electrum_banner: "Welcome to mempool-electrs 3.0.0-dev-f78c07a(dirty)", mempool_backlog_stats_ttl: 10, mempool_recent_txs_size: 10, rest_default_block_limit: 10, rest_default_chain_txs_per_page: 25, rest_default_max_mempool_txs: 50, rest_max_mempool_page_size: 1000 }
electrs-1 exited with code 137

Despite the fact that I have RUST_BACKTRACE set to 1 and am passing the -v flag to the electrs command, no additional information is given.

Restarting the container continues the indexing process (based on a growth of the data directory) but then exits in a similar fashion.

question: upgrade path from Blockstream/electrs?

Hi all

Sorry for creating an issue with just a question, but the discussions section isn't enabled in this project.
Feel free to close immediately if you think I should ask somewhere else.

My question is: I'm currently running the https://github.com/Blockstream/electrs fork of electrs, but would like to upgrade to this one. Can I directly start using a docker image based on this fork or do I need to do a full re-index?

Thanks!

Electrs not indexing unconfirmed transactions as spent

We're seeing this on multiple (but not all) servers in production.

A transaction in the mempool https://node205.tk7.mempool.space/api/tx/b60793ec024f9425d8d7841d592adc8782b3d32921d0ce5a7e894bd62e00973c spending an input f7f8befc733c643f7455744d82e665a9c7dcc6fb37f1c0aa56862340aa0e698b is not registered in the address transactions API and shows up as unspent in the Outspends API endpoint https://node205.tk7.mempool.space/api/tx/f7f8befc733c643f7455744d82e665a9c7dcc6fb37f1c0aa56862340aa0e698b/outspend/344

{"spent":false}

Version mempool-electrs 3.0.0-dev-3c66022

On another server with the exact same githash, it does work https://node206.tk7.mempool.space/api/tx/f7f8befc733c643f7455744d82e665a9c7dcc6fb37f1c0aa56862340aa0e698b/outspend/344

{"spent":true,"txid":"b60793ec024f9425d8d7841d592adc8782b3d32921d0ce5a7e894bd62e00973c","vin":0,"status":{"confirmed":false}}

Add support for Sig Op calculation and adjusted sizes

What we have to do now in the NodeJS backend should have native support in electrs.

We currently only support sigop calculation and adjusted vsize/weight on mempool-transactions, but on mined transactions we lose this. When implemented in Electrs we would be able to see the adjusted vsizes and sigop data in the frontend for mined transactions. This will also affect the CPFP clusters.

Reference:
https://github.com/mempool/mempool/blob/master/backend/src/api/transaction-utils.ts#L147

Bump deps

rust-bitcoin, rust-elements, and basically anything we can. Bump it to recent.

Access to blk files

I see that this version of electrs needs to have read access to .bitcoin/blocks/*.blk files.

Thing is, Bitcoin Core creates them with 600 rights. Because of this, the only user that technically has access those files, is Bitcoin Core's user itself.

Is this app designed to be run with Bitcoin Core's user?

Is there a more secure way to execute it?

Support batched JSON-RPC requests

It would be great to support batched requests - wallet loading is significantly faster, in part due to reduced network activity and data overhead. Electrs, Fulcrum and ElectrumX have supported batched JSON-RPC requests for several years now.

Stale block address indexing bug

Electrs appears to double-count certain transactions during address indexing, which produces incorrect chain stats (funded_txo_sum etc) and therefore incorrect balances displayed on mempool.space.

---

Specifically, the error seems to occur when a transaction is included in a stale block, but not the "winning" block at the same height.

For example, transaction 3738...e88e
was included in a recent stale block 000000000000000000032956cbfd8721abe0572d81542e75e38e6185312bee09 at height 792379, but eventually entered the main chain in block 792380.

The transaction is then double-counted during address indexing,
causing address 132kFhZzkFytWAefWy7nCYgJHcU84RjJNa
to show a negative balance.

Similarly, transaction 3264...d9cb appears in the same stale block, enters the main chain in block 792382, and is counted twice towards address bc1qzuupxjkwv4dj69m3xzack5sdqtacn6l8czndlf, causing a positive balance to display when in fact the address should be empty.

---

Transactions which appear at the same height on both sides of a chain split appear to be handled correctly.

For example, address bc1pfajpg0fp0a62wjrax682hkzvrczxx7j3htrtm0ae6cnx72ye2rnscarxrp has a transaction which is included in both versions of block 792379, but reports the correct balance.

block difficulty value is integer instead of floating point

Moved from Blockstream/esplora#370

How to Reproduce

https://blockstream.info/api/block/000000006e082edaab177e74a7cc0db2a6b908812ae801d535e0f2033aa8657d

Expected Value

"difficulty": 1.182899534312841,

Actual Value

"difficulty":1

Update README

We have changed a lot from Blockstream/electrs... so we should really update the README

electrs occasionally crashes due to EWOULDBLOCK

thread 'thread 'tokio-runtime-workertokio-runtime-worker' panicked at '' panicked at 'called `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }called `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }thread '', thread 'thread 'thread 'tokio-runtime-workertokio-runtime-workerthread '' panicked at 'tokio-runtime-workercalled `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }' panicked at 'thread 'called `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }thread 'tokio-runtime-workerthread '' panicked at 'thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }', src/new_index/schema.rscalled `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }tokio-runtime-worker', thread 'src/new_index/schema.rs' panicked at 'tokio-runtime-workercalled `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }' panicked at ':called `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }1034', :10src/new_index/schema.rs
', note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
:src/new_index/schema.rs' panicked at ':tokio-runtime-worker1034called `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }', src/new_index/schema.rs:thread 'thread 'tokio-runtime-worker:1034:tokio-runtime-workerthread '' panicked at 'called `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }thread '' panicked at 'tokio-runtime-workerthread ':thread '' panicked at 'tokio-runtime-workercalled `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }', src/new_index/schema.rs:1034:thread '10tokio-runtime-worker' panicked at '
called `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }', src/new_index/schema.rs:1034:10
' panicked at 'called `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }', thread '', tokio-runtime-worker' panicked at 'src/new_index/schema.rscalled `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }:', 1034src/new_index/schema.rs::101034
:10
tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }', src/new_index/schema.rs:1034:10
', src/new_index/schema.rs:1034:10
called `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: IOError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }) }', src/new_index/schema.rs:1034:10
tokio-runtime-workersrc/new_index/schema.rs1034tokio-runtime-workerthread '10' panicked at '', ', tokio-runtime-workersrc/new_index/schema.rs101034

Disable sigops for Liquid

Low priority, since the mempool/backend already sets sigops to 0 when Liquid network is active...

But just in case someone uses mempool/electrs for other purposes we'd like to be correct.

As you can see here, Liquid counts sigops slightly differently for peg-in inputs' witnessScripts...

Liquid
https://github.com/ElementsProject/elements/blob/ae023ea98db010953bfc8aaf646b8e81ad30cdcd/src/consensus/tx_verify.cpp#L170-L188

Bitcoin
https://github.com/bitcoin/bitcoin/blob/f01416e23c9c820517c37003a2a98dd46d1022ba/src/consensus/tx_verify.cpp#L158-L164

Rather than code that difference in, it'd probably be simpler to just make sigops a non-Liquid only feature.

Choices (either or):

1. Support Liquid and change the calculations based on liquid feature.
2. Disable sigops when using liquid

Crash with error message: Too many open files

Dear Team,
I ran on ubuntu and got bellow message:

thread 'blkfiles_reader-3' panicked at 'failed to read "/root/.bitcoin/blocks/blk00008.dat": Os { code: 24, kind: Uncategorized, message: "Too many open files" }', src/new_index/fetch.rs:159:41
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Aborted (core dumped)

config:

Config { log: StdErrLog { verbosity: Error, quiet: false, show_level: true, timestamp: Off, modules: [], writer: "stderr", color_choice: Auto }, network_type: Bitcoin, db_path: "./db/mainnet", daemon_dir: "/root/.bitcoin", blocks_dir: "/root/.bitcoin/blocks", daemon_rpc_addr: 127.0.0.1:8332, electrum_rpc_addr: 127.0.0.1:50001, http_addr: 0.0.0.0:5500, http_socket_file: None, rpc_socket_file: None, monitoring_addr: 0.0.0.0:4224, jsonrpc_import: false, light_mode: false, address_search: false, index_unspendables: false, cors: None, precache_scripts: None, precache_threads: 16, utxos_limit: 300000, electrum_txs_limit: 500, electrum_banner: "Welcome to mempool-electrs 3.0.0-dev-b379d24", mempool_backlog_stats_ttl: 10, mempool_recent_txs_size: 10, rest_default_block_limit: 10, rest_default_chain_txs_per_page: 25, rest_default_max_mempool_txs: 50, rest_max_mempool_page_size: 1000 }

version: mempool-electrs 3.0.0-dev-b379d24
Thank you in advance.

Index CPFP clusters

Index the CPFP clusters in rocksdb and offer an API endpoint to query them.

Set a long cache for blocks older than 10.

From Element

{some code: 1 byte}
{cluster id: 32 bytes}
{effective feerate: f64: 8 bytes}
{list of txids excluding root: 32 bytes x 1.3 non-root tx on average}

82.6 bytes per cluster x 180 cluster per block x 800k blocks = 11.8944 GB (probably less since it was only avg for last 15k blocks)

{code: 1}
{txid: 32}
{cluster id: 32}

65 bytes x 2.3 tx per cluster x 180 clusters per block x 800k blocks = 21.528 GB (probably less)

{code: 1}
{blockhash: 32}
{list of indexes of cluster roots: u16 (2 bytes) x 180}

393 bytes per block x 800k blocks = 314.4 MB

future incompatibility warning

warning: `electrs` (lib) generated 3 warnings
    Finished release [optimized] target(s) in 0.12s
warning: the following packages contain code that will be rejected by a future version of Rust: rustc-serialize v0.3.24
note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`

Add option to pass signet-magic for custom signet network

[Feature request] WebTransport over WebSockets

Just better: https://developer.mozilla.org/en-US/docs/Web/API/WebTransport_API