spesmilo / electrumx Goto Github PK
View Code? Open in Web Editor NEWAlternative implementation of spesmilo/electrum-server
License: MIT License
electrumx's People
Contributors
Stargazers
Watchers
Forkers
unitedbitcoin vug1uskr btcatom ivansib groestlcoin thejoulethief sombernight justintarthur tank0226 whizz onnateldome cpeezy81 cryptapus jasonh-ca optisecure savyp mladenilic justthedoctor cipig vitoshi21 jamiejoin easyx-community aussiehash aggressivelurker general-programming pox global19 global-localhost global19-atlassian-net cryptodev777 coderjoe francobenner thekvn smogm cjdelisle dev-helix ca333 subpar42 dev-zeo lxngoddess5321 steliord emzy june-sky-dev dev-bsk shsmith skatefish gertjaap unobtanium-120 thohemp nirvananimbusa ser madcatmining fluidchains primecoin 731mace crypto-node instaksh noirofficial digitalderrickcs vericonomy tribecrypto thilinah clashic-services cryptodevfr rav3npl justinvforvendetta michelvankessel stevenans985900 romanz hyperpeek jeremyrand alexw3f scalaris-project riabow firoorg lulworm pungotoken uniasset raptor3um willyfromtheblock wakiyamap masi-cyber computercraftr imaginaryusername murd4m1773n-m4f14 josephclarkclayton cryptooz decenomy checho1989 xazabevo bitweb-project ciripel freelancedeveloper025 akhavr silverchen0402 m0rty01 widecoin-project captain13128 fastcoin-project vecopayelectrumx's Issues
Compressed block headers & merkle proofs over low bandwidth communications
This issue by @willcl-ark was closed in the kyuupichan/eletrumx parent fork without explanation by the maintainer just before he declared that he would not support Bitcoin anymore.
Is there any interest in this idea here? For supporting low-bandwidth light clients this would be very useful.
blockchain.estimatefee should be cached (instead of doing a daemon RPC every time)
I think it's expensive to call bitcoind in general, so it might be beneficial to eliminate at least the low hanging fruit:
specifically it should be relatively easy to cache "estimatefee" coming from bitcoind, e.g. in a time based cache for 1 minute, and return those values when requested from clients through blockchain.estimatefee from the cache without querying bitcoind.
Name and direction
I want to open a discussion about the name of this project and the direction it is taking.
I guess we are all aware of the history. It looks this fork is taking quite a distinct path from the original work done by Neil. I am hoping it will be more active and flourish. However, the fact that it's still called Electrumx might confuse people (both ways).
Wouldn't it be better if we renamed it to something else and clearly cut it from it's heritage? Something like happened with OpenOffice/LibreOffice, MySQL/MariaDB etc. I am really bad at creative, so I don't have any clever suggestions, but I am sure we can come up with something better than ElectrumY, which is the only thing that came up to me :)
Also, I think it would be better to clearly decide on altcoin support. I, for one, do not care about anything other than Bitcoin, but I also understand that there are people who do. If altcoins are internal to the code, they should be supported and maintained. If the team is not ready to support them, they should be removed. A middle ground could be, that the coin support is somehow modularized and removed from the repo itself, with people able to build and support their own modules, which could be installed externally during the setup process.
At this point, I'd also like to thank all the contributors of this project, you are doing awesome job. And I think we also need to acknowledge the tremendous work Neil has done, as the original author, even though we might not agree with the way he has handled the project in the last few months.
Watchtower protocol support?
Is it possible that the server-side parts of electrum's watchtower protocol could be added to electrumx? Here's my particular setup:
- Home linux server running 24/7 with electrumx
- Home windows desktop running whenever with electrum-qt and the full wallet, connecting to the server
It would be nice to not have to run a second program on my server to get watchtower support.
Server crash soon after start on 'ill-formed response error object: no new valid entries'
Version 1.15.0, Ubuntu focal
Server starts up, but soon after crashes. On first look it seems like a problem with some peer. I'll investigate further. Here are the logs:
ERROR:electrumx:ElectrumX server terminated abnormally
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/EGG-INFO/scripts/electrumx_server", line 35, in main
File "/usr/lib/python3.8/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/lib/server_base.py", line 125, in run
await server_task
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/lib/server_base.py", line 98, in serve
await self.serve(shutdown_event)
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/controller.py", line 134, in serve
await group.spawn(wait_for_catchup())
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/session.py", line 621, in serve
await group.spawn(self._manage_servers())
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/peers.py", line 459, in discover_peers
await group.spawn(self._import_peers())
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/peers.py", line 234, in _monitor_peer
if await self._should_drop_peer(peer):
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/peers.py", line 277, in _should_drop_peer
await self._verify_peer(session, peer)
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/peers.py", line 383, in _verify_peer
await session.send_request('server.add_peer', [features])
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/session.py", line 529, in send_request
return await self._send_concurrent(message, future, 1)
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/session.py", line 499, in _send_concurrent
return await future
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/jsonrpc.py", line 721, in receive_message
item, request_id = self._protocol.message_to_item(message)
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/jsonrpc.py", line 273, in message_to_item
return cls._process_response(payload)
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/jsonrpc.py", line 220, in _process_response
raise cls._error(code, message, False, request_id)
aiorpcx.jsonrpc.ProtocolError: (-32600, 'ill-formed response error object: no new valid entries')
Confusing documentation
I think some small changes should be made to avoid working on the incorrect chain.
At the end README.rst links to the Documentation (https://electrumx-spesmilo.readthedocs.io/) which itself under "Getting Started" links to the installer (https://github.com/bauerj/electrumx-installer).
This link to the installer should make clear, that it links to an istallation, which will only enable BitcoinBSV. This is visible in the file install.sh.
install.sh downloads from "https://github.com/kyuupichan/electrumx" and this repository only includes BitcoinBSV, visible in coins.py (https://github.com/kyuupichan/electrumx/blob/master/electrumx/lib/coins.py)
electrumx reports an error in the container
electrumx reports an error in the container，My error specific information below
INFO:electrumx:ElectrumX server starting
INFO:electrumx:logging level: INFO
INFO:Controller:Python version: 3.7.9 (default, Aug 18 2020, 04:11:54) [GCC 9.3.0]
INFO:Controller:software version: ElectrumX 1.15.0
INFO:Controller:aiorpcX version: 0.18.4
INFO:Controller:supported protocol versions: 1.4-1.4.2
INFO:Controller:event loop policy: None
INFO:Controller:reorg limit is 200 blocks
INFO:Daemon:daemon #1 at 172.17.0.1:18332/ (current)
INFO:DB:switching current directory to /data
INFO:DB:using rocksdb for DB backend
INFO:DB:opened UTXO DB (for sync: True)
INFO:DB:UTXO DB version: 8
INFO:DB:coin: Bitcoin
INFO:DB:network: mainnet
INFO:DB:height: 656,477
INFO:DB:tip: 0000000000000000000457df9961a123212340daf566b899aaa54197d0648d1b
INFO:DB:tx count: 585,898,367
INFO:DB:flushing DB cache at 1,200 MB
INFO:History:history DB version: 1
INFO:History:flush count: 15,018
INFO:SessionManager:RPC server listening on 0.0.0.0:8000
INFO:Prefetcher:catching up to daemon height 656,684 (207 blocks behind)
INFO:BlockProcessor:chain reorg detected
INFO:BlockProcessor:chain was reorganised replacing 1 block at heights 656,477-656,477
INFO:BlockProcessor:read 1 blocks from disk
INFO:Prefetcher:cancelled; prefetcher stopping
INFO:SessionManager:closing down server for rpc://0.0.0.0:8000
INFO:Controller:shutting down
INFO:Controller:shutdown complete
ERROR:electrumx:ElectrumX server terminated abnormally
Traceback (most recent call last):
File "/electrumx/electrumx_server", line 35, in main
asyncio.run(controller.run())
File "/usr/local/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
return future.result()
File "/electrumx/electrumx/lib/server_base.py", line 129, in run
await server_task
File "/electrumx/electrumx/lib/server_base.py", line 102, in serve
await self.serve(shutdown_event)
File "/electrumx/electrumx/server/controller.py", line 134, in serve
await group.spawn(wait_for_catchup())
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/electrumx/electrumx/server/block_processor.py", line 702, in fetch_and_process_blocks
await group.spawn(self._process_prefetched_blocks())
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/electrumx/electrumx/server/block_processor.py", line 663, in _process_prefetched_blocks
await self.check_and_advance_blocks(blocks)
File "/electrumx/electrumx/server/block_processor.py", line 240, in check_and_advance_blocks
await self.reorg_chain()
File "/electrumx/electrumx/server/block_processor.py", line 282, in reorg_chain
await self.run_in_thread_with_lock(self.backup_blocks, raw_blocks)
File "/electrumx/electrumx/server/block_processor.py", line 212, in run_in_thread_with_lock
return await asyncio.shield(run_in_thread_locked())
File "/electrumx/electrumx/server/block_processor.py", line 211, in run_in_thread_locked
return await run_in_thread(func, *args)
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/curio.py", line 68, in run_in_thread
return await get_event_loop().run_in_executor(None, func, *args)
File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/electrumx/electrumx/server/block_processor.py", line 500, in backup_blocks
self.backup_txs(block.transactions, is_unspendable)
File "/electrumx/electrumx/server/block_processor.py", line 547, in backup_txs
assert n == 0
AssertionError
Has anyone encountered this problem
DB::flush_count overflow
My Electrumx crashes everytime during flushing the history with following error:
INFO:DB:flushing DB cache at 1,200 MB
INFO:History:history DB version: 1
INFO:History:flush count: 65,535
INFO:SessionManager:RPC server listening on localhost:8000
INFO:Prefetcher:catching up to daemon height 661,350 (6,282 blocks behind)
INFO:BlockProcessor:our height: 655,078 daemon: 661,350 UTXOs 17MB hist 17MB
INFO:BlockProcessor:processed 10 blocks size 13.27 MB in 351.0s
INFO:BlockProcessor:our height: 655,092 daemon: 661,350 UTXOs 35MB hist 38MB
INFO:BlockProcessor:processed 14 blocks size 18.18 MB in 290.1s
INFO:BlockProcessor:our height: 655,100 daemon: 661,350 UTXOs 45MB hist 51MB
INFO:BlockProcessor:processed 8 blocks size 11.27 MB in 116.5s
INFO:BlockProcessor:our height: 655,114 daemon: 661,350 UTXOs 65MB hist 74MB
INFO:BlockProcessor:processed 14 blocks size 19.03 MB in 161.2s
INFO:BlockProcessor:our height: 655,128 daemon: 661,350 UTXOs 81MB hist 96MB
INFO:BlockProcessor:processed 14 blocks size 18.05 MB in 147.0s
INFO:BlockProcessor:our height: 655,144 daemon: 661,351 UTXOs 104MB hist 124MB
INFO:BlockProcessor:processed 16 blocks size 20.90 MB in 176.4s
INFO:BlockProcessor:our height: 655,152 daemon: 661,351 UTXOs 114MB hist 137MB
INFO:BlockProcessor:processed 8 blocks size 10.56 MB in 58.5s
INFO:BlockProcessor:our height: 655,167 daemon: 661,352 UTXOs 129MB hist 160MB
INFO:BlockProcessor:processed 15 blocks size 19.63 MB in 108.0s
INFO:BlockProcessor:our height: 655,183 daemon: 661,352 UTXOs 143MB hist 179MB
INFO:BlockProcessor:processed 16 blocks size 19.63 MB in 120.6s
INFO:BlockProcessor:our height: 655,199 daemon: 661,352 UTXOs 157MB hist 197MB
INFO:BlockProcessor:processed 16 blocks size 19.64 MB in 136.0s
INFO:BlockProcessor:our height: 655,215 daemon: 661,353 UTXOs 169MB hist 216MB
INFO:BlockProcessor:processed 16 blocks size 20.10 MB in 108.7s
INFO:BlockProcessor:our height: 655,231 daemon: 661,353 UTXOs 182MB hist 234MB
INFO:BlockProcessor:processed 16 blocks size 19.42 MB in 94.6s
INFO:BlockProcessor:our height: 655,247 daemon: 661,353 UTXOs 201MB hist 257MB
INFO:DB:flushed filesystem data in 0.16s
INFO:Prefetcher:cancelled; prefetcher stopping
INFO:SessionManager:closing down server for rpc://localhost:8000
INFO:Controller:shutting down
INFO:Controller:shutdown complete
ERROR:electrumx:ElectrumX server terminated abnormally
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/e_x-1.16.0-py3.8.egg/EGG-INFO/scripts/electrumx_server", line 35, in main
File "/usr/lib/python3.8/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/lib/server_base.py", line 125, in run
await server_task
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/lib/server_base.py", line 98, in serve
await self.serve(shutdown_event)
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/controller.py", line 134, in serve
await group.spawn(wait_for_catchup())
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/block_processor.py", line 681, in fetch_and_process_blocks
await group.spawn(self._process_prefetched_blocks())
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/block_processor.py", line 642, in _process_prefetched_blocks
await self.check_and_advance_blocks(blocks)
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/block_processor.py", line 220, in check_and_advance_blocks
await self._maybe_flush()
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/block_processor.py", line 358, in _maybe_flush
await self.flush(flush_arg)
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/block_processor.py", line 348, in flush
await self.run_in_thread_with_lock(flush)
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/block_processor.py", line 202, in run_in_thread_with_lock
return await asyncio.shield(run_in_thread_locked())
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/block_processor.py", line 201, in run_in_thread_locked
return await run_in_thread(func, *args)
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/curio.py", line 68, in run_in_thread
return await get_event_loop().run_in_executor(None, func, *args)
File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/block_processor.py", line 346, in flush
self.db.flush_dbs(self.flush_data(), flush_utxos,
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/db.py", line 219, in flush_dbs
self.flush_history()
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/db.py", line 288, in flush_history
self.history.flush()
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/history.py", line 141, in flush
flush_id = pack_be_uint16(self.flush_count)
struct.error: 'H' format requires 0 <= number <= 65535
Print complete tor v3 onion addresses for rpc command peers
First of all: thanks for the support of Bitcoin and Electrum!
The command electrumx_rpc peers prints a maximum of 30 characters for the host item. However, onion v3 addresses are 56 characters without the ".onion" extension and thus 62 characters with the ".onion" extension. If a user wants to see how many onion addresses are connected to his electrumx server (for example via electrumx_rpc peers | grep onion | wc -l) only the version 2 tor addresses will contribute.
It would be nice if the peers rpc command shows the ".onion" extension for all tor addresses.
EDIT: I found that this can be achieved by changing the two occurrences of 30 in electrumx/lib/text.py into 62
electrumx fails to serve transaction data to electrum client
Hello,
I recently set up my own private electrumx server in Docker (using this Dockerfile). I intend to run electrumx to serve Bitcoin (original) blockchaindata only, thats why I switched from kyuupichan's github repository to this one.
Here is my setup (environment variables):
DAEMON_URL=http://[user]:[password]@bitcoind-node:8332/
COIN=Bitcoin
SERVICES=tcp://:50001,ssl://:50002,wss://:50004,rpc://
LOG_LEVEL=debug
PEER_DISCOVERY=self
I am using the latest official Electrum 4.0.6 Client for Windows 10, which works fine when connecting to other servers. However, when connecting to my own electrumx server only and after creating a new wallet and importing my private keys, the transaction history remains empty. The Electrum logfile has the following errors:
20201208T221439.173758Z | WARNING | interface.[127.0.0.1:50002] | disconnecting due to RPCError(2, "daemon error: DaemonError({'code': -5, 'message': 'No such mempool transaction. Use -txindex or provide a block hash to enable blockchain transaction queries. Use gettransaction for wallet transactions.'})")
20201208T221439.174771Z | DEBUG | interface.[127.0.0.1:50002] | (disconnect) trace for RPCError(2, "daemon error: DaemonError({'code': -5, 'message': 'No such mempool transaction. Use -txindex or provide a block hash to enable blockchain transaction queries. Use gettransaction for wallet transactions.'})")
Traceback (most recent call last):
File "electrum\interface.py", line 480, in wrapper_func
File "electrum\interface.py", line 504, in run
File "electrum\interface.py", line 647, in open_session
File "aiorpcx\curio.py", line 242, in __aexit__
File "aiorpcx\curio.py", line 211, in join
File "electrum\synchronizer.py", line 81, in _start_tasks
File "aiorpcx\curio.py", line 242, in __aexit__
File "aiorpcx\curio.py", line 211, in join
File "electrum\synchronizer.py", line 188, in _on_address_status
File "electrum\synchronizer.py", line 208, in _request_missing_txs
File "aiorpcx\curio.py", line 242, in __aexit__
File "aiorpcx\curio.py", line 211, in join
File "electrum\synchronizer.py", line 213, in _get_transaction
File "electrum\interface.py", line 914, in get_transaction
File "electrum\interface.py", line 172, in send_request
File "asyncio\tasks.py", line 414, in wait_for
File "aiorpcx\session.py", line 529, in send_request
File "aiorpcx\session.py", line 499, in _send_concurrent
aiorpcx.jsonrpc.RPCError: (2, "daemon error: DaemonError({'code': -5, 'message': 'No such mempool transaction. Use -txindex or provide a block hash to enable blockchain transaction queries. Use gettransaction for wallet transactions.'})")
My electrumx server is up and running, the blockchain appears to be fully synced with my bitcoind full node (currently 660546 Blocks) and all necessary ports are open.
Here is an excerpt from my electrumx log output:
electrumx_1 | INFO:Prefetcher:new block height 660,547 hash 0000000000000000000b6d28b329b449dab7f1d994eb2dee46efaf4031b1133c
electrumx_1 | DEBUG:MemPool:waiting for DB to sync
electrumx_1 | INFO:DB:flush #1,003 took 0.1s. Height 660,547 txs: 594,365,652 (+2,582)
electrumx_1 | INFO:BlockProcessor:processed 1 block size 1.35 MB in 0.7s
electrumx_1 | INFO:MemPool:16,161 txs 12.96 MB touching 95,507 addresses
electrumx_1 | INFO:MemPool:compact fee histogram: [(68.3, 100881), (58.8, 110962), (39.6, 123444), (29.9, 133492), (23.0, 147888), (21.9, 186221), (21.1, 255896), (21.0, 197979), (20.1, 146372), (20.0, 869830), (19.7, 287961), (17.3, 293590), (15.1, 334200), (12.3, 356225), (11.0, 393677), (10.0, 457659), (8.7, 485773), (7.5, 517204), (6.0, 737039), (5.0, 1059147), (4.0, 766306), (3.1, 531671), (3.0, 1867263), (2.0, 904623), (1.0, 1697338)]
electrumx_1 | INFO:MemPool:16,400 txs 13.06 MB touching 96,388 addresses
I have no idea how to continue from here. I've already tried deleting the whole database and indexing the whole blockchain again, with same results. Please give me some advice.
XSS attack with SVG
try to access you explorer with https://{explorer.com}/tx/121212<SvG OnLoad=alert('hello')>
BUG Restarting electrumx will cause a stuck
I installed the electrumx of the latest master branch. Thank you very much for the memory leak problem that has been solved after testing, but I now encounter a new problem. I try to kill the electrumx service. When I start it again, it will get stuck in a position and cannot continue. , And it will be stuck to the position of the log below after restarting many times
Oct 28 00:52:48 ubuntu-PowerEdge-R740 electrumx_server[41195]: INFO:DB:coin: Bitcoin
Oct 28 00:52:48 ubuntu-PowerEdge-R740 electrumx_server[41195]: INFO:DB:network: mainnet
Oct 28 00:52:48 ubuntu-PowerEdge-R740 electrumx_server[41195]: INFO:DB:height: 654,158
Oct 28 00:52:48 ubuntu-PowerEdge-R740 electrumx_server[41195]: INFO:DB:tip: 0000000000000000000cec4112e9b0baa1f22f5ae1771cc55b42d63f57737d02
Oct 28 00:52:48 ubuntu-PowerEdge-R740 electrumx_server[41195]: INFO:DB:tx count: 580,836,867
Oct 28 00:52:48 ubuntu-PowerEdge-R740 electrumx_server[41195]: INFO:DB:flushing DB cache at 1,200 MB
Oct 28 00:52:48 ubuntu-PowerEdge-R740 electrumx_server[41195]: INFO:History:history DB version: 1
Oct 28 00:52:48 ubuntu-PowerEdge-R740 electrumx_server[41195]: INFO:History:flush count: 13,156
Oct 28 00:52:49 ubuntu-PowerEdge-R740 electrumx_server[41195]: INFO:Prefetcher:catching up to daemon height 654,467 (309 blocks behind)
Oct 28 00:52:49 ubuntu-PowerEdge-R740 electrumx_server[41195]: INFO:SessionManager:RPC server listening on localhost:8000
electrumx to get all wallet addresses?
Is there a way for electrumx to get all wallet addresses? I try to do btc address distribution, try to get all addresses and cache them in my own redis, is there any good idea to learn?
how to check electrumx server status and fingerprint etc?
I have a server up and running now. I wonder if there are any online tools or commnadline tools that I can use to check its status (e.g., version number, banner, etc)
Thanks!
Not able to withdraw my BTC,showing error
I have BTC in hardware and I have tried it withdrawing electrum wallet showing error and locked for 24 hrs and everytime when I m trying to withdraw it locked.
Please help me
ElectrumX bitcoin support ? Other altcoins ?
Hi,
I am a little confused and wondered if anybody can confirm something?
I found this repo stating it was a fork of the original electrumx which seems to have dropped support for bitcoin (according to this readme) - although checking their repo it states they are dropping altcoin support and only bitcoin sv will be supported. So bitcoin (BTC) is still supported ?
Does this repo support BTC and does it support any altcoins ? I was looking specifically for litecoin.
Any help really appreciated.
Thanks
How to cache data to memory
Due to the previous electrumx memory leak, I added the server to 128G memory. Now the memory problem is solved, but the remaining memory is close to 110G. How can I cache all the rocksdb data in the memory to improve the query speed, so that no ssd hard disk is needed.
ElectrumX crash of two instances simultaneously
Version : 1.15.0
I have two instances of electrumX running and both crashed at the same time with same log.
crash log: https://paste.ubuntu.com/p/8fqJh93d95/
DAEMON_URL definition unclear
Is DAEMON_URL supposed to be json-rpc address or bitcoin-protocol address?
electrumx crashed with general protection fault ip:50fcc4 sp:7ffcc2465108 error:0 in python3.7[421000+232000]
Electrumx has been running for 2d 11h and crashed. Dmesg:
[Tue Sep 15 08:43:16 2020] traps: electrumx_serve[3171321] general protection fault ip:50fcc4 sp:7ffcc2465108 error:0 in python3.7[421000+232000]
Electrumx log has nothing interesting:
2020-09-15 08:43:12,097:INFO:ElectrumX:[2771418] disconnected whilst throttled 2020-09-15 08:43:12,099:INFO:ElectrumX:[2771413] disconnected whilst throttled 2020-09-15 08:43:12,099:INFO:ElectrumX:[2771415] disconnected whilst throttled 2020-09-15 08:43:12,107:INFO:ElectrumX:[2771414] disconnected whilst throttled 2020-09-15 08:43:12,167:INFO:ElectrumX:[2771412] disconnected whilst throttled 2020-09-15 08:43:12,175:INFO:ElectrumX:[2771425] SSL 14.185.171.91:54000, 283 total
System:
OS: Debian GNU/Linux 10 (buster) x86_64
Kernel: 5.7.0-0.bpo.2-amd64
Electrum straight from git, latest commit:
commit 9ca59facfdb50daf063a5813fc9b08eb6fa1b246 (HEAD -> master, origin/master, origin/HEAD)
Merge: 148df48 23633c2
Author: ghost43 <[email protected]>
Date: Mon Aug 24 17:30:54 2020 +0000
Log excessively filled with entries such as "TCP xx.xx.xx.xx:xx, 20 total"
Hello,
I'm running Debian Buster, with ElectrumX 1.15. All python packages have been updated. I've had this issue for the past few versions of ElectrumX.
At the moment, my instance will fill log files with this sort of message:
: INFO:ElectrumX:[12] TCP xx.xx.xx.xx:xx, 5 total
: INFO:ElectrumX:[13] TCP xx.xx.xx.xx:xx, 5 total
: INFO:ElectrumX:[14] TCP xx.xx.xx.xx:xx, 6 total
: INFO:PeerManager:[ **removed**:50002 SSL] verified in 0.9s
: INFO:ElectrumX:[15] TCP xx.xx.xx.xx:xx, 7 total
: INFO:ElectrumX:[15] attempting to crash old client with version 3.2.2
: INFO:ElectrumX:[16] TCP xx.xx.xx.xx:xx, 6 total
: INFO:ElectrumX:[17] SSL xx.xx.xx.xx:xx, 7 total
: INFO:ElectrumX:[18] TCP xx.xx.xx.xx:xx, 8 total
: INFO:ElectrumX:[19] TCP xx.xx.xx.xx:xx, 9 total
: INFO:ElectrumX:[20] TCP xx.xx.xx.xx:xx, 9 total
: INFO:ElectrumX:[21] TCP xx.xx.xx.xx:xx, 6 total
: INFO:ElectrumX:[22] TCP xx.xx.xx.xx:xx, 7 total
: INFO:ElectrumX:[23] TCP xx.xx.xx.xx:xx, 7 total
: INFO:ElectrumX:[23] attempting to crash old client with version 3.2.2
: INFO:ElectrumX:[24] SSL xx.xx.xx.xx:xx, 6 total
: INFO:ElectrumX:[25] TCP xx.xx.xx.xx:xx, 7 total
: INFO:ElectrumX:[26] TCP xx.xx.xx.xx:xx, 8 total
: INFO:ElectrumX:[27] TCP xx.xx.xx.xx:xx, 8 total
: INFO:ElectrumX:[28] TCP xx.xx.xx.xx:xx, 9 total
: INFO:ElectrumX:[29] TCP xx.xx.xx.xx:xx, 8 total
: INFO:ElectrumX:[30] TCP xx.xx.xx.xx:xx, 9 total
: INFO:ElectrumX:[31] TCP xx.xx.xx.xx:xx, 10 total
The xx.xx.xx.xx:xx would be replaced by an IP/port if ANON_LOGS was not set. For now, I disable the NOTICE level in logging as to not fill my syslog.
What exactly causes this message and how can I prevent it from spamming my log?
Thank you
Server crash: AttributeError: 'NoneType' object has no attribute 'fail'
After several days of operation, the server crashed. I was running master on commit 1061610. Looking at the logs, might also be a problem in aiorpcX.
Log output follows:
INFO:PeerManager:[ns3079944.ip-217-182-196.eu:50002 SSL] [Errno 104] Connection reset by peer
INFO:PeerManager:[ns3079944.ip-217-182-196.eu:50002 SSL] [Errno -5] No address associated with hostname
Exception in callback RSTransport.connection_made(<_SelectorSoc...e, bufsize=0>>)
handle: <Handle RSTransport.connection_made(<_SelectorSoc...e, bufsize=0>>)>
Traceback (most recent call last):
File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
self._context.run(self._callback, *self._args)
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/rawsocket.py", line 80, in connection_made
self._remote_address = NetAddress(peername[0], peername[1])
TypeError: 'NoneType' object is not subscriptable
Exception in callback _SelectorSocketTransport._call_connection_lost(ConnectionRes...eset by peer'))
handle: <Handle _SelectorSocketTransport._call_connection_lost(ConnectionRes...eset by peer'))>
Traceback (most recent call last):
File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
self._context.run(self._callback, *self._args)
File "/usr/lib/python3.8/asyncio/selector_events.py", line 968, in _call_connection_lost
super()._call_connection_lost(exc)
File "/usr/lib/python3.8/asyncio/selector_events.py", line 726, in _call_connection_lost
self._protocol.connection_lost(exc)
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/rawsocket.py", line 94, in connection_lost
self._framer.fail(ConnectionLostError())
AttributeError: 'NoneType' object has no attribute 'fail'
...
ERROR:electrumx:ElectrumX server terminated abnormally
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/EGG-INFO/scripts/electrumx_server", line 35, in main
File "/usr/lib/python3.8/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/lib/server_base.py", line 125, in run
await server_task
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/lib/server_base.py", line 98, in serve
await self.serve(shutdown_event)
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/controller.py", line 134, in serve
await group.spawn(wait_for_catchup())
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/mempool.py", line 335, in keep_synchronized
await group.spawn(self._logging(synchronized_event))
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/mempool.py", line 232, in _refresh_hashes
await self.api.on_mempool(touched, height)
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/controller.py", line 67, in on_mempool
await self._maybe_notify()
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/controller.py", line 55, in _maybe_notify
await self.notify(height, touched)
File "/usr/local/lib/python3.8/dist-packages/electrumX-1.15.0-py3.8.egg/electrumx/server/session.py", line 766, in _notify_sessions
await self._task_group.spawn(session.notify, touched, height_changed)
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/curio.py", line 148, in spawn
self._add_task(task)
File "/usr/local/lib/python3.8/dist-packages/aiorpcX-0.18.4-py3.8.egg/aiorpcx/curio.py", line 123, in _add_task
raise RuntimeError('task group is closed')
RuntimeError: task group is closed
Time-based groups?
Like everyone, my server is under constant DDoS. Currently ElectrumX has a simple cost-based mitigation, that works for individual sessions but also groups sessions together based on their /24 network address and also a time-window when the session was created. The network-based grouping kind of works, but I don't get the time-based bucketing.
If the server is under constant load, the time-based groups will always be over budget and therefore all sessions will get throttled, regardless whether they are legit or not. Sooner or later, they will disconnect. Am I missing something, or is this type of protection contra-productive? Is anyone using this in production anyway?
If no-one is using it, shouldn't it be removed?
electrumx crash when bitcoind is restarted + log spamming on INFO and ERROR levels
Today my bitcoind crashed (and restarted automatically, I have script for it), however, electrumx crashed too, shortly after, it didn't handled reconnecting to bitcoind successfully.
log:
electrumx.log
Problems with large transactions and problems with a lot of address history
Hello there.
This bitcoin-tx
0048c164bf6da763bcae4983d570af709d96310987c6ae3cbde4f3ecb81a0e57
with many outputs delivers the error:
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 99998 (char 99997)
command: get_from_electrum('blockchain.transaction.get',[txhash,True])
Another issue:
An very busy address with many transactions delivers the error:
#{'jsonrpc': '2.0', 'error': {'code': 1, 'message': 'history too large'}, 'id': 0}
command: get_from_electrum('blockchain.scripthash.get_history', scripthash)
any ideas?
thank you.
AttributeError: 'TxOutput' object has no attribute 'script'
BCH electrums stopped working 30 mins ago and restarting shows this error on all 3:
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:electrumx:ElectrumX server starting
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:electrumx:logging level: INFO
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:Controller:Python version: 3.7.3 (default, Dec 20 2019, 18:57:59) [GCC 8.3.0]
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:Controller:software version: ElectrumX 1.15.0
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:Controller:aiorpcX version: 0.18.4
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:Controller:supported protocol versions: 1.4-1.4.2
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:Controller:event loop policy: <uvloop.EventLoopPolicy object at 0x7fa7aee4b400>
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:Controller:reorg limit is 200 blocks
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:Daemon:daemon #1 at 127.0.0.1:33333/ (current)
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:DB:switching current directory to /electrumdb/BCH
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:DB:using leveldb for DB backend
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:DB:opened UTXO DB (for sync: True)
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:DB:UTXO DB version: 8
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:DB:coin: BitcoinCashABC
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:DB:network: mainnet
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:DB:height: 645,029
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:DB:tip: 000000000000000000527de23648977f88fe21ebe6fb97eb8f9cfc3cf1e336e1
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:DB:tx count: 290,344,947
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:DB:flushing DB cache at 1,200 MB
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:History:history DB version: 1
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:History:flush count: 3,217
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:DB:deleted 2,307 stale undo entries
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:SessionManager:RPC server listening on localhost:8055
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:Prefetcher:catching up to daemon height 645,040 (11 blocks behind)
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:LTORBlockProcessor:chain reorg detected
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:LTORBlockProcessor:chain was reorganised replacing 1 block at heights 645,029-645,029
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:LTORBlockProcessor:read 1 blocks from disk
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:Prefetcher:cancelled; prefetcher stopping
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:SessionManager:closing down server for rpc://localhost:8055
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:Controller:shutting down
Jul 22 14:38:43 electrum3 electrumx_server[286753]: INFO:Controller:shutdown complete
Jul 22 14:38:43 electrum3 electrumx_server[286753]: ERROR:electrumx:ElectrumX server terminated abnormally
Jul 22 14:38:43 electrum3 electrumx_server[286753]: Traceback (most recent call last):
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/local/bin/electrumx_server", line 35, in main
Jul 22 14:38:43 electrum3 electrumx_server[286753]: asyncio.run(controller.run())
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
Jul 22 14:38:43 electrum3 electrumx_server[286753]: return loop.run_until_complete(main)
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/local/lib/python3.7/dist-packages/electrumx/lib/server_base.py", line 125, in run
Jul 22 14:38:43 electrum3 electrumx_server[286753]: await server_task
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/local/lib/python3.7/dist-packages/electrumx/lib/server_base.py", line 98, in serve
Jul 22 14:38:43 electrum3 electrumx_server[286753]: await self.serve(shutdown_event)
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/local/lib/python3.7/dist-packages/electrumx/server/controller.py", line 134, in serve
Jul 22 14:38:43 electrum3 electrumx_server[286753]: await group.spawn(wait_for_catchup())
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/local/lib/python3.7/dist-packages/aiorpcx/curio.py", line 242, in __aexit__
Jul 22 14:38:43 electrum3 electrumx_server[286753]: await self.join()
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/local/lib/python3.7/dist-packages/aiorpcx/curio.py", line 211, in join
Jul 22 14:38:43 electrum3 electrumx_server[286753]: raise task.exception()
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/local/lib/python3.7/dist-packages/electrumx/server/block_processor.py", line 681, in fetch_and_process_blocks
Jul 22 14:38:43 electrum3 electrumx_server[286753]: await group.spawn(self._process_prefetched_blocks())
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/local/lib/python3.7/dist-packages/aiorpcx/curio.py", line 242, in __aexit__
Jul 22 14:38:43 electrum3 electrumx_server[286753]: await self.join()
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/local/lib/python3.7/dist-packages/aiorpcx/curio.py", line 211, in join
Jul 22 14:38:43 electrum3 electrumx_server[286753]: raise task.exception()
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/local/lib/python3.7/dist-packages/electrumx/server/block_processor.py", line 642, in _process_prefetched_blocks
Jul 22 14:38:43 electrum3 electrumx_server[286753]: await self.check_and_advance_blocks(blocks)
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/local/lib/python3.7/dist-packages/electrumx/server/block_processor.py", line 230, in check_and_advance_blocks
Jul 22 14:38:43 electrum3 electrumx_server[286753]: await self.reorg_chain()
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/local/lib/python3.7/dist-packages/electrumx/server/block_processor.py", line 272, in reorg_chain
Jul 22 14:38:43 electrum3 electrumx_server[286753]: await self.run_in_thread_with_lock(self.backup_blocks, raw_blocks)
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/local/lib/python3.7/dist-packages/electrumx/server/block_processor.py", line 202, in run_in_thread_with_lock
Jul 22 14:38:43 electrum3 electrumx_server[286753]: return await asyncio.shield(run_in_thread_locked())
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/local/lib/python3.7/dist-packages/electrumx/server/block_processor.py", line 201, in run_in_thread_locked
Jul 22 14:38:43 electrum3 electrumx_server[286753]: return await run_in_thread(func, *args)
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/local/lib/python3.7/dist-packages/aiorpcx/curio.py", line 68, in run_in_thread
Jul 22 14:38:43 electrum3 electrumx_server[286753]: return await get_event_loop().run_in_executor(None, func, *args)
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
Jul 22 14:38:43 electrum3 electrumx_server[286753]: result = self.fn(*self.args, **self.kwargs)
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/local/lib/python3.7/dist-packages/electrumx/server/block_processor.py", line 484, in backup_blocks
Jul 22 14:38:43 electrum3 electrumx_server[286753]: self.backup_txs(block.transactions, is_unspendable)
Jul 22 14:38:43 electrum3 electrumx_server[286753]: File "/usr/local/lib/python3.7/dist-packages/electrumx/server/block_processor.py", line 835, in backup_txs
Jul 22 14:38:43 electrum3 electrumx_server[286753]: hashX = script_hashX(txout.script)
Jul 22 14:38:43 electrum3 electrumx_server[286753]: AttributeError: 'TxOutput' object has no attribute 'script'
Problem with parameter id:0
First of all thank you for your help, I have a little doubt, here is my example：
import socket
import json
from time import sleep
import sys,os
ABSPATH = os.path.abspath(os.path.realpath(os.path.dirname(__file__)))
ABSPATH = ABSPATH
sys.path.append(ABSPATH)
from bitcoin import *
port = 50001
host = '127.0.0.1'
hash = address_to_scripthash("386eAUqL3ZNZPmHeABXLo658DTQuJeLzUR")
content = {
"method": "blockchain.scripthash.get_balance",
"params": [hash],
"id": 0
}
def electrumx(host, port, content):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((host, port))
sock.sendall(json.dumps(content).encode('utf-8')+b'\n')
sleep(0.5)
sock.shutdown(socket.SHUT_WR)
res = ""
while True:
data = sock.recv(1024)
if (not data):
break
res += data.decode()
print(res)
sock.close()
electrumx(host, port, content)
What does id:0 mean in the following code?
content = {
"method": "blockchain.scripthash.get_balance",
"params": [hash],
"id": 0
}
Why don’t I return the result when I delete id:0, what is the meaning of it?
python websocket electrumx
I tried to use ws to get data, but I couldn't get any response from any server. I have solved this problem for several days, and I have not been able to solve it. I hope to get your help, thank you very much,Below is my code
import websockets
from jsonrpcclient.clients.websockets_client import WebSocketsClient
async def electrumx(self):
async with websockets.connect('ws://10.0.100.10:50003') as ws:
res = await WebSocketsClient(ws).request('blockchain.address.get_history', ['3D2oetdNuZUqQHPJmcMDDHYoqkyNVsFk9r'])
print(res.data)
return res.data
No results can be obtained by executing print, and I see the following prompt disconnected whilst throttled in the electrumx log
If I execute the following code method is correct, can return normal data
async def electrumx(self):
async with websockets.connect('ws://10.0.100.10:50003') as ws:
res = await WebSocketsClient(ws).request('server.features')
print(res.data)
return res.data
So where am I going wrong? , Please help me, thank you very much
how to force using Tor to broadcast new transactions?
I have set up bitcoin core with the following options:
txindex=1
server=1
disablewallet=1
walletbroadcast=0
I have set up electrumx as follows:
export DB_DIRECTORY=$HOME/.electrumx/db
export DAEMON_URL="http://$(cat "$HOME/.bitcoin/.cookie")@localhost:8332"
export COIN=Bitcoin
export SERVICES=tcp://:50001,ssl://:50002
export SSL_CERTFILE=/etc/ssl/certs/ssl-cert-snakeoil.pem
export SSL_KEYFILE=/etc/ssl/private/ssl-cert-snakeoil.key
electrumx_server
I have an electrum wallet that only connects to my own electrumx server. When I create a transaction, apparently the transaction is broadcasted out and carried out successfully. What is the route through which transaction gets broadcasted? Specifically is my electrumx server braodcasting the transaction to its peers? Or is my bitcoin core daemon sending to its peers? Is there a way force the broadcast of our own transactions going through Tor only, while keeping other transaction relay using the usual route?
Thanks.
Put electrumx db into tmpfs memory disk
My server has a very large memory. I put all the data of electrumx into the memory for quick connection query. Will electrumx and bitcoin generate query events during my query transaction? During my large number of queries to electrumx, bitcoin was reading a lot of disk data, which could not reach the rocket speed.
Bitcoin Cash electrumx-1.15.0 cant start, seems like reorg issue
Logs
INFO:DB:tip: 000000000000000000527de23648977f88fe21ebe6fb97eb8f9cfc3cf1e336e1
INFO:DB:tx count: 290,344,947
INFO:DB:flushing DB cache at 1,200 MB
INFO:History:history DB version: 0
INFO:History:flush count: 55,767
INFO:SessionManager:RPC server listening on 0.0.0.0:8000
INFO:Prefetcher:catching up to daemon height 645,042 (13 blocks behind)
INFO:LTORBlockProcessor:chain reorg detected
INFO:LTORBlockProcessor:chain was reorganised replacing 1 block at heights 645,029-645,029
INFO:LTORBlockProcessor:read 1 blocks from disk
INFO:Prefetcher:cancelled; prefetcher stopping
INFO:LTORBlockProcessor:flushing to DB for a clean shutdown...
INFO:SessionManager:closing down server for rpc://0.0.0.0:8000
INFO:Controller:shutting down
INFO:Controller:shutdown complete
ERROR:electrumx:ElectrumX server terminated abnormally
Traceback (most recent call last):
File "/home/electrumx/.local/bin/electrumx_server", line 35, in main
asyncio.run(controller.run())
File "/usr/local/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
return future.result()
File "/home/electrumx/.local/lib/python3.7/site-packages/electrumx/lib/server_base.py", line 125, in run
await server_task
File "/home/electrumx/.local/lib/python3.7/site-packages/electrumx/lib/server_base.py", line 98, in serve
await self.serve(shutdown_event)
File "/home/electrumx/.local/lib/python3.7/site-packages/electrumx/server/controller.py", line 134, in serve
await group.spawn(wait_for_catchup())
File "/home/electrumx/.local/lib/python3.7/site-packages/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/home/electrumx/.local/lib/python3.7/site-packages/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/home/electrumx/.local/lib/python3.7/site-packages/electrumx/server/block_processor.py", line 681, in fetch_and_process_blocks
await self.flush(True)
File "/home/electrumx/.local/lib/python3.7/site-packages/electrumx/server/block_processor.py", line 343, in flush
await self.run_in_thread_with_lock(flush)
File "/home/electrumx/.local/lib/python3.7/site-packages/electrumx/server/block_processor.py", line 199, in run_in_thread_with_lock
return await asyncio.shield(run_in_thread_locked())
File "/home/electrumx/.local/lib/python3.7/site-packages/electrumx/server/block_processor.py", line 198, in run_in_thread_locked
return await run_in_thread(func, *args)
File "/home/electrumx/.local/lib/python3.7/site-packages/aiorpcx/curio.py", line 68, in run_in_thread
return await get_event_loop().run_in_executor(None, func, *args)
File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/electrumx/.local/lib/python3.7/site-packages/electrumx/server/block_processor.py", line 342, in flush
self.estimate_txs_remaining)
File "/home/electrumx/.local/lib/python3.7/site-packages/electrumx/server/db.py", line 202, in flush_dbs
self.assert_flushed(flush_data)
File "/home/electrumx/.local/lib/python3.7/site-packages/electrumx/server/db.py", line 190, in assert_flushed
assert flush_data.tip == self.db_tip
AssertionError
Same issue: kyuupichan/electrumx#1017
PeerManager._monitor_peer can get cancelled if remote peer closes transport unluckily
As in title, if a server peer closes the transport/session with us while we are testing if it is a valid peer, we might never re-test that server again, as PeerManager._monitor_peer can get cancelled.
PeerManager._monitor_peer is the task responsible to, well, monitor a server peer:
electrumx/electrumx/server/peers.py
Line 231 in c070573
periodically it calls
_should_drop_peer:electrumx/electrumx/server/peers.py
Line 234 in c070573
which tries to establish a session with the remote server peer and run some sanity check RPCs on it:
electrumx/electrumx/server/peers.py
Lines 274 to 279 in c070573
Consider the following scenario:
connect_rs succeeds establishing a session.
_verify_peer is called and entered.
and then near the end of _verify_peer this line runs:
electrumx/electrumx/server/peers.py
Line 383 in c070573
we have sent the request and are waiting for the response.
Meanwhile, the remote peer closes the transport, and connection_lost gets called:
https://github.com/kyuupichan/aiorpcX/blob/bd2185aaeb7cd8b18e24866c4a04387650ff4834/aiorpcx/session.py#L509-L512
This will result in cancelling all pending requests; in practice:
await session.send_request will raise an asyncio.CancelledError
electrumx/electrumx/server/peers.py
Line 383 in c070573
This CancelledError will then propagate out fully to
_monitor_peer and even get that cancelled.(it stops propagating further due to the enclosing TaskGroup).
As _monitor_peer is now cancelled, (and the peer remains in our peer list), this peer will now never be retried again.
Weird fee estimation
Im constantly comparing result of estimatefee with https://mempool.space and electrumx
always provides way higher values. And occasionally I use Electrum-desktop - I think I noticed if offers incorrect (too high) fee as well.
bitcoind ver: 0.20.1
electrumx ver: 1.15.0 (from this repo)
Default LOG_LEVEL=info equals to 3M lines of output per day
Hi all,
In default setting, ElectrumX is spamming messages like this to INTO output level:
INFO:ElectrumX:[1195011] changing task concurrency from 10 to 8
INFO:ElectrumX:[1195011] SSL 186.3.207.118:52858, 678 total
Yes my friends, this is line number 1,195,011. 1.2M output messages like this and 3M of lines (counted by cat -n command) in total (including other messages, which are counted separately, as I understand). This is on default INFO level. When I switch to WARNING level (one level up), I see nothing, no informative messages about peers, network, configuration, nothing, just errors when peer is blacklisted or something like that. I want to see relevant information, like peer summary (configured via LOG_SESSIONS variable). But when I go back to LOG_LEVEL=info, I will have 3M of messages per day, which amounted to 250MB on the disk. That's like 34 messages per second!
Can we please move messages like:
INFO:ElectrumX:[1195011] changing task concurrency from 10 to 8
INFO:ElectrumX:[1195011] SSL 186.3.207.118:52858, 678 total
to DEBUG level, instead INFO level? This is certainly DEBUG message and I should not see it during normal operation. Not a million of them per day, at least. Thanks!
TypeError: '<' not supported between instances of 'UTXO' and 'UTXO'
On master:
2020-06-19T17:35:01.902288380Z ERROR:ElectrumX:[110] exception handling Request('blockchain.scripthash.listunspent', ['bf5cba1ab27952ef2c119cab412adf0aafaa74ba395e6d974fab21b7a2961593'])
2020-06-19T17:35:01.902308915Z Traceback (most recent call last):
2020-06-19T17:35:01.902312317Z File "/usr/local/lib/python3.8/site-packages/aiorpcx/session.py", line 467, in _throttled_request
2020-06-19T17:35:01.902315232Z result = await self.handle_request(request)
2020-06-19T17:35:01.902318482Z File "/usr/local/lib/python3.8/site-packages/electrumx/server/session.py", line 889, in handle_request
2020-06-19T17:35:01.902321465Z return await coro
2020-06-19T17:35:01.902323689Z File "/usr/local/lib/python3.8/site-packages/electrumx/server/session.py", line 1119, in scripthash_listunspent
2020-06-19T17:35:01.902326238Z return await self.hashX_listunspent(hashX)
2020-06-19T17:35:01.902328657Z File "/usr/local/lib/python3.8/site-packages/electrumx/server/session.py", line 1059, in hashX_listunspent
2020-06-19T17:35:01.902330992Z utxos = sorted(utxos)
2020-06-19T17:35:01.902333032Z TypeError: '<' not supported between instances of 'UTXO' and 'UTXO'
ElectrumX server keeps crashing randomly
From version 1.15.0 ElectrumX server keeps crashing randomly (sometimes daily). If you need more info let me know.
Debian 10.4
Python 3.7.3
Jul 13 05:52:49 server1 electrumx_server[10865]: INFO:Prefetcher:cancelled; prefetcher stopping
Jul 13 05:52:49 server1 electrumx_server[10865]: INFO:SessionManager:closing down server for rpc://127.0.0.1:8000
Jul 13 05:52:49 server1 electrumx_server[10865]: INFO:SessionManager:closing down server for ssl://all_interfaces:50002
Jul 13 05:52:49 server1 electrumx_server[10865]: INFO:ElectrumX:[156216] disconnected whilst throttled
Jul 13 05:52:50 server1 electrumx_server[10865]: INFO:Controller:shutting down
Jul 13 05:52:50 server1 electrumx_server[10865]: INFO:Controller:shutdown complete
Jul 13 05:52:50 server1 electrumx_server[10865]: ERROR:electrumx:ElectrumX server terminated abnormally
Jul 13 05:52:50 server1 electrumx_server[10865]: Traceback (most recent call last):
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/home/electrumx/electrumx_server", line 35, in main
Jul 13 05:52:50 server1 electrumx_server[10865]: asyncio.run(controller.run())
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
Jul 13 05:52:50 server1 electrumx_server[10865]: return loop.run_until_complete(main)
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
Jul 13 05:52:50 server1 electrumx_server[10865]: return future.result()
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/home/electrumx/electrumx/lib/server_base.py", line 125, in run
Jul 13 05:52:50 server1 electrumx_server[10865]: await server_task
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/home/electrumx/electrumx/lib/server_base.py", line 98, in serve
Jul 13 05:52:50 server1 electrumx_server[10865]: await self.serve(shutdown_event)
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/home/electrumx/electrumx/server/controller.py", line 134, in serve
Jul 13 05:52:50 server1 electrumx_server[10865]: await group.spawn(wait_for_catchup())
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/usr/local/lib/python3.7/dist-packages/aiorpcx/curio.py", line 242, in __aexit__
Jul 13 05:52:50 server1 electrumx_server[10865]: await self.join()
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/usr/local/lib/python3.7/dist-packages/aiorpcx/curio.py", line 211, in join
Jul 13 05:52:50 server1 electrumx_server[10865]: raise task.exception()
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/home/electrumx/electrumx/server/block_processor.py", line 681, in fetch_and_process_blocks
Jul 13 05:52:50 server1 electrumx_server[10865]: await group.spawn(self._process_prefetched_blocks())
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/usr/local/lib/python3.7/dist-packages/aiorpcx/curio.py", line 242, in __aexit__
Jul 13 05:52:50 server1 electrumx_server[10865]: await self.join()
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/usr/local/lib/python3.7/dist-packages/aiorpcx/curio.py", line 211, in join
Jul 13 05:52:50 server1 electrumx_server[10865]: raise task.exception()
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/home/electrumx/electrumx/server/block_processor.py", line 642, in _process_prefetched_blocks
Jul 13 05:52:50 server1 electrumx_server[10865]: await self.check_and_advance_blocks(blocks)
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/home/electrumx/electrumx/server/block_processor.py", line 220, in check_and_advance_blocks
Jul 13 05:52:50 server1 electrumx_server[10865]: await self._maybe_flush()
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/home/electrumx/electrumx/server/block_processor.py", line 354, in _maybe_flush
Jul 13 05:52:50 server1 electrumx_server[10865]: await self.flush(True)
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/home/electrumx/electrumx/server/block_processor.py", line 348, in flush
Jul 13 05:52:50 server1 electrumx_server[10865]: await self.run_in_thread_with_lock(flush)
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/home/electrumx/electrumx/server/block_processor.py", line 202, in run_in_thread_with_lock
Jul 13 05:52:50 server1 electrumx_server[10865]: return await asyncio.shield(run_in_thread_locked())
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/home/electrumx/electrumx/server/block_processor.py", line 201, in run_in_thread_locked
Jul 13 05:52:50 server1 electrumx_server[10865]: return await run_in_thread(func, *args)
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/usr/local/lib/python3.7/dist-packages/aiorpcx/curio.py", line 68, in run_in_thread
Jul 13 05:52:50 server1 electrumx_server[10865]: return await get_event_loop().run_in_executor(None, func, *args)
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
Jul 13 05:52:50 server1 electrumx_server[10865]: result = self.fn(*self.args, **self.kwargs)
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/home/electrumx/electrumx/server/block_processor.py", line 347, in flush
Jul 13 05:52:50 server1 electrumx_server[10865]: self.estimate_txs_remaining)
Jul 13 05:52:50 server1 electrumx_server[10865]: File "/home/electrumx/electrumx/server/db.py", line 225, in flush_dbs
Jul 13 05:52:50 server1 electrumx_server[10865]: self.flush_state(batch)
Jul 13 05:52:50 server1 electrumx_server[10865]: File "plyvel/_plyvel.pyx", line 634, in plyvel._plyvel.WriteBatch.__exit__
Jul 13 05:52:50 server1 electrumx_server[10865]: File "plyvel/_plyvel.pyx", line 617, in plyvel._plyvel.WriteBatch.write
Jul 13 05:52:50 server1 electrumx_server[10865]: File "plyvel/_plyvel.pyx", line 91, in plyvel._plyvel.raise_for_status
Jul 13 05:52:50 server1 electrumx_server[10865]: plyvel._plyvel.CorruptionError: b'Corruption: corrupted compressed block contents'
Jul 13 05:52:51 server1 systemd[1]: electrumx.service: Succeeded.
Get scripthash for known scriptPubKey in python？
have to query addresses in ElectrumX server. How do I generate scripthash for a known scriptPubKey in python? If my bitcoin wallet address is 1P5ZEDWTKTFGxQjZphgWPQUpe554WKDfHQ
ElectrumX dogecoin doesn't work
Server crashes after synchronization with "Killed" error.
ElectrumX log:
INFO:electrumx:ElectrumX server starting
WARNING:Env:lowered maximum sessions from 1,000 to 674 because your open file limit is 1,024
INFO:electrumx:logging level: INFO
INFO:Controller:Python version: 3.9.1 (default, Feb 1 2021, 09:08:06) [GCC 7.5.0]
INFO:Controller:software version: ElectrumX 1.16.0
INFO:Controller:aiorpcX version: 0.18.5
INFO:Controller:supported protocol versions: 1.4-1.4.2
INFO:Controller:event loop policy: None
INFO:Controller:reorg limit is 2,000 blocks
INFO:Daemon:daemon #1 at 127.0.0.1:22555/ (current)
INFO:DB:switching current directory to /home/ubuntu/db
INFO:DB:using leveldb for DB backend
INFO:DB:opened UTXO DB (for sync: True)
INFO:DB:UTXO DB version: 8
INFO:DB:coin: Dogecoin
INFO:DB:network: mainnet
INFO:DB:height: 3,592,619
INFO:DB:tip: d69cd535b31679cd5c505ca4ad40e5236a67a619844294af9279261bf3939e72
INFO:DB:tx count: 70,575,752
INFO:DB:flushing DB cache at 500 MB
INFO:History:history DB version: 1
INFO:History:flush count: 156
INFO:Prefetcher:catching up to daemon height 3,593,544 (925 blocks behind)
INFO:BlockProcessor:our height: 3,592,629 daemon: 3,593,544 UTXOs 0MB hist 0MB
INFO:BlockProcessor:processed 10 blocks size 0.22 MB in 2.4s
INFO:BlockProcessor:processed 700 blocks size 11.12 MB in 9.3s
INFO:BlockProcessor:processed 215 blocks size 4.27 MB in 1.5s
INFO:BlockProcessor:caught up to height 3593544
INFO:DB:flushed filesystem data in 0.01s
INFO:History:flushed history in 0.2s for 61,712 addrs
INFO:DB:flushed 925 blocks with 27,817 txs, 43,144 UTXO adds, 45,925 spends in 0.2s, committing...
INFO:DB:flush #157 took 0.5s. Height 3,593,544 txs: 70,603,569 (+27,817)
INFO:DB:tx/sec since genesis: 6,957, since last flush: 1,845
INFO:DB:sync time: 02h 49m 07s ETA: 00s
INFO:DB:closing DBs to re-open for serving
INFO:DB:opened UTXO DB (for sync: False)
INFO:DB:UTXO DB version: 8
INFO:DB:coin: Dogecoin
INFO:DB:network: mainnet
INFO:DB:height: 3,593,544
INFO:DB:tip: 5f5beb8d2d61f519d62a1b42fadfcf2b4814a9e53c5e8533643f536e087f0ea2
INFO:DB:tx count: 70,603,569
INFO:History:history DB version: 1
INFO:History:flush count: 157
INFO:DB:deleted 925 stale undo entries
INFO:DB:populating header merkle cache...
INFO:MemPool:beginning processing of daemon mempool. This can take some time...
INFO:SessionManager:max session count: 674
INFO:SessionManager:session timeout: 600 seconds
INFO:SessionManager:session cost hard limit 10,000
INFO:SessionManager:session cost soft limit 1,000
INFO:SessionManager:bandwidth unit cost 5,000
INFO:SessionManager:request sleep 2,500ms
INFO:SessionManager:request timeout 30s
INFO:SessionManager:initial concurrent 10
INFO:SessionManager:max response size 10,000,000 bytes
INFO:MemPool:compact fee histogram: []
INFO:MemPool:synced in 0.28s
INFO:MemPool:182 txs 0.07 MB touching 581 addresses
INFO:SessionManager:TCP server listening on all_interfaces:50001
INFO:PeerManager:peer discovery: ON
INFO:PeerManager:announce ourself: True
INFO:PeerManager:my clearnet self: None
INFO:PeerManager:force use of proxy: False
INFO:PeerManager:beginning peer discovery...
INFO:PeerManager:trying to detect proxy on "localhost" ports [9050, 9150, 1080]
INFO:PeerManager:no proxy detected, will try later
Killed
ElectrumX config:
export DAEMON_URL=http://*:*@127.0.0.1:22555/
export DB_DIRECTORY=/home/ubuntu/db
export USERNAME=ubuntu
export COIN=Dogecoin
export NET=mainnet
export CACHE_MB=500
export SERVICES=tcp://:50001
Could there be a problem with coin setting in coin.py?
Error installing electrumx
I'm using Debian Buster and running a full bitcoin node.
I tried to install electrumx from the readthedocs.io instructions.
I ran this commands to install dependencies:
sudo apt install python3-aiohttp
pip3 install pylru
git clone https://github.com/spesmilo/electrumx.git
pip3 install .
But I got this error:
https://pastebin.com/M7PyQyMC
Then I tried issuing:
pip3 install plyvel
but got this error:
https://pastebin.com/6tgRxAGc
and
pip3 install .[rocksdb,ujson]
more errors:
https://pastebin.com/9QXD9tVq
What am I missing? Help, please!
Edited;
I was missing 2 libraries: libleveldb-dev && python3-leveldb
But I still can't install RocksDB with theabove command. Error is the same:
pip3 install .[rocksdb,ujson]
ElectrumX becoming unresponsive
Versions
python=3.7.1, 3.9.1
electrumx=1.16.0
aiohttp=3.7.0, 4.0
Context
After running ElectrumX for a few hours, when under very high load (2k+ sessions), sometimes it "locks up".
That is, the main thread blocks and stops making progress, it uses 100% CPU,
and it stops accepting new sessions.
The LocalRPC interface stops responding too; all requests there time out.
It never recovers on its own; the server (python process) must be restarted.
Symptoms
ElectrumX Nonresponsive for tcp, ssl, or rpc services
ElectrumX 100% cpu
Stack Trace
main thread traceback
--- Stack for thread 140338657445696 ---
File "/electrumx.git/./electrumx_server", line 43, in <module>
main()
File "/electrumx.git/./electrumx_server", line 35, in main
asyncio.run(controller.run())
File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 629, in run_until_complete
self.run_forever()
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
self._run_once()
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once
handle._run()
File "/usr/local/lib/python3.9/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/usr/local/lib/python3.9/site-packages/aiorpcx/session.py", line 496, in _throttled_request
await self._send_message(message)
File "/usr/local/lib/python3.9/site-packages/aiorpcx/session.py", line 153, in _send_message
await self.transport.write(message)
File "/usr/local/lib/python3.9/site-packages/aiorpcx/rawsocket.py", line 118, in write
self._asyncio_transport.write(framed_message)
File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 386, in write
self._ssl_protocol._write_appdata(data)
File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 598, in _write_appdata
self._process_write_backlog()
File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 674, in _process_write_backlog
ssldata, offset = self._sslpipe.feed_appdata(data, offset)
File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 261, in feed_appdata
offset += self._sslobj.write(view[offset:])
File "/usr/local/lib/python3.9/ssl.py", line 897, in write
return self._sslobj.write(data)
Known Workarounds
- set up nginx and let it terminate SSL
- If one places nginx in front of ElectrumX, and lets nginx handle SSL termination, the problem should not occur.
- In such a setup, nginx is reporting these errors:
2021/01/11 02:28:30 [crit] 21#21: *437620 SSL_shutdown() failed (SSL: error:14094123:SSL routines:ssl3_read_bytes:application data after close notify) while proxying connection, client: REDACTED, server: 0.0.0.0:50002, upstream: "127.0.0.1:50213", bytes from/to client:81/205, bytes from/to upstream:205/0
- Patch CPython.
- see #92 (comment)
- Use
uvloop- the problematic code is in the SSL wrappers in the stdlib asyncio code, which is not used when using uvloop
- set
EVENT_LOOP_POLICY=uvloop; this needsuvloopas an additional dependency
https://electrumx-spesmilo.readthedocs.io/en/latest/environment.html#envvar-EVENT_LOOP_POLICY
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 1022035: invalid start byte
ElectrumX been running for 5 days straight and just crashed with error:
2020-08-08 04:56:34,674:WARNING:Controller:received SIGINT signal, initiating shutdown
2020-08-08 04:56:34,689:ERROR:electrumx:ElectrumX server terminated abnormally
Traceback (most recent call last):
File "/home/nv01/.local/bin/electrumx_server", line 35, in main
asyncio.run(controller.run())
File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
return future.result()
File "/home/nv01/.local/lib/python3.7/site-packages/electrumx/lib/server_base.py", line 125, in run
await server_task
File "/home/nv01/.local/lib/python3.7/site-packages/electrumx/lib/server_base.py", line 98, in serve
await self.serve(shutdown_event)
File "/home/nv01/.local/lib/python3.7/site-packages/electrumx/server/controller.py", line 134, in serve
await group.spawn(wait_for_catchup())
File "/usr/local/lib/python3.7/dist-packages/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/usr/local/lib/python3.7/dist-packages/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/home/nv01/.local/lib/python3.7/site-packages/electrumx/server/mempool.py", line 335, in keep_synchronized
await group.spawn(self._logging(synchronized_event))
File "/usr/local/lib/python3.7/dist-packages/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/usr/local/lib/python3.7/dist-packages/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/home/nv01/.local/lib/python3.7/site-packages/electrumx/server/mempool.py", line 218, in _refresh_hashes
hex_hashes = await self.api.mempool_hashes()
File "/home/nv01/.local/lib/python3.7/site-packages/electrumx/server/daemon.py", line 233, in mempool_hashes
return await self._send_single('getrawmempool')
File "/home/nv01/.local/lib/python3.7/site-packages/electrumx/server/daemon.py", line 176, in _send_single
return await self._send(payload, processor)
File "/home/nv01/.local/lib/python3.7/site-packages/electrumx/server/daemon.py", line 134, in _send
result = await self._send_data(data)
File "/home/nv01/.local/lib/python3.7/site-packages/electrumx/server/daemon.py", line 108, in _send_data
return await resp.json(loads=json_deserialize)
File "/usr/local/lib/python3.7/dist-packages/aiohttp/client_reqrep.py", line 1040, in json
return loads(stripped.decode(encoding))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 1022035: invalid start byte
dmesg says:
[Mon Aug 3 19:47:08 2020] electrumx_serve[3458550]: segfault at 0 ip 0000000000000000 sp 00007ffce9084e28 error 14 in python3.7[400000+21000]
[Mon Aug 3 19:47:08 2020] Code: Bad RIP value.
System info:
OS: Debian GNU/Linux 10 (buster) x86_64
Kernel: 5.6.0-0.bpo.2-amd64
ElectrumX latest from git
ElectrumX server hang with "INFO:Prefetcher:cancelled; prefetcher stopping", and then crashed on SIGTERM
I observed that my electrumx_server hang completely and it was not responding to any queries, by clients or by RPC. I inspected the log:
(...)
2020-09-11 12:39:16,695:INFO:ElectrumX:[8329980] SSL 85.26.232.243:34434, 236 total
2020-09-11 12:39:16,823:INFO:ElectrumX:[8329981] SSL 123.25.70.15:65462, 236 total
2020-09-11 12:39:16,826:INFO:ElectrumX:[8329982] SSL 168.0.176.3:49000, 237 total
2020-09-11 12:39:17,147:INFO:ElectrumX:[8329983] SSL 103.224.185.201:57543, 232 total
2020-09-11 12:39:18,725:INFO:Prefetcher:cancelled; prefetcher stopping
2020-09-11 12:39:18,733:INFO:ElectrumX:[8329984] SSL 122.176.77.218:49194, 232 total
2020-09-11 12:39:18,805:INFO:BlockProcessor:flushing to DB for a clean shutdown...
2020-09-11 12:39:18,806:INFO:ElectrumX:[8329985] SSL 14.188.159.221:59038, 233 total
2020-09-11 12:39:18,806:INFO:ElectrumX:[8329986] SSL 5.127.4.37:35106, 234 total
2020-09-11 12:39:18,806:INFO:ElectrumX:[8329987] SSL 43.241.145.198:17845, 235 total
2020-09-11 12:39:18,807:INFO:ElectrumX:[8329988] SSL 115.74.225.96:57235, 236 total
2020-09-11 12:39:18,813:INFO:SessionManager:closing down server for ssl://all_interfaces:50002
2020-09-11 12:39:18,813:INFO:SessionManager:closing down server for rpc://localhost:8000
2020-09-11 12:39:18,857:INFO:ElectrumX:[940752] disconnected. Sent 1,694,708 bytes in 29,573 messages
2020-09-11 12:39:18,857:INFO:ElectrumX:[241949] disconnected. Sent 2,020,329 bytes in 35,213 messages
(...dozens disconnected messages and disconnected whilst throttled)
So by the look of it, electrumx_server has stopped operation, shutdown DB on it's own and just sat there. All clients were disconnected and all activity ceased.
Day after I discovered this state. I issued signal 15 SIGTERM from htop to kill hanging process. Then I noticed in the log:
2020-09-12 11:50:13,135:WARNING:Controller:received SIGTERM signal, initiating shutdown
2020-09-12 11:50:13,135:INFO:Controller:shutting down
2020-09-12 11:50:13,135:INFO:Controller:shutdown complete
2020-09-12 11:50:13,160:INFO:ElectrumX:[6417548] disconnected whilst throttled
2020-09-12 11:50:13,160:INFO:ElectrumX:[1721458] disconnected whilst throttled
2020-09-12 11:50:13,160:INFO:ElectrumX:[2920030] disconnected whilst throttled
2020-09-12 11:50:13,160:INFO:ElectrumX:[4422139] disconnected whilst throttled
2020-09-12 11:50:13,160:INFO:ElectrumX:[5994779] disconnected whilst throttled
Last 5 "disconnected whilst throttled" messages. And I thought electrum is completely disconnected, because it's been silent for 24 hours. Straight after these lines we have:
2020-09-12 11:50:13,161:ERROR:electrumx:ElectrumX server terminated abnormally
Traceback (most recent call last):
File "/home/nv01/.local/bin/electrumx_server", line 35, in main
asyncio.run(controller.run())
File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
return future.result()
File "/home/nv01/.local/lib/python3.7/site-packages/electrumx/lib/server_base.py", line 125, in run
await server_task
File "/home/nv01/.local/lib/python3.7/site-packages/electrumx/lib/server_base.py", line 98, in serve
await self.serve(shutdown_event)
File "/home/nv01/.local/lib/python3.7/site-packages/electrumx/server/controller.py", line 134, in serve
await group.spawn(wait_for_catchup())
File "/home/nv01/.local/lib/python3.7/site-packages/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/home/nv01/.local/lib/python3.7/site-packages/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/home/nv01/.local/lib/python3.7/site-packages/electrumx/server/mempool.py", line 335, in keep_synchronized
await group.spawn(self._logging(synchronized_event))
File "/home/nv01/.local/lib/python3.7/site-packages/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/home/nv01/.local/lib/python3.7/site-packages/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/home/nv01/.local/lib/python3.7/site-packages/electrumx/server/mempool.py", line 221, in _refresh_hashes
hashes = {hex_str_to_hash(hh) for hh in hex_hashes}
File "/home/nv01/.local/lib/python3.7/site-packages/electrumx/server/mempool.py", line 221, in <setcomp>
hashes = {hex_str_to_hash(hh) for hh in hex_hashes}
File "/home/nv01/.local/lib/python3.7/site-packages/electrumx/lib/hash.py", line 60, in hex_str_to_hash
return bytes(reversed(hex_to_bytes(x)))
ValueError: non-hexadecimal number found in fromhex() arg at position 50
Expected behaviour: electrumx_server should never issue "INFO:Prefetcher:cancelled; prefetcher stopping" and "INFO:BlockProcessor:flushing to DB for a clean shutdown..." unasked.
Also, on SIGTERM is should not crash like it did.
I restarted electrumX server and it's working fine. I issued SIGTERM once again right now to see how it behaves in normal way, this is what happens, as expected:
2020-09-12 20:10:46,137:WARNING:Controller:received SIGTERM signal, initiating shutdown
2020-09-12 20:10:46,137:INFO:Controller:shutting down
2020-09-12 20:10:46,138:INFO:Prefetcher:cancelled; prefetcher stopping
2020-09-12 20:10:46,140:INFO:BlockProcessor:flushing to DB for a clean shutdown...
2020-09-12 20:10:46,141:INFO:SessionManager:closing down server for ssl://all_interfaces:50002
2020-09-12 20:10:46,142:INFO:SessionManager:closing down server for rpc://localhost:8000
2020-09-12 20:10:46,161:INFO:ElectrumX:[110755] disconnected whilst throttled (...dozens of times...)
2020-09-12 20:10:47,156:INFO:Controller:shutdown complete
2020-09-12 20:10:47,157:INFO:electrumx:ElectrumX server terminated normally
Notice that during normal operation message "INFO:Controller:shutdown complete" was not instant like before. Before, electrumx was in "hung" state and there was nothing more to close, so "shutdown complete" message was instant at 0ms. In normal operation, it took 1019ms between "initiating shutdown" and "shutdown complete".
System:
OS: Debian GNU/Linux 10 (buster) x86_64
Kernel: 5.7.0-0.bpo.2-amd64
Electrum straight from git, latest commit:
commit 9ca59facfdb50daf063a5813fc9b08eb6fa1b246 (HEAD -> master, origin/master, origin/HEAD)
Merge: 148df48 23633c2
Author: ghost43 <[email protected]>
Date: Mon Aug 24 17:30:54 2020 +0000
electrumx memory leak
I am using latest electrumX from git:
commit 39d8ec50edcba0eabc3787eeb9229caf6f81cf42 (HEAD -> master, origin/master,$
Merge: 10f0faa f146e82
Author: ghost43 <[email protected]>
Date: Fri Oct 23 15:42:01 2020 +0000
Have to restart electrumX every day otherwise it will eat all memory. My environment variables:
datadir="/home/nv01/.electrumx"
export COIN=BitcoinSegwit
export DAEMON_URL=http://(my bitcoin core credentials)@localhost:8332
export NET=mainnet
export CACHE_MB=1200
export DB_DIRECTORY=$datadir
export SSL_CERTFILE=$datadir/fullchain.pem
export SSL_KEYFILE=$datadir/privkey.pem
export LOG_SESSIONS=300
export LOG_LEVEL=INFO
export LOG_FORMAT="%(asctime)s:%(levelname)s:%(name)s:%(message)s"
export BANDWIDTH_UNIT_COST=5000
export REQUEST_TIMEOUT=30
export DROP_CLIENT="3\.[0-4]\.\d+"
export DROP_CLIENT_UNKNOWN=YES
export SERVICES=ssl://:50002,rpc://:8000
export REPORT_SERVICES=ssl://(my address):50002
export EVENT_LOOP_POLICY=uvloop
export MAX_SESSIONS=1000
ulimit -n 10000
OS: Debian GNU/Linux 10 (buster) x86_64
Kernel: 5.8.0-0.bpo.2-amd64
$ python3 --version
Python 3.7.3
Anyt suggestions how to debug this?
The problem with hashx storage
I now want to get the direct read database to get all the wallet addresses. I see that the KEY storage in ROSKSDB is the last few digits of hashX, so is there any way I can convert the KEY in rosksdb to the wallet address and read it?
https://github.com/spesmilo/electrumx/blob/master/electrumx/server/session.py#L44
def scripthash_to_hashX(scripthash):
try:
bin_hash = hex_str_to_hash(scripthash)
if len(bin_hash) == 32:
return bin_hash[:HASHX_LEN]
except (ValueError, TypeError):
pass
raise RPCError(BAD_REQUEST, f'{scripthash} is not a valid script hash')
Electrumx keeps crashing randomly, 'ill-formed response error object: discovery is disabled'
Using latest version of ElectrumX, the server crashes randomly every x minutes to hours. It always fails with the below error:
aiorpcx.jsonrpc.ProtocolError: (-32600, 'ill-formed response error object: discovery is disabled')
I'm having the same error in the dockerized version as the normally built one. Is this a malicious node sending bad json which the server can't handle?
INFO:electrumx:ElectrumX server starting
INFO:electrumx:logging level: INFO
INFO:Controller:Python version: 3.7.7 (default, May 20 2020, 21:33:25) [GCC 9.2.0]
INFO:Controller:software version: ElectrumX 1.15.0
INFO:Controller:aiorpcX version: 0.18.4
INFO:Controller:supported protocol versions: 1.4-1.4.2
INFO:Controller:event loop policy: None
INFO:Controller:reorg limit is 200 blocks
INFO:Daemon:daemon #1 at 192.168.1.7:16789/ (current)
INFO:DB:switching current directory to /data
INFO:DB:using leveldb for DB backend
INFO:DB:opened UTXO DB (for sync: True)
INFO:DB:UTXO DB version: 8
INFO:DB:coin: BitcoinSegwit
INFO:DB:network: mainnet
INFO:DB:height: 649,428
INFO:DB:tip: 000000000000000000097e681bda79f82ebd041774467fb06e30b48c40b6ed42
INFO:DB:tx count: 570,555,113
INFO:DB:flushing DB cache at 1,200 MB
INFO:History:history DB version: 1
INFO:History:flush count: 1,036
INFO:DB:deleted 19 stale undo entries
INFO:SessionManager:RPC server listening on 0.0.0.0:8000
...
INFO:PeerManager:[xtrum.com:50002 SSL] verified in 0.2s
ERROR:PeerManager:[electrum5.hodlister.co:50002 SSL] marking bad: (too many peers already in bucket 142.44.179.119)
INFO:PeerManager:[electrum2.taborsky.cz:50002 SSL] verified in 0.5s
INFO:PeerManager:[bitcoins.sk:56002 SSL] verified in 0.4s
INFO:PeerManager:[167.172.226.175:50002 SSL] verified in 0.6s
INFO:PeerManager:[ttbit.mine.bz:50002 SSL] verified in 0.5s
INFO:PeerManager:[endthefed.onthewifi.com:50002 SSL] verified in 1.0s
INFO:PeerManager:[68.183.188.105:50002 SSL] verified in 1.1s
INFO:PeerManager:[ultracloud.tk:50002 SSL] verified in 0.8s
INFO:PeerManager:[satoshi.fan:50002 SSL] verified in 1.1s
INFO:SessionManager:closing down server for tcp://all_interfaces:50001
INFO:SessionManager:closing down server for wss://all_interfaces:50004
INFO:SessionManager:closing down server for rpc://0.0.0.0:8000
INFO:SessionManager:closing down server for ssl://all_interfaces:50002
INFO:Prefetcher:cancelled; prefetcher stopping
INFO:BlockProcessor:flushing to DB for a clean shutdown...
INFO:Controller:shutting down
INFO:Controller:shutdown complete
ERROR:electrumx:ElectrumX server terminated abnormally
Traceback (most recent call last):
File "/electrumx/electrumx_server", line 35, in main
asyncio.run(controller.run())
File "/usr/local/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
return future.result()
File "/electrumx/electrumx/lib/server_base.py", line 125, in run
await server_task
File "/electrumx/electrumx/lib/server_base.py", line 98, in serve
await self.serve(shutdown_event)
File "/electrumx/electrumx/server/controller.py", line 134, in serve
await group.spawn(wait_for_catchup())
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/electrumx/electrumx/server/session.py", line 621, in serve
await group.spawn(self._manage_servers())
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/electrumx/electrumx/server/peers.py", line 459, in discover_peers
await group.spawn(self._import_peers())
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/electrumx/electrumx/server/peers.py", line 234, in _monitor_peer
if await self._should_drop_peer(peer):
File "/electrumx/electrumx/server/peers.py", line 277, in _should_drop_peer
await self._verify_peer(session, peer)
File "/electrumx/electrumx/server/peers.py", line 373, in _verify_peer
(session, peer))
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/curio.py", line 242, in __aexit__
await self.join()
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/curio.py", line 211, in join
raise task.exception()
File "/electrumx/electrumx/server/peers.py", line 411, in _send_server_features
features = await session.send_request(message)
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/session.py", line 529, in send_request
return await self._send_concurrent(message, future, 1)
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/session.py", line 499, in _send_concurrent
return await future
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/jsonrpc.py", line 721, in receive_message
item, request_id = self._protocol.message_to_item(message)
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/jsonrpc.py", line 273, in message_to_item
return cls._process_response(payload)
File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.4-py3.7.egg/aiorpcx/jsonrpc.py", line 220, in _process_response
raise cls._error(code, message, False, request_id)
aiorpcx.jsonrpc.ProtocolError: (-32600, 'ill-formed response error object: discovery is disabled')
Electrumx keeps crashing during it's initial sync with Bitcoind. Out of memory?
Electrumx keeps crashing during it's initial sync with Bitcoind, seems memory related.
I'm running ElectrumX 1.15.0 on a Raspberry Pi4 with 4GB RAM, Raspbian 4.19.97.
Other running applications: Bitcoind 0.20.0 as backend, and LND 0.10.1 taking up some memory.
This is the error:
Jul 01 20:50:47 Pi electrumx_server[24345]: INFO:BlockProcessor:our height: 339,907 daemon: 637,215 UTXOs 1,621MB hist 582MB
Jul 01 20:51:47 Pi electrumx_server[24345]: INFO:BlockProcessor:our height: 340,004 daemon: 637,215 UTXOs 1,624MB hist 601MB
Jul 01 20:51:49 Pi electrumx_server[24345]: INFO:DB:flushed filesystem data in 1.89s
Jul 01 20:52:04 Pi electrumx_server[24345]: terminate called after throwing an instance of 'std::bad_alloc'
Jul 01 20:52:04 Pi electrumx_server[24345]: what(): std::bad_alloc
Jul 01 20:52:05 Pi systemd[1]: electrumx.service: Main process exited, code=killed, status=6/ABRT
Jul 01 20:52:05 Pi systemd[1]: electrumx.service: Failed with result 'signal'.
Jul 01 20:52:10 Pi systemd[1]: electrumx.service: Service RestartSec=5s expired, scheduling restart.
Jul 01 20:52:10 Pi systemd[1]: electrumx.service: Scheduled restart job, restart counter is at 1.
Jul 01 20:52:10 Pi systemd[1]: Stopped Electrumx server daemon.
I've set CACHE_MB = 2000, and then 3000, this delays the crash significantly but does not prevent it.
During sync I can see the process eating up my RAM over the course of an hour, when the system hits around 3.3g out of 3.81g total RAM the process crashes and the memory is freed up.
The process does not seem to release any RAM it uses even after succesfully flushing filesystem data several times before eventually crashing.
Releases newer than 1.2.1 are not tagged
Not sure why these didn't carry over with the fork, but they're currently missing.
electrumx memory leak
Another memory leak has been found.
I am using older, stable version on electrumx, since problems with memory leak last time, I settled on older, stable version and stayed there.
commit 835aa492e9a3563d89d99c35168378d65321b849 (HEAD, tag: 1.15.0)
Author: Neil Booth <[email protected]>
Date: Wed May 27 11:23:38 2020 +0100
Prepare 1.15
This memory leak is a lot slower. Process has reached 5 GB RAM after 6 days:
electrumx_rpc getinfo
{
"coin": "BitcoinSegwit",
"daemon": "localhost:8332/",
"daemon height": 655159,
"db height": 655159,
"db_flush_count": 1648,
"groups": 3115,
"history cache": "3,243,657 lookups 536,633 hits 1,000 entries",
"merkle cache": "22,953 lookups 10,313 hits 1,000 entries",
"peers": {
"bad": 6,
"good": 71,
"never": 12,
"stale": 14,
"total": 103
},
"pid": 501797,
"request counts": {
"blockchain.block.header": 70370,
"blockchain.block.headers": 2660,
"blockchain.estimatefee": 23735419,
"blockchain.headers.subscribe": 135572,
"blockchain.relayfee": 24953,
"blockchain.scripthash.get_balance": 283748,
"blockchain.scripthash.get_history": 155753,
"blockchain.scripthash.listunspent": 165276,
"blockchain.scripthash.subscribe": 2920814,
"blockchain.transaction.broadcast": 1511,
"blockchain.transaction.get": 43903,
"blockchain.transaction.get_merkle": 23266,
"getinfo": 41,
"mempool.get_fee_histogram": 1897426,
"server.add_peer": 1146,
"server.banner": 24969,
"server.donation_address": 24962,
"server.features": 10299,
"server.peers.subscribe": 78146,
"server.ping": 1464476,
"server.version": 3696202
},
"request total": 34760912,
"sessions": {
"count": 931,
"count with subs": 248,
"errors": 21,
"logged": 0,
"pending requests": 37,
"subs": 114081
},
"tx hashes cache": "23,266 lookups 9,745 hits 1,000 entries",
"txs sent": 1400,
"uptime": "6d 12h 41m",
"version": "ElectrumX 1.15.0"
}
Electrum is running perfectly fine, serving a ton of clients, but memory is slowly creeping, 788 MB per day.
OS: Debian GNU/Linux 10 (buster) x86_64
Kernel: 5.8.0-0.bpo.2-amd64
$ python3 --version
Python 3.7.3
$ pip list |grep aio
aiohttp 3.7.1
aiorpcX 0.18.4
I'd appreciate any help debugging this. What should I upgrade/downgrade first? Waiting for suggestions. I want to make accurate and worthwile test runs, as each test takes few days.
ElectrumX & Websocket error at coro=<RPCSession._throttled_request() with a 1005 code
Hello,
I am currently using ElectrumX 1.15.0 with Protocol version 1.4.2 for Bitcoin and Websockets 8.1. I was trying to debug another issue and I got the following in my way. I need to mention that the following error did not cause my server to crash, but when I tried to reproduce it multiple times it eventually crashed but with a different exception.
Coin: Bitcoin (happens with BitcoinSV as well)
ElectrumX: latest (1.15.0 with Protocol version 1.4.2)
Services: RPC, WebSockets 8.1
OS: Ubuntu 18.04 (in Docker)
I was testing it with artillery.io in a local REGTEST enviroment, setup:
npm install -g artilleryvim test.yml- In test.yml insert the following:
config:
target: "ws://XXXX:XXX"
tls:
rejectUnauthorized: false
plugins:
fuzzer: {}
phases:
- duration: 60
arrivalRate: 20
subprotocols:
- json
scenarios:
- engine: "ws"
flow:
- send: '{"id":0,"method":"server.version","params":["1.18.8-dev_289_android","1.4.2"]}'
- send: '{"id":1,"method":"blockchain.headers.subscribe"}'
- send: '{"id":2,"method":"blockchain.scripthash.subscribe","params":["963c7985bf4123e2d650c707458d6f731b2a0de6ffebbbcb3198f973aecb52c5"]}'
- send: '{"id":3,"method":"blockchain.scripthash.subscribe","params":["578b41ac4333609a6216b0595d88df72070e734fcbd6ec58c3f2b6900d1f14cf"]}'
- send: '{"id":4,"method":"blockchain.scripthash.subscribe","params":["6be90d7a1835ad86fa647e429e7e4f849c8da575abbcbf77178b4e80008217ba"]}'
- send: '{"id":5,"method":"blockchain.scripthash.subscribe","params":["fc9794b06359653b29a2425f16401dba7e18e2f59ba63275d8146f2e420b8f59"]}'
artillery run test.yml
The error logs are the following:
INFO:electrumx:ElectrumX server starting
INFO:electrumx:logging level: DEBUG
INFO:Controller:Python version: 3.9.0b3 (default, Jun 12 2020, 00:52:24) [GCC 9.3.0]
INFO:Controller:software version: ElectrumX 1.15.0
INFO:Controller:aiorpcX version: 0.18.4
INFO:Controller:supported protocol versions: 1.4-1.4.2
INFO:Controller:event loop policy: None
INFO:Controller:reorg limit is 8,000 blocks
INFO:Daemon:daemon #1 at node:19001/ (current)
INFO:DB:switching current directory to /data
INFO:DB:using leveldb for DB backend
ERROR:Daemon:starting up checking blocks. Retrying occasionally...
INFO:Daemon:running normally
INFO:DB:created new database
INFO:DB:creating metadata directory
INFO:DB:UTXO DB version: 8
INFO:DB:coin: BitcoinSegwit
INFO:DB:network: regtest
INFO:DB:height: -1
INFO:DB:tip: 0000000000000000000000000000000000000000000000000000000000000000
INFO:DB:tx count: 0
INFO:DB:flushing DB cache at 1,200 MB
INFO:DB:sync time so far: 00s
INFO:History:history DB version: 1
INFO:History:flush count: 0
INFO:SessionManager:RPC server listening on 0.0.0.0:8000
INFO:Prefetcher:catching up to daemon height 0 (1 blocks behind)
INFO:Prefetcher:verified genesis block with hash 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206
INFO:BlockProcessor:our height: 0 daemon: 0 UTXOs 0MB hist 0MB
INFO:BlockProcessor:caught up to height 0
INFO:DB:flushed filesystem data in 0.00s
INFO:History:flushed history in 0.0s for 0 addrs
INFO:DB:flushed 1 blocks with 0 txs, 0 UTXO adds, 0 spends in 0.0s, committing...
INFO:DB:flush #1 took 0.0s. Height 0 txs: 0 (+0)
INFO:DB:tx/sec since genesis: 0, since last flush: 1
INFO:DB:sync time: 00s ETA: -1d 23h 59m
INFO:BlockProcessor:ElectrumX 1.15.0 synced to height 0
INFO:DB:closing DBs to re-open for serving
INFO:DB:opened UTXO DB (for sync: False)
INFO:DB:UTXO DB version: 8
INFO:DB:coin: BitcoinSegwit
INFO:DB:network: regtest
INFO:DB:height: 0
INFO:DB:tip: 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206
INFO:DB:tx count: 0
INFO:History:history DB version: 1
INFO:History:flush count: 1
INFO:DB:populating header merkle cache...
INFO:MemPool:beginning processing of daemon mempool. This can take some time...
INFO:DB:header merkle cache populated in 0.0s
INFO:SessionManager:max session count: 1,000
INFO:SessionManager:session timeout: 600 seconds
INFO:SessionManager:session cost hard limit 10,000
INFO:SessionManager:session cost soft limit 1,000
INFO:SessionManager:bandwidth unit cost 5,000
INFO:SessionManager:request sleep 2,500ms
INFO:SessionManager:request timeout 30s
INFO:SessionManager:initial concurrent 10
INFO:SessionManager:max response size 1,000,000 bytes
INFO:ElectrumX:[265] WS 172.18.0.1:51658, 0 total
INFO:ElectrumX:[266] WS 172.18.0.1:51662, 0 total
INFO:ElectrumX:[267] WS 172.18.0.1:51666, 0 total
Task exception was never retrieved
future: <Task finished name='Task-3565' coro=<RPCSession._process_messages() done, defined at /usr/local/lib/python3.9/site-packages/aiorpcX-0.18.4-py3.9.egg/aiorpcx/session.py:432> exception=ConnectionClosedError('code = 1005 (no status code [internal]), no reason')>
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/aiorpcX-0.18.4-py3.9.egg/aiorpcx/session.py", line 435, in _process_messages
message = await recv_message()
File "/usr/local/lib/python3.9/site-packages/aiorpcX-0.18.4-py3.9.egg/aiorpcx/websocket.py", line 63, in recv_message
message = await self.websocket.recv()
File "/usr/local/lib/python3.9/site-packages/websockets/protocol.py", line 509, in recv
await self.ensure_open()
File "/usr/local/lib/python3.9/site-packages/websockets/protocol.py", line 812, in ensure_open
raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1005 (no status code [internal]), no reason
Task exception was never retrieved
future: <Task finished name='Task-3569' coro=<RPCSession._throttled_request() done, defined at /usr/local/lib/python3.9/site-packages/aiorpcX-0.18.4-py3.9.egg/aiorpcx/session.py:458> exception=ConnectionClosedError('code = 1005 (no status code [internal]), no reason')>
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/aiorpcX-0.18.4-py3.9.egg/aiorpcx/session.py", line 488, in _throttled_request
await self._send_message(message)
File "/usr/local/lib/python3.9/site-packages/aiorpcX-0.18.4-py3.9.egg/aiorpcx/session.py", line 153, in _send_message
await self.transport.write(message)
File "/usr/local/lib/python3.9/site-packages/aiorpcX-0.18.4-py3.9.egg/aiorpcx/websocket.py", line 85, in write
await self.websocket.send(framed_message)
File "/usr/local/lib/python3.9/site-packages/websockets/protocol.py", line 555, in send
await self.ensure_open()
File "/usr/local/lib/python3.9/site-packages/websockets/protocol.py", line 812, in ensure_open
raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1005 (no status code [internal]), no reason
I noticed that these errors occur for:
- send: '{"id":any_id,"method":"blockchain.scripthash.subscribe","params":["put_anything_valid_here"]}'
and again, not all the times. Sometimes they take place, others don't, what I have observed is that they don't take place that much at the start of the server.
Any ideas what is causing this type of behavior?
PS: I posted it on kyuupichan's repo as well because it happens for BitcoinSV (and I guess for all Bitcoin-based coins).
electrumx crash
electrumx_server[3841]: ERROR:electrumx:ElectrumX server terminated abnormally
electrumx_server[3841]: Traceback (most recent call last):
electrumx_server[3841]: File "/usr/local/bin/electrumx_server", line 35, in main
electrumx_server[3841]: asyncio.run(controller.run())
electrumx_server[3841]: File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
electrumx_server[3841]: return loop.run_until_complete(main)
electrumx_server[3841]: File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
electrumx_server[3841]: return future.result()
electrumx_server[3841]: File "/home/electrum/electrumx/electrumx/lib/server_base.py", line 129, in run
electrumx_server[3841]: await server_task
electrumx_server[3841]: File "/home/electrum/electrumx/electrumx/lib/server_base.py", line 102, in serve
electrumx_server[3841]: await self.serve(shutdown_event)
electrumx_server[3841]: File "/home/electrum/electrumx/electrumx/server/controller.py", line 134, in serve
electrumx_server[3841]: await group.spawn(wait_for_catchup())
electrumx_server[3841]: File "/home/electrum/.local/lib/python3.7/site-packages/aiorpcx/curio.py", line 242, in __aexit__
electrumx_server[3841]: await self.join()
electrumx_server[3841]: File "/home/electrum/.local/lib/python3.7/site-packages/aiorpcx/curio.py", line 211, in join
electrumx_server[3841]: raise task.exception()
electrumx_server[3841]: File "/home/electrum/electrumx/electrumx/server/mempool.py", line 366, in keep_synchronized
electrumx_server[3841]: await group.spawn(self._logging(synchronized_event))
electrumx_server[3841]: File "/home/electrum/.local/lib/python3.7/site-packages/aiorpcx/curio.py", line 242, in __aexit__
electrumx_server[3841]: await self.join()
electrumx_server[3841]: File "/home/electrum/.local/lib/python3.7/site-packages/aiorpcx/curio.py", line 211, in join
electrumx_server[3841]: raise task.exception()
electrumx_server[3841]: File "/home/electrum/electrumx/electrumx/server/mempool.py", line 263, in _refresh_hashes
electrumx_server[3841]: await self.api.on_mempool(touched, height)
electrumx_server[3841]: File "/home/electrum/electrumx/electrumx/server/controller.py", line 67, in on_mempool
electrumx_server[3841]: await self._maybe_notify()
electrumx_server[3841]: File "/home/electrum/electrumx/electrumx/server/controller.py", line 55, in _maybe_notify
electrumx_server[3841]: await self.notify(height, touched)
electrumx_server[3841]: File "/home/electrum/electrumx/electrumx/server/session.py", line 787, in _notify_sessions
electrumx_server[3841]: await self._task_group.spawn(session.notify, touched, height_changed)
electrumx_server[3841]: File "/home/electrum/.local/lib/python3.7/site-packages/aiorpcx/curio.py", line 148, in spawn
electrumx_server[3841]: self._add_task(task)
electrumx_server[3841]: File "/home/electrum/.local/lib/python3.7/site-packages/aiorpcx/curio.py", line 123, in _add_task
electrumx_server[3841]: raise RuntimeError('task group is closed')
electrumx_server[3841]: RuntimeError: task group is closed
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.