chris-belcher / electrum-personal-server Goto Github PK

If you're using BitcoinQT then the watch-only wallet used for the Electron server adds visual clutter. It would be nice if the Python script loaded the wallet at launch and then unloads it on exit.

This wouldn't be appropriate on a pruned node though, because it may be force to rescan beyond the prune horizon.

I'm seeing [DEBUG] utxo not found(!) and then the server crashes with:

  File "electrum-personal-server/electrumpersonalserver/transactionmonitor.py", line 219, in generate_new_history_element
    total_input_value += int(Decimal(utxo["value"]) * Decimal(1e8))
TypeError: 'NoneType' object is not subscriptable

I think this happened as a result of me spending an unconfirmed (very low fee) transaction using another transaction (with a low fee).

I added both raw transactions to the node and they do show up with getrawtransaction, but the error won't go away.

The server log shows adding obtained tx for the parent transaction. Crash seems to happen while processing the child. I think it's trying to access a (confirmed) parent of the parent, using gettxout. Without txindex it won't find that.

Would be nice to have some indication of rescan progress, say for example a message every 500 or 1000 blocks.

After a double spend, server.py produces
20:49:21,991456 [ LOG] Building history with 3600 addresses
Traceback (most recent call last):
File "./server.py", line 464, in
main()
File "./server.py", line 450, in main
if not txmonitor.build_address_history(relevant_spks_addrs):
File "/home/kid/Projects/electrum-personal-server/electrumpersonalserver/transactionmonitor.py", line 140, in build_address_history
new_history_element = self.generate_new_history_element(tx, txd)
File "/home/kid/Projects/electrum-personal-server/electrumpersonalserver/transactionmonitor.py", line 215, in generate_new_history_element
blockheader = self.rpc.call("getblockheader", [tx['blockhash']])
KeyError: 'blockhash'

================================================================
The problem is cleared up when:

1. delete ~/.bitcoin/wallet.dat
2. bitcoin-cli rescanblockchain 490000 (or any relevent block)
3. run ./server.py
   21:01:35,874627 [ LOG] Importing 360 addresses in total
   [0%]... [10%]... [20%]... [30%]... [40%]... [50%]... [60%]... [70%]... [80%]... [90%]... [100%]
   21:03:15,973858 [ LOG] Importing done
   21:03:15,974018 [ LOG] Done.
   If recovering a wallet which already has existing transactions, then
   run the rescan script. If you're confident that the wallets are new
   and empty then there's no need to rescan, just restart this script
   ==================================================================

Looks OK. A better error reporting is needed to catch this error.

Keep up the awesome work. Thanks.

While reviewing #38 it struck me, most of the rough edges are because EPS does not use setuptools, and isn't installable with pip. I'm guessing since EPS isn't completely matured yet, simply no one has gotten around to addressing mundane issues like this?

Electrum seems to use xpub..., ypub... an zpub... to distinguish between legacy, p2sh-segwit and bech32 derivations, but afaik that's not a standard.

This causes problems if you e.g. export the xpub from a ledger, and simply replacing "x" with "y" or "z" isn't a viable workaround either because the checksum won't match.

Adding an explicit address type config option would be useful here.

As a workaround, either get the ypub / zpub from Electrum, or edit deterministicwallet.py and change SingleSigP2PKHWallet in parse_electrum_master_public_key (3...) to SingleSigP2WPKH_P2SHWallet or SingleSigP2WPKHWallet (bech32).

Hi, can this please be explained to me. I can just broadcast transaction with electrum when connected to eps? The transaction then goes into my bitcoind node? And then, how do I ensure it mixes with other transactions that are broadcast by my node?

Or basically how do I broadcast a transaction without revealing my ip?

Error with latest Bitcoind build, depreciated 'getaddressesbyaccount', which has to be manually added to bitcoin.conf, after which electrum-personal-server works again. Output below.

./server.py
2018-08-13 01:23:29,699618 [ LOG] Obtaining bitcoin addresses to monitor . . .
Traceback (most recent call last):
File "./server.py", line 559, in
main()
File "./server.py", line 526, in main
get_scriptpubkeys_to_monitor(rpc, config)
File "./server.py", line 384, in get_scriptpubkeys_to_monitor
[transactionmonitor.ADDRESSES_LABEL]))
File "/home/user/electrum-personal-server/electrumpersonalserver/jsonrpc.py", line 58, in call
raise JsonRpcError(response["error"])
electrumpersonalserver.jsonrpc.JsonRpcError: {'message': 'getaddressesbyaccount is deprecated and will be removed in V0.18. To use this command, start bitcoind with -deprecatedrpc=accounts.', 'code': -32}

https://github.com/robust-cuttlefish/electrum-personal-server-GUI
What do y'all think? Is it useful to anyone? It's 700 lines of code, but it can be improved by converting repeated procedures to functions.

When an electrum client is connected, a block reorg causes EPS to crash with error :

electrumpersonalserver.jsonrpc.JsonRpcError: {'code': -8, 'message': 'Block height out of range'}

How to reproduce :

1. mine a few blocks
2. launch EPS
3. launch electrum
4. initiate invalidateblock <current block hash> with bitcoin-cli
5. mine a single block

EPS will crash with log :

20:23:26,880314 [  LOG] Building history with 200 addresses
20:23:26,893473 [  LOG] Found 0 txes. History built in 0.002653360366821289sec
20:23:26,893597 [  LOG] Starting electrum server
20:23:26,893667 [  LOG] Listening for Electrum Wallet on ('127.0.0.1', 50012)
20:23:39,616938 [  LOG] Electrum connected from ('127.0.0.1', 51648)
20:23:39,618929 [  LOG] Electrum wallet disconnected
20:23:39,829962 [  LOG] Electrum connected from ('127.0.0.1', 51650)
Traceback (most recent call last):
  File "p2pkh/server.py", line 485, in <module>
    main()
  File "p2pkh/server.py", line 482, in main
    poll_interval_connected, certfile, keyfile)
  File "p2pkh/server.py", line 302, in run_electrum_server
    txmonitor)
  File "p2pkh/server.py", line 152, in handle_query
    blockhash = rpc.call("getblockhash", [query["params"][0]])
  File "/home/user/software/crypto/electrum-personal-server/electrumpersonalserver/jsonrpc.py", line 58, in call
    raise JsonRpcError(response["error"])
electrumpersonalserver.jsonrpc.JsonRpcError: {'message': 'Block height out of range', 'code': -8}

The crash doesn't happen if an electrum client is not connected to the server.

I updated bitcoind to 0.17.0.
EPS had no issues with 0.17.0 until I deleted wallet.dat to refresh it with new master public keys.

After deleting wallet.dat, I stared EPS with
python3.6 server.py to populate the wallet with new address. I received an error from EPS right of the bat.

python3.6 server.py
2018-10-05 13:56:31,693779 [ LOG] Obtaining bitcoin addresses to monitor . . .
Traceback (most recent call last):
File "server.py", line 388, in get_scriptpubkeys_to_monitor
[transactionmonitor.ADDRESSES_LABEL]))
File "/home/dietpi/A/eps/electrumpersonalserver/jsonrpc.py", line 58, in call
raise JsonRpcError(response["error"])
electrumpersonalserver.jsonrpc.JsonRpcError: {'code': -32, 'message': 'getaddressesbyaccount is deprecated and will be removed in V0.18. To use this command, start bitcoind with -deprecatedrpc=accounts.'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "server.py", line 568, in
main()
File "server.py", line 535, in main
get_scriptpubkeys_to_monitor(rpc, config)
File "server.py", line 393, in get_scriptpubkeys_to_monitor
[transactionmonitor.ADDRESSES_LABEL]).keys())
File "/home/dietpi/A/eps_bk/electrumpersonalserver/jsonrpc.py", line 58, in call
raise JsonRpcError(response["error"])
electrumpersonalserver.jsonrpc.JsonRpcError: {'code': -11, 'message': 'No addresses with label electrum-watchonly-addresses'}

Hi,
I get the following error when trying to connect Electrum wallet to EPS on Lubuntu.

2018-08-08 16:31:07,926835 [ LOG] Electrum connected from ('127.0.0.1', 39146)
2018-08-08 16:31:40,306907 [ LOG] IOError: BrokenPipeError(32, 'Broken pipe')
2018-08-08 16:40:11,107133 [ LOG] IOError: OSError(0, 'Error')
2018-08-08 16:40:11,710609 [ LOG] IOError: OSError(0, 'Error')
2018-08-08 16:40:11,925471 [ LOG] IOError: OSError(0, 'Error')
2018-08-08 16:40:13,560771 [ LOG] Electrum connected from ('127.0.0.1', 39464)
2018-08-08 16:41:17,486015 [ LOG] IOError: BrokenPipeError(32, 'Broken pipe')
2018-08-08 16:41:17,708431 [ LOG] IOError: OSError(0, 'Error')
2018-08-08 16:41:17,947199 [ LOG] IOError: OSError(0, 'Error')`

spesmilo/electrum@master...bill-walker:master

i fixed electrum to work with electrum-personal-server and be idiot proof. not even possible to leak addresses by mistake. (as long as noob doesn't put a public electrum server in there.)

anyone want to review my hacks?

The issue is similar to #2. This occurs in rescan-script.py

It is not in the master branch, but it is part of the release zips

Since electrum doesn't support regtest out of the box, current testing is done on testnet, but this is not optimal for a few reasons.
Testnet is slow, some miners do not include segwit transactions in their blocks and at this point even the coins are scarce enough that if some are lost, it becomes quite annoying to get more.
Also testing the more exotic things like double spending and reorgs requires mining (and competing against other miners). Even at difficulty 1, cpu mining is cumbersome and can take minutes to mine even one block.

For the EPS setup to support regtest, first we need to patch Electrum for regtest support.
Between themselves, testnet and regtest are not so different, except for two things (might be more that I didn't consider):

1. Bech32 addresses are different. Testnet uses tb as the human readable part while regtest uses bcrt
2. Checkpoints make no sense on regtest. The only block that would be the same for a new regtest run is the genesis block.

For EPS itself, point (1) does not seem like issue. It relies on the underlying bitcoind for getting the addresses and as long as it supports bech32 on regtest (which Core does), it should be alright.
As for the checkpoints, it seems that both get_block_header() and the blockchain.block.get_chunk method in handle_query() both expect a previousblockhash in the json return by getblockheader. This value doesn't exist when the genesis block's header is requested.

This causes EPS to fail with :

Traceback (most recent call last):                                  
  File "p2pkh/server.py", line 473, in <module>                              
    main()                                                                                                                                     
  File "p2pkh/server.py", line 470, in main                                                                      
    poll_interval_connected, certfile, keyfile)                                                                                                
  File "p2pkh/server.py", line 290, in run_electrum_server                                                       
    txmonitor)                                                                                                                                 
  File "p2pkh/server.py", line 153, in handle_query                                                              
    header = get_block_header(rpc, blockhash)                                                                                                  
  File "p2pkh/server.py", line 224, in get_block_header                                                          
    "prev_block_hash": rpc_head["previousblockhash"],                                                                                          
KeyError: 'previousblockhash'

and :

Traceback (most recent call last):                                                                               
  File "p2pkh/server.py", line 482, in <module>                                                                                                
    main()                                                                                                       
  File "p2pkh/server.py", line 479, in main                                                                                                    
    poll_interval_connected, certfile, keyfile)                                                                  
  File "p2pkh/server.py", line 299, in run_electrum_server                                                                                     
    txmonitor)                                                                                                                                 
  File "p2pkh/server.py", line 170, in handle_query                                     
    binascii.unhexlify(header["previousblockhash"])[::-1],      
KeyError: 'previousblockhash'

Luckily, the previous block hash of genesis is defined as 0000000000000000000000000000000000000000000000000000000000000000 (at least in the header of the block), so it's possible to patch that in as well, in case the previousblockhash doesn't exist.

I've done the needed patching for both Electrum and EPS and it seems to work well (I only tested p2pkh and p2wpkh for now).
Electrum is at:
https://github.com/fivepiece/electrum/tree/add-network-regtest

And the diff is (done against version 3.1.2 of electrum):

diff --git a/electrum b/electrum
index 783d5c13..caa0b2a4 100755
--- a/electrum
+++ b/electrum
@@ -417,6 +417,9 @@ if __name__ == '__main__':
     if config.get('testnet'):
         constants.set_testnet()
 
+    if config.get('regtest'):
+        constants.set_regtest()
+
     # run non-RPC commands separately
     if cmdname in ['create', 'restore']:
         run_non_RPC(config)
diff --git a/lib/commands.py b/lib/commands.py
index 8435cf9e..35b41e62 100644
--- a/lib/commands.py
+++ b/lib/commands.py
@@ -806,6 +806,7 @@ def add_global_options(parser):
     group.add_argument("-P", "--portable", action="store_true", dest="portable", default=False, help="Use local 'electrum_data' directory")
     group.add_argument("-w", "--wallet", dest="wallet_path", help="wallet path")
     group.add_argument("--testnet", action="store_true", dest="testnet", default=False, help="Use Testnet")
+    group.add_argument("--regtest", action="store_true", dest="regtest", default=False, help="Use Regtest")
 
 def get_parser():
     # create main parser
diff --git a/lib/constants.py b/lib/constants.py
index ec35cbe3..620db80e 100644
--- a/lib/constants.py
+++ b/lib/constants.py
@@ -93,6 +93,34 @@ class BitcoinTestnet:
     }
 
 
+class BitcoinRegtest:
+
+    TESTNET = True
+    WIF_PREFIX = 0xef
+    ADDRTYPE_P2PKH = 111
+    ADDRTYPE_P2SH = 196
+    SEGWIT_HRP = "bcrt"
+    GENESIS = "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206"
+    DEFAULT_PORTS = {'t': '51001', 's': '51002'}
+    DEFAULT_SERVERS = read_json('servers_regtest.json', {})
+    CHECKPOINTS = []
+
+    XPRV_HEADERS = {
+        'standard':    0x04358394,  # tprv
+        'p2wpkh-p2sh': 0x044a4e28,  # uprv
+        'p2wsh-p2sh':  0x024285b5,  # Uprv
+        'p2wpkh':      0x045f18bc,  # vprv
+        'p2wsh':       0x02575048,  # Vprv
+    }
+    XPUB_HEADERS = {
+        'standard':    0x043587cf,  # tpub
+        'p2wpkh-p2sh': 0x044a5262,  # upub
+        'p2wsh-p2sh':  0x024285ef,  # Upub
+        'p2wpkh':      0x045f1cf6,  # vpub
+        'p2wsh':       0x02575483,  # Vpub
+    }
+
+
 # don't import net directly, import the module instead (so that net is singleton)
 net = BitcoinMainnet
 
@@ -105,3 +133,8 @@ def set_mainnet():
 def set_testnet():
     global net
     net = BitcoinTestnet
+
+
+def set_regtest():
+    global net
+    net = BitcoinRegtest
diff --git a/lib/servers_regtest.json b/lib/servers_regtest.json
new file mode 100644
index 00000000..c54339ed
--- /dev/null
+++ b/lib/servers_regtest.json
@@ -0,0 +1,8 @@
+{
+    "127.0.0.1": {
+        "pruning": "-",
+        "s": "50002",
+        "t": "50001",
+        "version": "1.2"
+    }
+}
diff --git a/lib/simple_config.py b/lib/simple_config.py
index 34c62d3f..7e6ff2d6 100644
--- a/lib/simple_config.py
+++ b/lib/simple_config.py
@@ -116,6 +116,10 @@ class SimpleConfig(PrintError):
             path = os.path.join(path, 'testnet')
             make_dir(path)
 
+        if self.get('regtest'):
+            path = os.path.join(path, 'regtest')
+            make_dir(path)
+
         self.print_error("electrum directory", path)
         return path
 
diff --git a/setup.py b/setup.py
index 58c938e5..a513d3b5 100755
--- a/setup.py
+++ b/setup.py
@@ -74,6 +74,7 @@ setup(
         'electrum': [
             'servers.json',
             'servers_testnet.json',
+            'servers_regtest.json',
             'currencies.json',
             'checkpoints.json',
             'checkpoints_testnet.json',

EPS PR is coming up at #7 .

(edit to electrum diff : added creation of the regtest directory)

Seems it tries to access a nonexistent key. I have currently two unconfirmed transactions.

 File "server.py", line 534, in <module>
    main()
  File "server.py", line 512, in main
    if not txmonitor.build_address_history(relevant_spks_addrs):
  File "/home/bitcoin/electrum-personal-server/electrumpersonalserver/transactionmonitor.py", line 145, in build_address_history
    new_history_element = self.generate_new_history_element(tx, txd)
  File "/home/bitcoin/electrum-personal-server/electrumpersonalserver/transactionmonitor.py", line 214, in generate_new_history_element
    utxo = {"confirmations": rawtx["confirmations"],
KeyError: 'confirmations'

I haven't found the way. If one derives the keys in a way to obtain those kind of addresses of also bech32, Ii is not documented how to do it in electrum-personal-server.

There is a proposal Pieter Wuille is working on (https://gist.github.com/sipa/e3d23d498c430bb601c5bca83523fa82) that seams to be related to how Electrum and other wallets could deal with these kind of issues. This could be related also with #27 because the scantxoutset proposal will probably take into account that.

running python3 server.py gives

11:54:27,417584 [ LOG] Unable to find .cookie file, try setting datadir config

my .bitcoin directory does not have .cookie file.
if it helps, my bitcoin.conf file:
zmqpubrawblock=tcp://127.0.0.1:8501
zmqpubrawtx=tcp://127.0.0.1:8501
server=1
daemon=1
dbcache=150
rpcuser=*******
rpcpassword=*******
rpcport=8332
rpcallowip=127.0.0.1
rpcthreads=10
port=6667
bind=127.0.0.1:6667
#disablewallet=1
#checklevel=1
#minrelaytxfee=0.000001
limitfreerelay=6
txindex=1
maxconnections=60

When I run ./server.py for the first time using rpc credentials, I get an Exception: Invalid checksum from ".electrum-server/bitcoin/deterministic.py", line 73, in bip32_deserialize

I'm using my Ledger Nano's YPUB to initiate the addresses. Is this something I can fix?

I was contacted by someone who had EPS crash when check_for_new_txes was called. The crash was the assertion assert(recent_tx_index != -1).

It ended up being fixed when MAX_TX_REQUEST_COUNT was increased to 2048. Which is strange.

They had about 800 addresses and about 450 transactions.

This could be debugged with testnet or by importing bitfinex's cold storage multisig which has many many transactions on it.

When I run ./server.py, EPS builds my address history and starts listening for Electrum wallet on 127.0.0.1, 50002. However, when I start Electrum, I can't find 'localhost' in the Tools > Network > Server menu, so I can't connect to EPS.

I'm running up to date versions of Linux Mint, Bitcoin Core, lnd, Electrum, and a hardware wallet, with everything saved in default locations.

Here's my bitcoin.conf:

txindex=1
rpcuser=OMITTED
rpcpassword=OMITTED
server=1
rpcport=8332
daemon=1
zmqpubrawblock=tcp://127.0.0.1:28332
zmqpubrawtx=tcp://127.0.0.1:28332
maxconnections=10

When I run netstat -l, I get this:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:28332 : LISTEN
tcp 0 0 localhost:8332 : LISTEN
tcp 0 0 :8333 : LISTEN
tcp 0 0 localhost:50002 : LISTEN
tcp 0 0 localhost:46453 : LISTEN
tcp 0 0 legendarybirds:domain : LISTEN
tcp 0 0 localhost:ipp : LISTEN
tcp6 0 0 ip6-localhost:8332 [::]: LISTEN
tcp6 0 0 [::]:8333 [::]:* LISTEN
tcp6 0 0 ip6-localhost:ipp [::]:* LISTEN

Here's an excerpt from my debug.log (the bottom rows repeat every 30 seconds):

00:26:18,906891 [ LOG] Found 79 txes. History built in 0.5241124629974365sec
00:26:18,907108 [ LOG] Starting electrum server
00:26:18,907187 [ LOG] Listening for Electrum Wallet on ('127.0.0.1', 50002)
00:26:48,920511 [DEBUG] on heartbeat listening
00:26:48,920890 [DEBUG] listtransactions tx_request_count=2
00:26:48,924491 [DEBUG] recent tx index = 0 ret = [('OMITTED', 'OMITTED'), ('OMITTED', 'OMITTED')]
00:26:48,924718 [DEBUG] last_known_wallet_txid = ('OMITTED', 'OMITTED')
00:26:48,924901 [DEBUG] check4con unconfirmed_txes = {}

Does anyone know what I'm doing wrong? Thanks in advance.

Hi, I'm having a bit of a puzzle with the title notice in the terminal output.

Clean config.cfg with 1 electrum wallet
run server.py to import addresses
run rescan script
run server.py again,

Then, "address scripthash not known to server" pops up in blocks of 26 lines. I'm a bit at a loss to get rid of this. Am I missing something?

Hello

I wanted to connect my computer with my Electrum Wallet to my computer with my Bitcoin Core and Electrum Personal Server.
I tried to specify my IP address, but could only choose between : localhost and 0.0.0.0
Being able to specify a LAN IP address would be a usefull feature imho.

Hi Chris,

Thanks so much for your contribution! I was waiting for something like this to come around.

I already managed to get my old cubox (ARM) to run a full node (0.16) and that was quite an experiment.

How much overhead does EPS add to an already working full node like mine?

Once I find some spare time I plan on trying to get EPS running on my cubox https://www.solid-run.com/nxp-family/cubox-i/

Keep up the excellent work. This spirit is what first got me into bitcoin way back in the early days. Glad to see it is still alive.

Cheers

Apologies if this is a stupid question, but how can I verify the download using your PGP key fingerprint? When I've done this in the past using gpg, I've always had a .asc file, but I can't see one for EPS...

Hi Chris,

can you give me any help on getting electrum server running on OS X?

I have electrum/trezor set up properly and I have a full bitcoin node running with 0.16.1... Just can't seem to figure out the connection and have gotten as far as " Run ./server.py "

I'm a layman at best, so be gentle :)

When starting the run-server.bat for the first time I get the following error :

'python3' is not recognized as an internal or external command,
operable program or batch file.
Press any key to continue . . .

OS = Windows 10 version 1803 build 17134.112
Python = version 3.6.5

I adjusted the run-server.bat to the following :
server.py
pause

Then it worked fine.

rescan-script.bat gives the same error with same solution

I am running Bitcoin Core 0.16.0 fully synced and Electrum personal server 0.1.3

Configured config.cfg to include my RPC user and pass, and added my master public key. As soon as I run server.py, I get this error:

2018-07-21 23:49:24,165209 [ LOG] Obtaining bitcoin addresses to monitor . . .
Traceback (most recent call last):
File "server.py", line 559, in
main()
File "server.py", line 526, in main
get_scriptpubkeys_to_monitor(rpc, config)
File "server.py", line 384, in get_scriptpubkeys_to_monitor
[transactionmonitor.ADDRESSES_LABEL]))
File "/home/bitcoin/eps/electrum-personal-server-eps-v0.1.3/electrumpersonalserver/jsonrpc.py", line 58, in call
raise JsonRpcError(response["error"])
electrumpersonalserver.jsonrpc.JsonRpcError: {'code': -32601, 'message': 'Method not found'}

Not sure if it's properly documented but the address status of an address with no history should be None.

https://github.com/kyuupichan/electrumx/blob/50e2785256ec94b340d33bd16fe17e7f7574bacb/server/session.py#L253
https://github.com/spesmilo/electrum/blob/8b1051fa24ff7f2c85b66a6012581f0287bbb827/lib/synchronizer.py#L82

e-x:

[127.0.0.1] --> ('blockchain.scripthash.subscribe', ['805c8ab4978dcf4468828e54dff2bebd3cb0d571ef69ca7c81d954b579f38209'], 11)
[127.0.0.1] <-- {'id': 11, 'jsonrpc': '2.0', 'result': None}

eps:

[127.0.0.1] --> ('blockchain.scripthash.subscribe', ['805c8ab4978dcf4468828e54dff2bebd3cb0d571ef69ca7c81d954b579f38209'], 11)
[127.0.0.1] <-- {'jsonrpc': '2.0', 'method': 'blockchain.scripthash.subscribe', 'result': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'id': 11, 'params': ['805c8ab4978dcf4468828e54dff2bebd3cb0d571ef69ca7c81d954b579f38209']}
[127.0.0.1] --> ('blockchain.scripthash.get_history', ['805c8ab4978dcf4468828e54dff2bebd3cb0d571ef69ca7c81d954b579f38209'], 12)
[127.0.0.1] <-- {'jsonrpc': '2.0', 'method': 'blockchain.scripthash.get_history', 'result': [], 'id': 12, 'params': ['805c8ab4978dcf4468828e54dff2bebd3cb0d571ef69ca7c81d954b579f38209']}
[Synchronizer] receiving history bcrt1qn7exzzhd96p044wrpywpmgcuu9mnpntsu30383 0
[Synchronizer] error: status mismatch: bcrt1qn7exzzhd96p044wrpywpmgcuu9mnpntsu30383

How many connections can eps handle? When I tried connecting with desktop it disconnected my smartphone.

I have a full node with the option disablewallet=1. I get the following error when running setup.py

Traceback (most recent call last):
  File "./server.py", line 473, in <module>
    main()
  File "./server.py", line 448, in main
    get_scriptpubkeys_to_monitor(rpc, config)
  File "./server.py", line 308, in get_scriptpubkeys_to_monitor
    [transactionmonitor.ADDRESSES_LABEL]))
  File "/home/-----/.opt/electrum-personal-server-0.1.1/electrumpersonalserver/jsonrpc.py", line 58, in call
    raise JsonRpcError(response["error"])
electrumpersonalserver.jsonrpc.JsonRpcError: {'code': -32601, 'message': 'Method not found'}

It works fine if I enable a wallet, but this is not optimal, since I do not want to use an empty wallet and this could potentially attract spammers.

Hi

When requesting headers with a start height and a count which reaches beyond the current tip, the server crashes. It should instead return the actual available number of headers that can be returned, as documented here:

https://github.com/kyuupichan/electrumx/blob/eb5c55107aa2da42365f2a93f6d8e689b92aabbe/docs/protocol-methods.rst#blockchainblockheaders

Stacktrace:

Traceback (most recent call last):
  File "./server.py", line 559, in <module>
    main()
  File "./server.py", line 556, in main
    poll_interval_listening, poll_interval_connected, certfile, keyfile)
  File "./server.py", line 364, in run_electrum_server
    txmonitor)
  File "./server.py", line 179, in handle_query
    headers_hex, n = get_block_headers_hex(rpc, start_height, count)
  File "./server.py", line 297, in get_block_headers_hex
    the_hash = rpc.call("getblockhash", [start_height])
  File "./electrumpersonalserver/jsonrpc.py", line 59, in call
    raise JsonRpcError(response["error"])
electrumpersonalserver.jsonrpc.JsonRpcError: {'code': -8, 'message': 'Block height out of range'}

There's a PR by @jonasschnelli which allows Bitcoin Core to generate wallets with private key support explicitly disabled: bitcoin/bitcoin#9662

Once that's merged, the README here should point that out.

Even better, if possible, would be for the server script to check this flag is set and refuse to run otherwise (at least for new wallets).

There's a Bitcoin Core PR that proposes to add scantxoutset which returns UTXOs for a given xpub.

This is obviously less powerful than the current watch-only approach, because you'd lose transaction history. Though perhaps Electrum itself can hold on to the history?

If bitcoind is compiled with --disable-wallet or launched with -disablewallet=1 the server throws a generic RPC error, suggesting there is a username and password problem.

When I quit Bitcoin Core server.py crashes:

    repr(exc))
electrumpersonalserver.jsonrpc.JsonRpcConnectionError: JSON-RPC connection failed. Err:ConnectionRefusedError(61, 'Connection refused')

It would be nice if instead it just turned off the Electrum server. That way the Electrum wallet would just show a red dot indicating it can't connect, which should be enough to remind me to start Bitcoin Core (though passing a more specific message to the wallet frontend would be nice too).

The other way around does work: if Bitcoin Core isn't running yet, the script starts and as soon as it can reach the RPC it boots up the server.

In eps-v0.1.4, when running against bitcoind 0.17.0rc3 eps halts before importing addresses. A master public key is set in config.cfg, but no watch-only-addresses.

$ ./server.py 
2018-09-05 12:45:53,406474 [  LOG] Obtaining bitcoin addresses to monitor . . .
Traceback (most recent call last):
  File "./server.py", line 388, in get_scriptpubkeys_to_monitor
    [transactionmonitor.ADDRESSES_LABEL]))
  File "/opt/eps/electrumpersonalserver/jsonrpc.py", line 58, in call
    raise JsonRpcError(response["error"])
electrumpersonalserver.jsonrpc.JsonRpcError: {'code': -32, 'message': 'getaddressesbyaccount is deprecated and will be removed in V0.18. To use this command, start bitcoind with -deprecatedrpc=accounts.'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./server.py", line 568, in <module>
    main()
  File "./server.py", line 535, in main
    get_scriptpubkeys_to_monitor(rpc, config)
  File "./server.py", line 393, in get_scriptpubkeys_to_monitor
    [transactionmonitor.ADDRESSES_LABEL]).keys())
  File "/opt/eps/electrumpersonalserver/jsonrpc.py", line 58, in call
    raise JsonRpcError(response["error"])
electrumpersonalserver.jsonrpc.JsonRpcError: {'code': -11, 'message': 'No addresses with label electrum-watchonly-addresses'}

It's scheduled to be removed in Bitcoin Core v0.18. This is used in get_scriptpubkeys_to_monitor.

It results in an error on bitcoind master, unless you launch with -deprecatedrpc=accounts.

I haven't checked exactly how the server uses this feature. Simply replacing it with getaddressesbylabel results in another error: No addresses with label electrum-watchonly-addresses. That new error is a feature apparently: bitcoin/bitcoin#12892 (comment) (use Incognito window if you get a Unicorn)

• getaddressesbylabel throws an error if called for a label that doesn't exist

You may need to call listlabels first to see if it exists.

Getting an error
Traceback (most recent call last):
File "server.py", line 7, in
from electrumpersonalserver.jsonrpc import JsonRpc, JsonRpcError
ImportError: No module named electrumpersonalserver.jsonrpc

When tying to run in other PC it is running fine
I have not provided any U/P
#if you dont want to use the .cookie method, uncomment to config u/p here
#rpc_user =
#rpc_password =

How to reproduce:

1. send funds to an address in electrum

EPS:

11:54:17,713776 [  LOG] Found new tx: {'fee': 3800, 'height': 0, 'tx_hash': '9a793494a83cb161d727342cf39f187bbab907440afe6c5aac17fde876d98286'}

electrum history:

{
    "summary": {
        "end_balance": "10. BTC",
        "end_date": null,
        "expenditures": "0. BTC",
        "income": "10. BTC",
        "start_balance": "0. BTC",
        "start_date": null
    },
    "transactions": [
        {
            "balance": "10. BTC",
            "confirmations": 0,
            "date": null,
            "height": -2,
            "label": "",
            "timestamp": null,
            "txid": "9a793494a83cb161d727342cf39f187bbab907440afe6c5aac17fde876d98286",
            "value": "10. BTC"
        }
    ]
}

2. use bumpfee to create a conflict (here bank is just a regtest node to distribute money. not the same one that EPS runs on top of)

bitcoin-cli:

$ bank-cli bumpfee 9a793494a83cb161d727342cf39f187bbab907440afe6c5aac17fde876d98286
{
  "txid": "2056ea4973d7d2236d086dfc89b8ece13669fa3e61b304b4f50814d4dbaf1fe7",
  "origfee": 0.00003800,
  "fee": 0.00004750,
  "errors": [
  ]
}

EPS:

11:56:07,976394 [  LOG] Found new tx: {'fee': 4750, 'height': 0, 'tx_hash': '2056ea4973d7d2236d086dfc89b8ece13669fa3e61b304b4f50814d4dbaf1fe7'}

electrum history:

{
    "summary": {
        "end_balance": "20. BTC",
        "end_date": null,
        "expenditures": "0. BTC",
        "income": "20. BTC",
        "start_balance": "0. BTC",
        "start_date": null
    },
    "transactions": [
        {
            "balance": "10. BTC",
            "confirmations": 0,
            "date": null,
            "height": -2,
            "label": "",
            "timestamp": null,
            "txid": "9a793494a83cb161d727342cf39f187bbab907440afe6c5aac17fde876d98286",
            "value": "10. BTC"
        },
        {
            "balance": "20. BTC",
            "confirmations": 0,
            "date": null,
            "height": -2,
            "label": "",
            "timestamp": null,
            "txid": "2056ea4973d7d2236d086dfc89b8ece13669fa3e61b304b4f50814d4dbaf1fe7",
            "value": "10. BTC"
        }
    ]
}

there are two unconfirmed transactions, electrum seems to think that our balance is increased by the amount in both

3. generate one block

EPS:

11:59:34,121537 [  LOG] A transaction confirmed: 9a793494a83cb161d727342cf39f187bbab907440afe6c5aac17fde876d98286
Traceback (most recent call last):
  File "p2pkh/server.py", line 290, in run_electrum_server
    recv_data = sock.recv(4096)
  File "/usr/lib/python3.5/ssl.py", line 922, in recv
    return self.read(buflen)
  File "/usr/lib/python3.5/ssl.py", line 799, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.5/ssl.py", line 585, in read
    v = self._sslobj.read(len)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "p2pkh/server.py", line 485, in <module>
    main()
  File "p2pkh/server.py", line 482, in main
    poll_interval_connected, certfile, keyfile)
  File "p2pkh/server.py", line 304, in run_electrum_server
    on_heartbeat_connected(sock, rpc, txmonitor)
  File "p2pkh/server.py", line 86, in on_heartbeat_connected
    updated_scripthashes = txmonitor.check_for_updated_txes()
  File "/home/user/software/crypto/electrum-personal-server/electrumpersonalserver/transactionmonitor.py", line 239, in check_for_updated_txes
    updated_scrhashes2 = self.check_for_confirmations()
  File "/home/user/software/crypto/electrum-personal-server/electrumpersonalserver/transactionmonitor.py", line 265, in check_for_confirmations
    block = self.rpc.call("getblockheader", [tx["blockhash"]])
KeyError: 'blockhash'

Perhaps we should be looking at the "walletconflicts" field in gettransaction, also not attempt to return proofs for unconfirmed transactions?

When starting server.py on startup with a systemd unit, the execution is handled by root and the command needs to specify the full path of both python and eps, e.g.

[Unit]
Description=Electrum Personal Server
After=bitcoind.service

[Service]
Type=simple
ExecStart=/usr/bin/python3 /home/bitcoin/electrum-personal-server/server.py

[Install]
WantedBy=multi-user.target

This gives the error message Non-existant configuration file 'config.cfg', as the script assumes that it is started from within the installation directory.

$ sudo systemctl start eps.service
$ sudo systemctl status eps.service
● eps.service - Electrum Personal Server
   Loaded: loaded (/etc/systemd/system/eps.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

Mar 30 01:21:02 RaspiBolt systemd[1]: Started Electrum Personal Server.
Mar 30 01:21:02 RaspiBolt python3[13963]: 01:21:02,712804 [  LOG] Non-existant configuration file 'config.cfg'

Is it possible to set the full path for the config.cfg on startup, or do you know of a better way to handle the automatic start with systemd on boot?

This is what I see on the terminal:

$ ./server.py
02:26:53,102823 [  LOG] Obtaining bitcoin addresses to monitor . . .
02:26:59,197439 [  LOG] Obtained list of addresses to monitor in 6.094537258148193sec
02:26:59,197553 [  LOG] Building history with 600 addresses . . .
02:26:59,422105 [  LOG] Found 8 txes. History built in 0.1979377269744873sec
02:26:59,422328 [  LOG] Starting electrum server
02:26:59,422415 [  LOG] Listening for Electrum Wallet on ('127.0.0.1', 50002)
Traceback (most recent call last):
  File "./server.py", line 309, in run_electrum_server
    sock, addr = server_sock.accept()
  File "/usr/lib64/python3.6/socket.py", line 205, in accept
    fd, addr = self._accept()
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./server.py", line 534, in <module>
    main()
  File "./server.py", line 531, in main
    poll_interval_listening, poll_interval_connected, certfile, keyfile)
  File "./server.py", line 318, in run_electrum_server
    on_heartbeat_listening(txmonitor)
  File "./server.py", line 84, in on_heartbeat_listening
    txmonitor.check_for_updated_txes()
  File "/scratch/build/bitcoin/electrum-personal-server/electrumpersonalserver/transactionmonitor.py", line 253, in check_for_updated_txes
    updated_scrhashes1 = self.check_for_new_txes()
  File "/scratch/build/bitcoin/electrum-personal-server/electrumpersonalserver/transactionmonitor.py", line 402, in check_for_new_txes
    txid_list = [(tx["txid"], tx["address"]) for tx in ret]
  File "/scratch/build/bitcoin/electrum-personal-server/electrumpersonalserver/transactionmonitor.py", line 402, in <listcomp>
    txid_list = [(tx["txid"], tx["address"]) for tx in ret]
KeyError: 'address'

In the debug log, the final two lines are:

02:27:29,451898 [DEBUG] on heartbeat listening
02:27:29,452064 [DEBUG] listtransactions tx_request_count=2

I tried looking at the code, all I could tell was sometimes the RPC call at transactionmonitor.py:395 does not give a complete response. Since I'm not familiar with the code, I did not dig further.

It was working fine until yesterday, when I added another address in the master-public-keys section. Since this is a wallet with a few transactions in it, after the first run, I used the rescan script. All seemed to go as expected with Electrum. I think I didn't notice the crashes earlier because systemd was restarting the process after every crash. I noticed it when I saw the connection was failing/succeeding intermittently (the red/green status in the lower right corner of the Electrum window).

Please let me know if I should provide other info.

If all goes well Bitcoin Core v0.17 ships with a new createwallet RPC that takes a disable_private_keys, which ensures the wallet will remain watch-only.

You could recommend that command in the README, or the script could just create a fresh wallet electrum-personal-server-wallet.dat if no wallet is explicitly configured.

The newest version of Electrum subscribes to raw headers and is not compatible with electrum-personal-server.
Here is the corresponding Electrum pull request:
https://github.com/spesmilo/electrum/pull/4402/files

blockchain.headers.subscribe with raw=False is deprecated:
https://github.com/kyuupichan/electrumx/blob/master/docs/protocol-methods.rst

Running Electrum 3.2 with electrum-personal-server results in the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/electrum/util.py", line 856, in run_with_except_hook
run_original(*args2, **kwargs2)
File "/usr/local/lib/python3.6/dist-packages/electrum/network.py", line 1040, in run
self.wait_on_sockets()
File "/usr/local/lib/python3.6/dist-packages/electrum/network.py", line 1022, in wait_on_sockets
self.process_responses(interface)
File "/usr/local/lib/python3.6/dist-packages/electrum/network.py", line 689, in process_responses
self.process_response(interface, response, callbacks)
File "/usr/local/lib/python3.6/dist-packages/electrum/network.py", line 594, in process_response
self.on_notify_header(interface, result)
File "/usr/local/lib/python3.6/dist-packages/electrum/network.py", line 1048, in on_notify_header
header_hex, height = header_dict['hex'], header_dict['height']
KeyError: 'hex'

It would be great if users could have a full node + electrum personal server running in their home, and then connect their smartphone electrum wallet to it. (Or any other computer running elsewhere, but smartphone wallet is the most obvious application)

Electrum Personal Server shouldn't have other people connecting to it, because an adversary could in theory request every address ever used on the blockchain and so figure out which wallet(s) are monitored by EPS.

Also any packet sniffers could easily decrypt the Electrum-EPS connection and spy on the wallet, although this could be remedied by having users generate their own SSL certificate instead of using the default one in the EPS repository.

This issue is about ideas for improving this situation.

Right now probably the safest way is to have the smartphone connect to the EPS computer using SSH and open a SSH tunnel, which provides encryption and authentication. Then the smartphone wallet connects through the tunnel. Unfortunately some smartphones don't have SSH tunnel apps and also this is kind of hard to set up for non-technical users.

Hi, I was following the README with a pruned node.
I added one address to the watch-only config section.
When running ./server.py for the second time, Im getting this:

[ LOG] Importing 0 wallets and 1 watch-only addresses into the Bitcoin node
[ LOG] Importing 1 addresses in total
[0%]... [10%]... [20%]... [30%]... [40%]... [50%]... [60%]... [70%]... [80%]... [90%]... Traceback (most recent call last):
File "./server.py", line 495, in
main()
File "./server.py", line 473, in main
log)
File "/home/user/electrum-personal-server/electrumpersonalserver/transactionmonitor.py", line 42, in import_addresses
rpc.call("importaddress", [a, ADDRESSES_LABEL, False])
File "/home/user/electrum-personal-server/electrumpersonalserver/jsonrpc.py", line 58, in call
raise JsonRpcError(response["error"])
electrumpersonalserver.jsonrpc.JsonRpcError: {'message': 'The wallet already contains the private key for this address or script', 'code': -4}

Is this a bug?

Hi Chris, thanks for your great work!

I have the issue that my hardware wallet produced a zpub key instead of a xpub key when I set it up. It seems that EPS/Electrum cannot read my public keys out of the zpub when I start ./rescan-script.py; it's not showing any transactions/keys. Is there a solution for this?

Thanks in advance!

I just found this old unfinished code. Dated from March 2017.

I started working on this in the context of BU and Classic in the scalability conflict.

It's meant to crawl all public Electrum servers and see which servers are on the bitcoin chain and which are on an altcoin.

import json
import Queue
import os
import pprint
import random
import socket
import threading
import time
import hashlib

from struct import pack

# Default server list from electrum client
# https://github.com/spesmilo/electrum/blob/753a28b452dca1023fbde548469c36a34555dc95/lib/network.py
DEFAULT_ELECTRUM_SERVER_LIST = [
    'electrum3.hachre.de:50001',
    'erbium1.sytes.net:50001',
    'ecdsa.net:50001',
    'electrum0.electricnewyear.net:50001',
    'VPS.hsmiths.com:50001',
    'ELECTRUM.jdubya.info:50001',
    'electrum.no-ip.org:50001',
    'us.electrum.be:50001',
    'bitcoins.sk:50001',
    'electrum.petrkr.net:50001',
    'electrum.dragonzone.net:50001',
    'Electrum.hsmiths.com:8080',
    'elec.luggs.co:80',
    'btc.smsys.me:110',
    'electrum.online:50001',
]

class ElectrumConn(threading.Thread):

    def __init__(self, electrum_server):
        threading.Thread.__init__(self)
        self.daemon = True
        self.msg_id = 0
        self.RetQueue = Queue.Queue()
        self.running = True
        try:
            hostport = (electrum_server.split(':')[0], int(electrum_server.split(':')[1]))
            self.s = socket.create_connection(hostport)
            print('connected')
        except Exception as e:
            print("Error connecting to electrum server. Try again to connect to a random server or set a server in the config.")
            os._exit(1)

    def run(self):
        self.ping()
        while self.running:
            all_data = None
            while True:
                data = self.s.recv(1024)
                if data is None or len(data) == 0:
                    print('EOF')
                    break
                if all_data is None:
                    all_data = data
                else:
                    all_data = all_data + data
                if '\n' in all_data:
                    #print('data = ' + all_data[:-1].decode())
                    data_json = json.loads(all_data[:-1].decode())
                    self.RetQueue.put(data_json)
                    break

    def close(self):
        self.running = False

    def ping(self):
        #print('sending server ping')
        self.send_json({'id':0,'method':'server.version','params':[]})
        if self.running:
            t = threading.Timer(60, self.ping)
            t.daemon = True
            t.start()

    def send_json(self, json_data):
        data = json.dumps(json_data).encode()
        self.s.send(data + b'\n')

    def call_server_method(self, method, params=[]):
        self.msg_id = self.msg_id + 1
        current_id = self.msg_id
        method_dict = {
            'id': current_id,
            'method': method,
            'params': params
        }
        self.send_json(method_dict)
        while True:
            ret_data = self.RetQueue.get()
            if ret_data.get('id', None) == current_id:
                return ret_data
            else:
                print('nonid = ' + json.dumps(ret_data))


con = ElectrumConn(DEFAULT_ELECTRUM_SERVER_LIST[0])
con.start()
time.sleep(2)


banner = con.call_server_method('server.banner', [])
banner_lines = banner['result'].split('\n')
for l in banner_lines:
    print(l)

'''
sh = 517241
for h in range(sh, sh+5):
    ret = con.call_server_method("blockchain.block.get_header", [h])
    print(ret)
'''

'''
numblocks = con.call_server_method('blockchain.numblocks.subscribe')
print(numblocks)
prev_blocks = 10
for height in range(numblocks['result'], numblocks['result']-prev_blocks, -1):

    header_ret = con.call_server_method('blockchain.block.get_header', [height])
    header = header_ret['result']
    header_bytes = pack('<i32s32sIII', header['version'], header['prev_block_hash'].decode('hex')[::-1], header['merkle_root'].decode('hex')[::-1], header['timestamp'], header['bits'], header['nonce'])
    blockhash = hashlib.sha256(hashlib.sha256(header_bytes).digest()).digest()
    #print(str(height) + ' blockhash = ' + blockhash[::-1].encode('hex'))

    print 'hash = ' + blockhash[::-1].encode('hex') + ' priv=' + header['prev_block_hash'] + '\n\theader=' + header_bytes.encode('hex')
'''


servers = con.call_server_method('server.peers.subscribe', [])
for server in servers['result']:
    address, name, ver_limit_protocols = server
    print(name + ' ' + address + ' => ' + str(ver_limit_protocols))

'''
u'result': [[u'108.44.238.26',
              u'b.1209k.com',
              [u'v1.0', u'p10000', u's50002', u't50001']],
             [u'shsmithsj5cjphxs.onion',
              u'shsmithsj5cjphxs.onion',
              [u'v1.0', u's51002', u't51001']],
             [u'89.163.243.185',
              u'ELEX02.blackpole.online',
              [u'v1.0', u's51002', u't51001']],
             [u'109.199.70.182', u'electrum.noip.pl', [u'v1.0', u's50002']],
             [u'178.63.19.124',
              u'electrum.rofl.cat',
              [u'v1.0', u'p10000', u's50002', u't50001']],
             [u'91.63.228.195',
              u'electrum3.hachre.de',
              [u'v1.0', u's50002', u't50001']],
             [u'94.177.253.78',
              u'beezling.bauerj.eu',
              [u'v1.0', u's50002', u't50001']],
             [u'yzgq5m227nhsi33j.onion',
              u'yzgq5m227nhsi33j.onion',
              [u'v1.0', u't50001']],
             [u'wawh3jbzstiovlyo.onion',
              u'wawh3jbzstiovlyo.onion',
              [u'v1.0', u's50002', u't50001']],
             [u'62.210.6.26',
              u'electrum.anduck.net',
              [u'v1.0', u'p100', u's50002', u't50001']],
             [u'3smoooajg7qqac2y.onion',
              u'3smoooajg7qqac2y.onion',
              [u'v1.0', u's50002', u't50001']],
             [u'46.4.125.2', u'46.4.125.2', [u'v1.0', u't50001']],
             [u'188.122.91.11', u'elec.luggs.co', [u'v1.0', u's443']],
             [u'213.239.201.137',
              u'213.239.201.137',
              [u'v1.0', u's50002', u't50001']],
'''

print('ending')

I am trying to run EPS on an Odroid HC2 running a fully synced Bitcoin 0.16 full node. I have made the necessary configurations in the config.cfg file i.e. providing my master public keys and RPC credentials that are used in bitcoin.conf. However, when I try to launch the server using ./server.py I get the following exception:

Traceback (most recent call last):
  File "./server.py", line 473, in <module>
    main()
  File "./server.py", line 448, in main
    get_scriptpubkeys_to_monitor(rpc, config)
  File "./server.py", line 308, in get_scriptpubkeys_to_monitor
    [transactionmonitor.ADDRESSES_LABEL]))
  File "/home/bitcoin/electrum-personal-server/electrumpersonalserver/jsonrpc.py", line 58, in call
    raise JsonRpcError(response["error"])
electrumpersonalserver.jsonrpc.JsonRpcError: {'code': -32601, 'message': 'Method not found'}

Any idea what I could be missing? I know that my RPC credentials are correct as I am able to connect my Samourai wallet on my phone to the same full node.