Coder Social home page Coder Social logo

mempoolco / spruned Goto Github PK

View Code? Open in Web Editor NEW
171.0 17.0 35.0 1.14 MB

A Bitcoin-without-Blockchain client w/ RPC that can fetch any block or transaction

License: MIT License

Python 99.50% Shell 0.18% Dockerfile 0.33%
bitcoin blockchain bitcoin-api bitcoind

spruned's Introduction

spruned

A Bitcoin-without-Blockchain client w/ RPC that can fetch any block or transaction


travis coveralls rtd PyPI version PyPI version

Interested in spruned development ? Join the telegram group

What's this?

spruned is a bitcoin client for lightweight systems.
it doesn't need to store the blockchain on the disk, just the headers.
128mb ram & 500mb hdd should be fairly enough to keep it up & running.

Supports both Bitcoin Mainnet and Testnet

The RPC API is able to support:

  • c-lightning
  • LND (zeromq must be enabled)
  • btc-rpc-explorer
  • ...many more, open an issue if you need other API calls.

Dive into spruned by reading the documentation

How it works?

spruned downloads and store the bitcoin blocks on demand, when you need them, directly from the peer to peer bitcoin network.

there's a "bootstrap" functionality, to keep the last ~50 (default settings) blocks on the local storage, because fetch blocks may require also up to 10 seconds with slow connections, and this "bootstrap mode" reduces latencies on usage.

You can use bitcoin-cli, or any other RPC client, as if you had bitcoind up & running.

For the transactions related APIs and utxo tracking, spruned uses the electrum network.

Dependencies

spruned works with Python >= 3.5.2. Right now it should work only on Linux systems.

It make intensive usage of connectrum, pybitcointools and pycoinnet libraries. Thanks to mantainers & contributors!
Especially at this stage of development (but it would be better always), it is recommended to use virtualenv to run spruned.

Installation from pypi

$ cd ~/src
$ sudo apt-get install libleveldb-dev python3-dev virtualenv gcc g++
$ virtualenv -p python3.5 venv
$ . venv/bin/activate
$ pip install spruned

Installation from github

$ cd ~/src
$ sudo apt-get install libleveldb-dev python3-dev git virtualenv gcc g++
$ git clone https://github.com/gdassori/spruned.git
$ cd spruned
$ virtualenv -p python3.6 venv
$ . venv/bin/activate
$ pip install -r requirements.txt
$ python setup.py install

Usage

spruned options:

$ ~/src/spruned/venv/bin/spruned

usage: spruned    [-h] [--rpcuser RPCUSER] [--rpcpassword RPCPASSWORD]
                  [--rpcport RPCPORT] [--rpcbind RPCBIND] [--datadir DATADIR]
                  [--daemon] [--keep-blocks KEEP_BLOCKS]
                  [--network {bitcoin.mainnet,bitcoin.testnet}] [--debug]
                  [--cache-size CACHE_SIZE] [--proxy PROXY] [--tor]
                  [--no-dns-seeds] [--add-p2p-peer ADD_P2P_PEER]
                  [--max-p2p-connections MAX_P2P_CONNECTIONS]
                  [--add-electrum-server ELECTRUM_SERVER]
                  [--max-electrum-connections MAX_ELECTRUM_CONNECTIONS]
                  [--disable-p2p-peer-discovery]
                  [--disable-electrum-peer-discovery]
                  [--zmqpubhashblock ZMQPUBHASHBLOCK]
                  [--zmqpubrawtx ZMQPUBRAWTX] [--zmqpubhashtx ZMQPUBHASHTX]
                  [--zmqpubrawblock ZMQPUBRAWBLOCK]
                  [--mempool-size MEMPOOL_SIZE]

A Bitcoin Lightweight Client

optional arguments:
  -h, --help            show this help message and exit
  --rpcuser RPCUSER     Username for JSON-RPC connections (default: rpcuser)
  --rpcpassword RPCPASSWORD
                        Password for JSON-RPC connections (default: passw0rd)
  --rpcport RPCPORT     Listen for JSON-RPC connections on <port> (default:
                        8332 or testnet: 18332) (default: None)
  --rpcbind RPCBIND     Bind to given address to listen for JSON-RPC
                        connections. (default: 127.0.0.1)
  --datadir DATADIR     Specify data directory (default: /home/guido/.spruned)
  --daemon              Run in the background as a daemon and accept commands
                        (default: False)
  --keep-blocks KEEP_BLOCKS
  --network {bitcoin.mainnet,bitcoin.testnet}
  --debug               Enable debug mode (default: False)
  --cache-size CACHE_SIZE
                        Cache size (in megabytes) (default: 50)
  --proxy PROXY         Proxy server (hostname:port) (default: None)
  --tor                 Connect only to hidden services. Use proxy on
                        localhost:9050, if nothing else is provided with
                        --proxy (default: False)
  --no-dns-seeds        Disable DNS seeds for P2P peers discovery (default:
                        False)
  --add-p2p-peer ADD_P2P_PEER
                        Add a P2P peer (default: None)
  --max-p2p-connections MAX_P2P_CONNECTIONS
                        How many P2P peers to connect (default: None)
  --add-electrum-server ELECTRUM_SERVER
                        Add an Electrum server (default: None)
  --max-electrum-connections MAX_ELECTRUM_CONNECTIONS
                        How many Electrum servers to connect (default: None)
  --disable-p2p-peer-discovery
                        Control P2P peers discovery (getaddr) (default: False)
  --disable-electrum-peer-discovery
                        Control electrum peers discovery (peer subscribe)
                        (default: False)
  --zmqpubhashblock ZMQPUBHASHBLOCK
                        Enable publish hash block in <address> (default: )
  --zmqpubrawtx ZMQPUBRAWTX
                        Enable publish raw transaction in <address> (default:
                        )
  --zmqpubhashtx ZMQPUBHASHTX
                        Enable publish hash transaction in <address> (default:
                        )
  --zmqpubrawblock ZMQPUBRAWBLOCK
                        Enable publish raw block in <address> (default: )
  --mempool-size MEMPOOL_SIZE
                        Set the mempool size in megabytes (0 = mempool
                        disabled, default) - VERY experimental (default: 0)

And, once you run spruned:

$ tail -f ~/.spruned/spruned.log # to see what's going on!

You'll see it will took about 15 minutes to sync block headers (up to 2 hours on a raspberry pi zero, I'm working on this.)

In sync ? Ok, see the list of available commands:

bitcoin-cli help

== Blockchain ==
getbestblockhash
getblock "blockhash" ( verbosity )
getblockchaininfo
getblockcount
getblockhash height
getblockheader "hash" ( verbose )
gettxout "txid" n ( include_mempool )
getmempoolinfo
getrawmempool

== Rawtransactions ==
getrawtransaction "txid" ( verbose )
sendrawtransaction "hexstring" ( allowhighfees )

== Util ==
estimatefee nblocks
estimatesmartfee conf_target ("estimate_mode")
uptime

== Network ==
getpeerinfo
getnetworkinfo

== Wallet ==
validateaddress

== Partially emulated for compatibility ==
getchaintxstats
getmininginfo
getnettotals

or check the status*:

$ bitcoin-cli getblockchaininfo
{
  "mediantime": 1523387051,
  "blocks": 517579,
  "headers": 517579,
  "verificationprogress": 100,
  "chain": "main",
  "chainwork": null,
  "difficulty": null,
  "bestblockhash": "00000000000000000018e502dec1f93d32521674019a45d7d095cbd390279dff",
  "warning": "spruned 0.0.1a7, emulating bitcoind v0.16",
  "pruned": false
}

Download a block:

$ bitcoin-cli getblock `bitcoin-cli getblockhash 1`
{
  "bits": 486604799,
  "mediantime": 1231469665,
  "nextblockhash": "000000006a625f06636b8bb6ac7b960a8d03705d1ace08b1a19da3fdcc99ddbd",
  "tx": [
    "0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098"
  ],
  "previousblockhash": "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f",
  "version": 1,
  "chainwork": "Not Implemented Yet",
  "nonce": 2573394689,
  "time": 1231469665,
  "height": 1,
  "hash": "00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048",
  "versionHex": "Not Implemented Yet",
  "merkleroot": "0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098",
  "difficulty": "Not Implemented Yet"
}

Or a transaction:

$ bitcoin-cli getrawtransaction 0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098
01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff0704ffff
001d0104ffffffff0100f2052a0100000043410496b538e853519c726a2c91e61ec11600ae1390813a627c66fb
8be7947be63c52da7589379515d4e0a604f8141781e62294721166bf621e73a82cbf2342c858eeac00000000

Or in verbose mode:

$ bitcoin-cli getrawtransaction 0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098 1
{
  "txid": "0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098",
  "hash": "0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098",
  "version": 1,
  "size": 134,
  "vsize": 134,
  "locktime": 0,
  "vin": [
    {
      "coinbase": "04ffff001d0104",
      "sequence": 4294967295
    }
  ],
  "vout": [
    {
      "value": 50.0,
      "n": 0,
      "scriptPubKey": {
        "asm": "0496b538e853519c726a2c91e61ec11600ae1390813a627c66fb8be7947be63c52da7589379515d4e0a604f8141781e62294721166bf621e73a82cbf2342c858ee OP_CHECKSIG",
        "hex": "410496b538e853519c726a2c91e61ec11600ae1390813a627c66fb8be7947be63c52da7589379515d4e0a604f8141781e62294721166bf621e73a82cbf2342c858eeac",
        "reqSigs": 1,
        "type": "pubkey",
        "addresses": [
          "12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX"
        ]
      }
    }
  ],
  "hex": "01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff0704ffff001d0104ffffffff0100f2052a0100000043410496b538e853519c726a2c91e61ec11600ae1390813a627c66fb8be7947be63c52da7589379515d4e0a604f8141781e62294721166bf621e73a82cbf2342c858eeac00000000",
  "blockhash": "00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048",
  "confirmations": 522493,
  "time": 1231469665,
  "blocktime": 1231469665
}

And, eventually, broadcast one:

$ bitcoin-cli sendrawtransaction 01000000011cee4c0dd7f1a90ae80311c414d48f3a16596e9ea08fa3edfb793734e2b2a100010000006a47304402205a665616085b4f425cccfde5be2113258f3c104c2c53ef918866ada8f02f7caf0220458bdbc220a3f1017b65d9138e5121a9c63decc89550a2e64e914013d26cb93b0121029643906e277eae677134d40356dfb575a2dfbe09a18a1fd7fadfd853715a7242ffffffff0234e3e600000000001976a91410a71790c6bbc2694c74b6fee9a449a11f74123388ac444c5501000000001976a9148c9e0a9029bbce075e2b5aae90010905aa4c64b188ac00000000
489feae0e317b9255031710eadc238bb1ba3009fff0e86b303b0963e34a332b0

* bitcoin-cli is not included

Emulated APIs as in bitcoind 0.16:

== Blockchain ==
- getbestblockhash
- getblock "blockhash" ( verbosity )
- getblockchaininfo
- getblockcount
- getblockhash height
- getblockheader "hash" ( verbose )

- getmempoolinfo [ may be disabled, see help, --mempoolsize ]
- getrawmempool [ may be disabled, see help, --mempoolsize ]

- gettxout "txid" n ( include_mempool )
- getrawtransaction "txid" ( verbose )
- sendrawtransaction "hexstring" ( allowhighfees )

- estimatefee nblocks
- estimatesmartfee conf_target ("estimate_mode")

- uptime
- getpeerinfo
- getnetworkinfo
- validateaddress
- getchaintxstats
- getmininginfo
- getnettotals

Requirements

  • An internet connection
  • less than 500mb of disk space :-)
  • Python >= 3.5.2

Limitations

  • Not fast as a full node: internet download is slower than a read from disk.
  • Doesn't help to keep the network healthy (I care about that, and this is really going to be addressed).
  • Privacy leaky if not behind Tor

Roadmap

  • Maintenance web ui
  • Wallet APIs (electrum-servers based utxo management)
  • sPRUNED P2P network layer with cached blocks to offload the P2P network
  • Distribution
  • Network bandwidth reduction

Support sPRUNED!

  • Donate BTC to 3FVGopUDc6tyAP6t4P8f3GkYTJ5JD5tPwV
  • Show your love. FOSS development is free, but time is scarce.

Partecipate

  • spruned is a unique client, and could be adapted to any cryptocurrency. Contact me on Twitter for requests.
  • Run Electrum servers and full nodes.
  • Write extensive spruned documentation and guides.
  • Help improving spruned support to third-party applications. (ask missing APIs).
  • Report about your usage!

spruned's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

spruned's Issues

emulate gettxout api

unfortunately electrum doesn't provide a wrapper to gettxout

emulate from public services.

use the p2p net to sendrawtransaction

sendrawtransaction is easy with electrod, but since we have p2p support, is worth to give it a try and use it to broadcast our own txs via sendrawtransaction without using the electrum network.

So we have to:

  • advertise TX InvItems with the new txhash
  • hook the GetData response to the db: after advertising the new txhash, target peers will ask us with a 'getdata' call the related TX Inv. We're not ready to respond to this request yet.

electrod: cryptoverify transactions

Merkle tree validation for transactions

Handle 2 cases:

  • Do I have the tx in storage? It's verified.
  • No? Obtain transaction. Ask a remote merkle proof from the electrum network and validate with the local headers.

(We don't have yet inclusion blocks for transactions, need PR kyuupichan/electrumx#397 to spread across servers to avoid too much errors, maybe Q2 2018).

Build fails for pyvel dependency (bump dependency version?)

Installing on Ubuntu 18.04 with Python 3.6, pip install fails for pyvel with the current required version of 0.9.0.

Changed required version to 1.0.4 and everything seems to be working fine so far.

Error output from 0.9.0:

  Failed building wheel for plyvel
  Running setup.py clean for plyvel
Failed to build plyvel
Installing collected packages: plyvel, daemonize, chardet, attrs, multidict, idna, idna-ssl, yarl, aiohttp, pycoin
  Running setup.py install for plyvel ... error
    Complete output from command /home/btcd/spruned/venv/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-f04a3fxe/plyvel/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-n34nxc1d/install-record.txt --single-version-externally-managed --compile --install-headers /home/btcd/spruned/venv/include/site/python3.6/plyvel:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/plyvel
    copying plyvel/_version.py -> build/lib.linux-x86_64-3.6/plyvel
    copying plyvel/__init__.py -> build/lib.linux-x86_64-3.6/plyvel
    running build_ext
    building 'plyvel._plyvel' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/plyvel
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fdebug-prefix-map=/build/python3.6-EKG1lX/python3.6-3.6.5=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -I/home/btcd/spruned/venv/include/python3.6m -c plyvel/_plyvel.cpp -o build/temp.linux-x86_64-3.6/plyvel/_plyvel.o -Wall -g
    x86_64-linux-gnu-gcc: error: /usr/share/dpkg/no-pie-compile.specs: No such file or directory
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/home/btcd/spruned/venv/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-f04a3fxe/plyvel/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-n34nxc1d/install-record.txt --single-version-externally-managed --compile --install-headers /home/btcd/spruned/venv/include/site/python3.6/plyvel" failed with error code 1 in /tmp/pip-install-f04a3fxe/plyvel/

support multiple versions emulation (now only 0.16 is supported)

  • implement jsonrpc validators with pycomb
  • create the serialization module, organized by emulation versions
  • create version-dependant output serializers
  • generalize the vo_service input\output
  • add in settings.py the emulated version
  • automount the serializer with a decorator

Note:
We always save segwit blocks. It MUST be in charge to the serializer to format outputs for the proper version (this is about <= 0.12 emulation, I guess, I don't think is worth to go backward).

pip: broken non-pip dependencies

I've spent a bit on setuptools to understand how to proper install github dependencies.

Looks like pip ignore external links, also if --process-dependency-links is provided, when the packages are homonyms of others on pip official repo, as it's in spruned (forked and customized pybitcointools, connectrum, pycoin, pycoinnet).

To enable the pip installation again, a method to install third party dependencies have to be found.

It MAY be some sort of "spruned --install-github-requirements" that must be executed before the first run.

Also, the packages version MUST be checked when running spruned and a proper suggestion must be returned, to avoid incomprehensible exceptions on the topic (The user must be warned to run --install-github-requirements before going further).

Other suggestions are welcome.

Support for testnet

Hi! Great work here!
One of the main usage I see for this project obviously : Continuous Integration testing !
And so come : Bitcoin Tesnet

Would be glad to help in few weeks maybe, stay tuned and keep up the good work!

Output validation

Validate the output and use specific services as described in #9 to obtain data when there's no choice.

Add prefetch neighbors caching policy

For blocks:

  • When an height is requested prefetch neighbors

For transactions

  • When a transaction is requested, prefetch the related transactions (by ancients, spents, addresses). specify the depth in any available direction.

Add param:
--prefetch-neighbors-blocks=
--prefetch-neighbors-txs=

This may introduce asyncio.

Add Electrum Servers as sources

Electrum is Python, and looks like their APIs have some sort of "getrawtransaction".
Evaluate how time consuming is to integrate the Electrum network as a source for, at least, raw transactions. I don't think blocks may be fetched, instead.

logging: remote_address and local_address incorrect?

Hi, my public IP is X.X.X.X in the log below.

So the logfile is always giving X.X.X.X/ as the remote_address, and 0:0:0:0:0:0:0:0/0 as the local_address.

I think that local_address should be X.X.X.X/<random port> and remote_address should be the IP/port of internet peers that spruned is connected to. As it is, I see no way to find addresses of connected peers.

2018-07-25 03:51:57,390 - p2p - INFO - Connected to peer {'timestamp': 1532515917, 'subversion': b'/Satoshi:0.15.1/', 'last_block_index': 533586, 'remote_address': X.X.X.X/49340, 'nonce': 17134752964282426299, 'relay': True, 'version': 70015, 'services': 13, 'local_address': 0:0:0:0:0:0:0:0/0}
2018-07-25 03:51:59,345 - p2p - INFO - Connected to peer {'timestamp': 1532515918, 'subversion': b'/Satoshi:0.16.1/', 'last_block_index': 533586, 'remote_address': X.X.X.X/40804, 'nonce': 7993688548404011381, 'relay': True, 'version': 70015, 'services': 1037, 'local_address': 0:0:0:0:0:0:0:0/0}
2018-07-25 03:52:07,598 - p2p - INFO - Connected to peer {'timestamp': 1532515927, 'subversion': b'/Satoshi:0.16.1/', 'last_block_index': 533586, 'remote_address': X.X.X.X/36696, 'nonce': 8390064861232150806, 'relay': True, 'version': 70015, 'services': 1037, 'local_address': 0:0:0:0:0:0:0:0/0}

Source: Add fallback services and policies

Services like Blocktrail API may provide block transactions, but a bit slowly. cause a large amount of data is returned, and not only txids.

Need some sort of "fallback services" to pick data from, if it's missing from the final artifact.
But this reduces the already low entropy of the sources, and should be done only for big queries (like blocks, and not rawtransactions).

To enable a choice, we may introduce "fullmode" and "fastmode" after #5 is done.

In "full mode" fallback services may be used as everything else, in "fast mode" they're used only when data is missing, and this introduce the need of final artifact validation.

Just public brainstorming.

Errors returned as results

While trying to get c-lightning set up with spruned I noticed that it kept dying once it catches up with the blockchain head. The problem is that for blockheights that are above the current blockchain length the following is returned:

{"result": {"error": {"code": -8, "message": "Block height out of range"}}, "jsonrpc": "2.0", "error": null, "id": 1}

This is an error wrapped inside of a non-error return, which throws c-lightning off, and is likely to result in errors in other programs as well.

For comparison, bitcoind will return the following in the same situation:

{"result":null,"error":{"code":-8,"message":"Block height out of range"},"id":1}

Excellent work otherwise, been using spruned for a while now 👍

Handle network outages

When there's no network for a while, spruned eventually dies, due blacklist on all servers.

Not synced, retrying fallback headers

I try to set up a sPRUNED node in a DigitalOceans droplet (Ubuntu 16.04.4 x64) and the installation went smoothly. Once I start the application, however, it does not seem to find the network.

$ ~/src/spruned/venv/bin/spruned --daemon --debug
$ tail ~/.spruned/bitcoin.testnet/bitcoin.testnet/spruned.log 
2018-07-14 20:30:33,094 - root - DEBUG - Running spruned daemon
2018-07-14 20:30:33,096 - root - WARNING - Starting daemon.
2018-07-14 20:30:33,097 - root - DEBUG - Arguments: Namespace(cache_size=50, daemonize=True, datadir='/home/bitcoin/.spruned/bitcoin.testnet', debug=True, keep_blocks=200, network=None, rpcbind='0.0.0.0', rpcpassword='..................', rpcport=18332, rpcuser='bitcoin')
2018-07-14 20:30:33,099 - leveldb - DEBUG - Ensuring integrity of the storage, and tracking missing items
2018-07-14 20:30:33,099 - leveldb - DEBUG - Ensuring no stales in blockchain repository
2018-07-14 20:30:33,104 - cache - DEBUG - Loaded index
2018-07-14 20:30:33,104 - cache - DEBUG - Empty index found
2018-07-14 20:30:33,105 - cache - INFO -
Purged from storage 0 elements not tracked by cache.
Total tracked: 0
Total protected: 0,
Total cached: 0,
Total entries: 0,
Total transactions: 0

2018-07-14 20:30:33,105 - leveldb - DEBUG - Checking cache limits
2018-07-14 20:30:33,106 - cache - DEBUG - Stales purge done, removed 0 items from index
2018-07-14 20:30:33,107 - cache - INFO - Cache is ok, size: 0, limit: 50
2018-07-14 20:30:33,107 - cache - DEBUG - Stales purge done, removed 0 items from index
2018-07-14 20:30:33,107 - cache - INFO - Cache is ok, size: 0, limit: 50
2018-07-14 20:30:33,134 - electrum - DEBUG - Checking internet connectivity
2018-07-14 20:30:33,305 - electrum - DEBUG - ConnectionPool: connect, needed: 1
2018-07-14 20:30:33,306 - electrum - DEBUG - Connecting to peers (1)
2018-07-14 20:30:33,307 - electrum - DEBUG - Created client instance: testnet.hsmiths.com
2018-07-14 20:30:33,308 - spruned.dependencies.connectrum - DEBUG - Connecting to: <ServerInfo testnet.hsmiths.com nick=bade5985a4fbd742 ports="['s', 't']" v=1.2 prune=0>
2018-07-14 20:30:33,493 - spruned.dependencies.connectrum - DEBUG - Transport connected ok
2018-07-14 20:30:33,833 - spruned.dependencies.connectrum - DEBUG - Connected to: <ServerInfo testnet.hsmiths.com nick=bade5985a4fbd742 ports="['s', 't']" v=1.2 prune=0>
2018-07-14 20:30:33,834 - electrum - DEBUG - Connected to testnet.hsmiths.com
2018-07-14 20:30:34,011 - spruned.dependencies.connectrum - INFO - Error response: '{'code': 2, 'message': "daemon error: {'code': -5, 'message': 'No such mempool or blockchain transaction. Use gettransaction for wallet transactions.'}"}'
2018-07-14 20:30:34,012 - electrum - DEBUG - Exception connecting to testnet.hsmiths.com ('blockhash')
2018-07-14 20:30:34,012 - electrum - DEBUG - Checking internet connectivity
2018-07-14 20:30:39,117 - spruned.dependencies.connectrum - DEBUG - Server version: ['ElectrumX 1.5.1', '1.2']
2018-07-14 20:30:39,120 - electrum - DEBUG - Fallback headers check: Not synced yet or sync locked (False), retrying fallback headers check in 30s
2018-07-14 20:31:09,131 - electrum - DEBUG - Fallback headers check: Not synced yet or sync locked (False), retrying fallback headers check in 30s
2018-07-14 20:31:39,139 - electrum - DEBUG - Fallback headers check: Not synced yet or sync locked (False), retrying fallback headers check in 30s
2018-07-14 20:32:09,150 - electrum - DEBUG - Fallback headers check: Not synced yet or sync locked (False), retrying fallback headers check in 30s
2018-07-14 20:32:39,151 - electrum - DEBUG - Fallback headers check: Not synced yet or sync locked (False), retrying fallback headers check in 30s
2018-07-14 20:32:39,297 - spruned.dependencies.connectrum - DEBUG - Server version: ['ElectrumX 1.5.1', '1.2']
2018-07-14 20:33:09,159 - electrum - DEBUG - Fallback headers check: Not synced yet or sync locked (False), retrying fallback headers check in 30s
2018-07-14 20:33:39,170 - electrum - DEBUG - Fallback headers check: Not synced yet or sync locked (False), retrying fallback headers check in 30s

I have a firewall set up, but only blocking incoming transactions. Outgoing, everything is open. Are there any network settings I need to configure. Or is it something completely different?

Thanks in advance for your support!

Evaluate to make sPRUNED also a serving peer for downloaded blocks data [spruned-p2p]

<brainstorming>

a) The 'getblock' API have to return, 
     basically, block headers + txids + block metas (size, hash, etc.)

b) Blocks downloaded from the P2P network are full blocks, 
     and this must be the last option, 
     since is very time\bandwidth consuming.

c) Download block from public services is annoying
     for several reasons (simply put: a "leech only" application is bad)

d) Electrum network doesn't have knowledge of blocks at this level.

e) There isn't out there a P2P network of sources, 
    excluding the block explorers, which return the data in point _a)_

f) sPRUNED may be a source for other sPRUNED instances, on this data.

g) ...gnutella or similar ?

</brainstorming>

support Tor

It must be tested that spruned works well behind Tor.

electrod: Put the reactor in charge to set a score on Peers.

  • Create a Peer Interface Class.
  • Incapsulate Peer and responses together in a single object, ElectrodResponse.
  • Extend the Peer Class with methods to open\close\set a score on a Peer after parsing the Peer response.
  • Use this methods from the Reator.
  • Handle Peer Score in the Peers ConnectionPool (Interface) to disconnect \ blacklist Peers.

Handle network outages

On the new ConnectionsPool, when an outage occurs for a bit, finally all the servers are banned, and there's no way for the pool to recover.

a) Expires bans after N, tracking errors timestamps.
b) Create a new special state for network outages and suspend some operations like the fallback check_headers.
c) Detect network outages and avoid to ban servers during them.

c-lightning: getblockhash 532924 exited with status 5

I run sPRUNED together with c-lightning and run into the issue that c-lightning requests a block that is not yet known, sPRUNED anwers with an error and this is not what c-lightning expects.

Is there something I am missing? Or is this something that could be fixed in sPRUNED, or unexpected behaviour by c-lightning?

bitcoin ~/.lightning ฿ tail lightning.log -f
2018-07-21T08:19:27.790Z lightningd(19444): Server started with public key 0283126f4322e67b4537674ae430b5eec1d53f4725ae578b25492296deda48eccf, alias Stadicus (color #028312) and lightningd v0.6
2018-07-21T08:19:29.213Z lightningd(19444): Adding block 532909: 0000000000000000001cf197ed5d33e83ed8ecbc44cce53159ab5867b4ae150f
2018-07-21T08:19:29.370Z lightningd(19444): No fee estimate for Immediate: basing on default fee rate
2018-07-21T08:19:29.370Z lightningd(19444): Immediate feerate 2274 (was 0)
2018-07-21T08:19:29.370Z lightningd(19444): No fee estimate for Normal: basing on Immediate rate
2018-07-21T08:19:29.370Z lightningd(19444): Normal feerate 1261 (was 0)
2018-07-21T08:19:29.370Z lightningd(19444): No fee estimate for Slow: basing on Immediate rate
2018-07-21T08:19:29.370Z lightningd(19444): No fee estimate for Slow: basing on Normal rate
2018-07-21T08:19:29.370Z lightningd(19444): Slow feerate 253 (was 0)
2018-07-21T08:19:29.505Z lightningd(19444): Adding block 532910: 00000000000000000013e99ce1fa83ec4bf2307454d3689b0cd2f700b463010f
2018-07-21T08:19:30.505Z lightningd(19444): Adding block 532911: 00000000000000000002829d2b02fde402f14e001ae27036c41eff06966197a7
2018-07-21T08:19:31.415Z lightningd(19444): Adding block 532912: 000000000000000000137420a9612a5282befeaff773c4e6c1306e47b922806c
2018-07-21T08:19:31.585Z lightningd(19444): Adding block 532913: 0000000000000000001398a986bb38770a50ee0b63d5f082f63ca547688c7388
2018-07-21T08:19:32.181Z lightningd(19444): Adding block 532914: 0000000000000000001f2deba673d0bb65ac20a3690240324b0bb6bfe9286e23
2018-07-21T08:19:33.695Z lightningd(19444): Adding block 532915: 0000000000000000000ca2e1e68383ac8c94445d641dbb623f5103fc4d054cc6
2018-07-21T08:19:34.300Z lightningd(19444): Adding block 532916: 00000000000000000028b958250802a46feb931b97b1a938c016ec9b56fdd65b
2018-07-21T08:19:36.670Z lightningd(19444): Adding block 532917: 00000000000000000012b5df76ccbe6087f1645deeae639888248dff094b35a6
2018-07-21T08:19:38.974Z lightningd(19444): Adding block 532918: 0000000000000000000bb429cbf751b8ef12e2ae624b8e7a22a1dcf469e2466b
2018-07-21T08:19:41.371Z lightningd(19444): Adding block 532919: 0000000000000000002aaccac3fed5b9b010af2f48690b196a05c3618af6b18e
2018-07-21T08:19:45.792Z lightningd(19444): Adding block 532920: 000000000000000000075a60644792ce3bc800353cff0fd990f6b7dad1cab9f8
2018-07-21T08:19:47.226Z lightningd(19444): Adding block 532921: 000000000000000000013e4e5f0fd21d768c3d591a331cbba322ae1e0e4d621a
2018-07-21T08:19:58.595Z lightningd(19444): Adding block 532922: 0000000000000000002d498b00da4f1ece0322d3d33bd621cbe02651ef02073b
2018-07-21T08:20:00.286Z lightningd(19444): Adding block 532923: 000000000000000000320bb059c9271f73c12553872e721c8d1749556c7b15f1
2018-07-21T08:20:00.376Z lightningd(19444): bitcoin-cli -rpcconnect=localhost -rpcport=8332  -rpcuser=XXX -rpcpassword=XXX getblockhash 532924 exited with status 5
2018-07-21T08:20:01.384Z lightningd(19444): bitcoin-cli -rpcconnect=localhost -rpcport=8332  -rpcuser=XXX -rpcpassword=XXX getblockhash 532924 exited with status 5
2018-07-21T08:20:02.394Z lightningd(19444): bitcoin-cli -rpcconnect=localhost -rpcport=8332  -rpcuser=XXX -rpcpassword=XXX getblockhash 532924 exited with status 5
2018-07-21T08:20:03.402Z lightningd(19444): bitcoin-cli -rpcconnect=localhost -rpcport=8332  -rpcuser=XXX -rpcpassword=XXX getblockhash 532924 exited with status 5

bitcoin ~/.lightning ฿ bitcoin-cli getblockchaininfo
{
  "chainwork": null,
  "bestblockhash": "000000000000000000320bb059c9271f73c12553872e721c8d1749556c7b15f1",
  "warning": "spruned 0.0.2a2, emulating bitcoind v0.16",
  "verificationprogress": 100,
  "pruned": false,
  "headers": 532923,
  "chain": "main",
  "blocks": 532923,
  "mediantime": 1532160671,
  "difficulty": null
}

bitcoin ~/.spruned ฿ tail spruned.log
2018-07-21 08:22:31,064 - jsonrpcserver.dispatcher.request - INFO - {'method': 'estimatesmartfee', 'id': 1, 'params': [2, 'CONSERVATIVE']}
2018-07-21 08:22:31,191 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2018:08:22:31 +0000] "POST / HTTP/1.1" 200 294 "-" "-"
2018-07-21 08:22:31,193 - jsonrpcserver.dispatcher.request - INFO - {'method': 'estimatesmartfee', 'id': 1, 'params': [4, 'ECONOMICAL']}
2018-07-21 08:22:31,317 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2018:08:22:31 +0000] "POST / HTTP/1.1" 200 294 "-" "-"
2018-07-21 08:22:31,319 - jsonrpcserver.dispatcher.request - INFO - {'method': 'estimatesmartfee', 'id': 1, 'params': [100, 'ECONOMICAL']}
2018-07-21 08:22:31,339 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2018:08:22:31 +0000] "POST / HTTP/1.1" 200 296 "-" "-"
2018-07-21 08:22:31,509 - jsonrpcserver.dispatcher.request - INFO - {'method': 'getblockhash', 'id': 1, 'params': [532924]}
2018-07-21 08:22:31,512 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2018:08:22:31 +0000] "POST / HTTP/1.1" 400 294 "-" "-"
2018-07-21 08:22:32,516 - jsonrpcserver.dispatcher.request - INFO - {'method': 'getblockhash', 'id': 1, 'params': [532924]}
2018-07-21 08:22:32,519 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2018:08:22:32 +0000] "POST / HTTP/1.1" 400 294 "-" "-"

Remove Bitcoind as dependency

spruned may do more than hijack some bitcoind APIs.
with the easy-to-use electrum interface, spruned can download block headers and work in 100% solo mode, providing a limited set of APIs and indexes.

sPRUNED works with very few sources

need more sources. check third_party/ module and add your own, it's very appreciated.
also, sP (sPRUNED Providers, check motivations) wanted.

Issue: getrawtransaction coinbase genesist block

$ ./bitcoin-cli getrawtransaction 4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b
error code: -32000
error message:
Server error

using other transactions work as espected #

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.