samrushing / caesure

Python Bitcoin Node

Caesure: a python bitcoin server/node

name

It's a pun on the words Caesar and Seizure. "Render unto Caesar..."

requirements

1. shrapnel
2. Cython
3. secp256k1 (optional)

install

$ sudo python setup.py install
$ mkdir /usr/local/caesure/

Make sure the /usr/local/caesure directory is writable by the user that will be running caesure.

To avoid a long startup time, fetch a copy of the blockchain and convert it. [see below for instructions]

status

Handles incoming & outgoing connections, does parallel blockchain download. Caches metadata for quick startup. Many-worlds ledger implementation nearly finished.

See TODO.txt for more detail on status.

design

Since this uses shrapnel, it leaves out Windows users, but still allows bsd, darwin/osx, & linux.

The target platform is a well-connected machine (i.e., in a colocation facility) with fast disk and lots of memory. [As of October 2014, the process size is approximately 3GB. I would recommend at least 6GB of memory, the more the merrier.]

Note: you should be using a 64-bit system. Although it should be possible to run most of the system in 32-bit mode, consider it an unsupported configuration, because of the memory requirements.

Performance-sensitive code is written in Cython, including packet codec, b58, hexify, etc...

The script engine is mostly done. Needs some work on failing constraints like stack size, sig count, etc.

The current plan is to have a pruning ledger in memory (with journalling-style checkpoints to disk). Without a ledger, this code is not a full forwarding node, and thus by default the 'relay' flag in the outgoing version packet is set to False.

See TODO.txt & ledger.py for details.

usage

$ scripts/caesure -h::

usage: caesure [-h] [-o OUTGOING] [-i INCOMING] [-s IP:PORT] [-c IP:PORT] [-m]
               [-a] [-r] [-u USER:PASS] [-v]

optional arguments:
  -h, --help            show this help message and exit
  -o OUTGOING, --outgoing OUTGOING
                        total number of outgoing connections
  -i INCOMING, --incoming INCOMING
                        total number of incoming connections
  -s IP:PORT, --serve IP:PORT
                        serve on this address
  -c IP:PORT, --connect IP:PORT
                        connect to this address
  -m, --monitor         run the monitor on /tmp/caesure.bd
  -a, --webui           run the web interface at http://localhost:8380/admin/
  -r, --relay           [hack] set relay=True
  -u USER:PASS, --user USER:PASS
                        webui user (will listen on INADDR_ANY)
  -v, --verbose         show verbose packet flow

Connecting to a local bitcoind::

$ scripts/caesure -o 1 -i 0 -c 127.0.0.1:8333 -m -a

Start up a node with 20 outgoing connections and 0 incoming (i.e., no server)::

$ scripts/caesure -o 20 -i 0 -c -m -a

Start up a node with 100 outgoing connections and 100 incoming::

$ scripts/caesure -o 100 -i 100 -m -a -s 1.2.3.4:8333

Once up and running, caesure will start downloading the block chain from the network if necessary.

You can monitor its progress via the web ui:

http://127.0.0.1:8380/admin/status

Or via the back door / monitor:

[from another terminal]

$ telnet /tmp/caesure.bd

telnet to a unix socket is bsd only, on linux try:

$ nc -CU /tmp/caesure.bd

you'll get a python prompt:

>>> db = G.block_db
>>> len(db.blocks)
10123
>>> 
>>> len(db.blocks)
18539
>>> 
>>> len(db.blocks)
66029
>>> 
[...]

bootstrap.dat

I recommend that you download the blockchain bootstrap.dat file (via bit torrent), and use that as your starting point. The format of the bootstrap.dat file is nearly identical to caesure's native format, and can be converted in-place::

$ python scripts/convert_bootstrap.py bootstrap.dat
$ mv bootstrap.dat /usr/local/caesure/blocks.bin

You should be able to find the torrent here: https://bitcoin.org/bin/blockchain/

testnet

Testnet support is currently missing, it's not hard to add it back in if needed.

logging

Caesure uses a binary (fast, machine-readable) logging system, using the ASN1 capabilities of shrapnel. In utils/catlog.py is a tool that can be used for decoding/processing the logfile. It can be combined with "tail -f" to tail the logs, e.g.::

$ cat /usr/local/caesure/log.asn1 | catlog | less
$ tail -f /usr/local/caesure/log.asn1 | catlog | less

fun with the block chain

Rather than running a client, you can just start up python and play with the block database. The block database is written in append-only mode, so it's safe to open it read-only from another process, even while the client is running.

$ python -i caesure/block_db.py::

  >>> db = BlockDB()
reading metadata...done 5.62 secs (last_block=302294)
reading block headers...starting at pos 19149083102...(302294)done. scanned 17 blocks in 0.00 secs
>>> db.last_block
302310
>>> db.by_num(_)
<__main__.BLOCK object at 0x800f090d0>
>>> b = _
>>> b.transactions[1]
<__main__.TX object at 0x800e30fa0>
>>> tx = _
>>> tx.dump()
hash: e8751d4130d77cdc3746dc6cce32e00f57b1abf9cef84104a5764cefc933f38b
inputs: 2
  0 0c8c368d00fe30e30426d3759bbb0c0244242d53eb1935ab6ac9e6b7e39e5356:1 ['0x3045022100f0a0e4c2bfd414e8232ae98a4ba564d0040338e8a94e563d6ab599900e2c93dd022054b68c96da5ba2e68bc46a9b65cd230028aa66caa83d85be9e7d155838a44c7e01', '0x04bb8234d9fbc26ad8e9c328805f8ff77cc3857ac3875ad56c74203b93fffe33a868cb870841128c0ce43838929be16da0a369c683c3d2e7fc395e4a21ae6faf30'] 4294967295
  1 540cff744f98351c7c8459b85a52ca6ab3a6c2b9cdcaf8aec1e4ad2c58cae2df:0 ['0x304502203aca08e12b347d6056f88367434d6ac9fb097eb0d6b677e7987c2122f8d8989402210088cf9838a9ccbd0878b44c381aedb1b8c4e84deeda04964aa4e53e8d9e086c2801', '0x0455d41ff12fbe28a8112d3027f100a80fe16a877b69e9fc66062777ece4d2327c9b7607f056afd4b59014b42daab284e09b64f92fff498e068bcee41752c5e26d'] 4294967295
2 outputs
  0 39.52026906 ['OP_DUP', 'OP_HASH160', '0x67a5b321d47682682249a4baa1cf53de4f6d2701', 'OP_EQUALVERIFY', 'OP_CHECKSIG']
  1 47.87863094 ['OP_DUP', 'OP_HASH160', '0x96a852c7f06db0d93e4bfac314b979976d1095cc', 'OP_EQUALVERIFY', 'OP_CHECKSIG']
lock_time: 0
>>> tx.inputs[0]
((<0c8c368d00fe30e30426d3759bbb0c0244242d53eb1935ab6ac9e6b7e39e5356>, 1), 'H0E\x02!\x00\xf0\xa0\xe4\xc2\xbf\xd4\x14\xe8#*\xe9\x8aK\xa5d\xd0\x04\x038\xe8\xa9NV=j\xb5\x99\x90\x0e,\x93\xdd\x02 T\xb6\x8c\x96\xda[\xa2\xe6\x8b\xc4j\x9be\xcd#\x00(\xaaf\xca\xa8=\x85\xbe\x9e}\x15X8\xa4L~\x01A\x04\xbb\x824\xd9\xfb\xc2j\xd8\xe9\xc3(\x80_\x8f\xf7|\xc3\x85z\xc3\x87Z\xd5lt ;\x93\xff\xfe3\xa8h\xcb\x87\x08A\x12\x8c\x0c\xe488\x92\x9b\xe1m\xa0\xa3i\xc6\x83\xc3\xd2\xe7\xfc9^J!\xaeo\xaf0', 4294967295)
>>> parse_script (_[1])
[(0, '0E\x02!\x00\xf0\xa0\xe4\xc2\xbf\xd4\x14\xe8#*\xe9\x8aK\xa5d\xd0\x04\x038\xe8\xa9NV=j\xb5\x99\x90\x0e,\x93\xdd\x02 T\xb6\x8c\x96\xda[\xa2\xe6\x8b\xc4j\x9be\xcd#\x00(\xaaf\xca\xa8=\x85\xbe\x9e}\x15X8\xa4L~\x01'), (0, '\x04\xbb\x824\xd9\xfb\xc2j\xd8\xe9\xc3(\x80_\x8f\xf7|\xc3\x85z\xc3\x87Z\xd5lt ;\x93\xff\xfe3\xa8h\xcb\x87\x08A\x12\x8c\x0c\xe488\x92\x9b\xe1m\xa0\xa3i\xc6\x83\xc3\xd2\xe7\xfc9^J!\xaeo\xaf0')]
>>> pprint_script (_)
['0x3045022100f0a0e4c2bfd414e8232ae98a4ba564d0040338e8a94e563d6ab599900e2c93dd022054b68c96da5ba2e68bc46a9b65cd230028aa66caa83d85be9e7d155838a44c7e01', '0x04bb8234d9fbc26ad8e9c328805f8ff77cc3857ac3875ad56c74203b93fffe33a868cb870841128c0ce43838929be16da0a369c683c3d2e7fc395e4a21ae6faf30']
>>> tx.outputs[0]
(3952026906L, 'v\xa9\x14g\xa5\xb3!\xd4v\x82h"I\xa4\xba\xa1\xcfS\xdeOm\'\x01\x88\xac')
>>> parse_script (_[1])
[(2, 118), (2, 169), (0, 'g\xa5\xb3!\xd4v\x82h"I\xa4\xba\xa1\xcfS\xdeOm\'\x01'), (2, 136), (3, 172, 'v\xa9\x14g\xa5\xb3!\xd4v\x82h"I\xa4\xba\xa1\xcfS\xdeOm\'\x01\x88\xac')]
>>> pprint_script (_)
['OP_DUP', 'OP_HASH160', '0x67a5b321d47682682249a4baa1cf53de4f6d2701', 'OP_EQUALVERIFY', 'OP_CHECKSIG']
>>> 

fetch a block by number::

>>> db.num_block[135000]
set([<00000000000001bf349e3e8195f95a080ea17efe012cf7f512664829f9d3772d>])

>>> db.by_num(135000)
<__main__.BLOCK object at 0x800f090d0>

dump all its transactions::

>>> for tx in b.transactions:
...     tx.dump()
... 

Note: you can also do all the above via the back door of a running caesure instance.

Your Support Appreciated: 1PDd8exMdhRTLAfNrjBQ9b8DYxkky3cFy1

caesure's Issues

block download stalls

On startup I get plenty of blocks, but this peters out after a few minutes. I'm still getting INV and ADDR messages, however.

assertion failed in convert_bootstrap.py

syadasti@caesure:/usr/local/caesure$ time python ~/caesure/scripts/convert_bootstrap.py bootstrap.dat 
Traceback (most recent call last):
  File "/home/syadasti/caesure/scripts/convert_bootstrap.py", line 23, in <module>
    assert (magic == '\xf9\xbe\xb4\xd9')
AssertionError

real    0m0.011s
user    0m0.000s
sys 0m0.011s
syadasti@caesure:/usr/local/caesure$

setup.py leaves source tree unusable

"setup.py install" ends up getting the shared libraries generated by cython and the contents of caesure/ installed into sys.path nicely, but a workaround is needed to run bitcoin.py from the tree:

syadasti@uberlemur:~/caesure$ python setup.py build
(all is well)
syadasti@uberlemur:~/caesure$ sudo python setup.py install
(all is well)
syadasti@uberlemur:~/caesure$ python bitcoin.py
Traceback (most recent call last):
  File "bitcoin.py", line 32, in <module>
    import caesure.proto
  File "/home/syadasti/caesure/caesure/__init__.py", line 2, in <module>
    import script
  File "/home/syadasti/caesure/caesure/script.py", line 529, in <module>
    from caesure._script import parse_script, unparse_script
ImportError: No module named _script
syadasti@uberlemur:~/caesure$ ls -l caesure
total 104
-rw-rw-r-- 1 syadasti syadasti    37 Sep 19 21:57 __init__.py
-rw-rw-r-- 1 syadasti syadasti   157 Sep 19 21:59 __init__.pyc
-rw-rw-r-- 1 syadasti syadasti  2481 Sep 19 21:57 opcodes.h
-rw-rw-r-- 1 syadasti syadasti 11840 Sep 19 21:57 proto.pyx
-rw-rw-r-- 1 syadasti syadasti 22618 Sep 19 21:57 script.py
-rw-rw-r-- 1 syadasti syadasti 35355 Sep 19 21:59 script.pyc
-rw-rw-r-- 1 syadasti syadasti  8020 Sep 19 21:57 _script.pyx
-rw-rw-r-- 1 syadasti syadasti  9204 Sep 19 21:57 wallet.py
syadasti@uberlemur:~/caesure$ ls -l /usr/local/lib/python2.7/dist-packages/caesure
total 740
-rw-rw-r-- 1 root staff     37 Sep 19 20:31 __init__.py
-rw-r--r-- 1 root staff    173 Sep 19 20:32 __init__.pyc
-rwxrwxr-x 1 root staff 434469 Sep 19 20:32 proto.so
-rw-rw-r-- 1 root staff  22618 Sep 19 20:31 script.py
-rw-r--r-- 1 root staff  37403 Sep 19 20:32 script.pyc
-rwxrwxr-x 1 root staff 221244 Sep 19 20:32 _script.so
-rw-rw-r-- 1 root staff   9204 Sep 19 20:31 wallet.py
-rw-r--r-- 1 root staff   7404 Sep 19 20:32 wallet.pyc
syadasti@uberlemur:~/caesure$ mv caesure/ ca
syadasti@uberlemur:~/caesure$ python bitcoin.py
(all is well)

latency warnings: probably the utxo outpoints datatype

I'm seeing multi-second latency warnings in feed_block... at first I thought it was disk I/O, but now I think there's a serious problem with the txmap::outpoints class... need to experiment with replacing that vector with a set. The reason I say this is that I can see a series of 20-30 blocks fly by, each taking a fraction of a second. Then a block comes in that takes several seconds to process.

use fd in logs to help track node status

setup.py: cython file fails compilation on ubuntu

I installed cython using sudo apt-get install cython. Any suggestions?

$ uname -a
Linux ip-172-31-25-90 3.13.0-29-generic #53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

$ cython --version
Cython version 0.20.1post0

$ sudo python setup.py install
...
...
cythoning caesure/txfaa.pyx to caesure/txfaa.cpp

Error compiling Cython file:
------------------------------------------------------------
...
        for n, d in walk_depth (self.root, 0):
            fout.write ('%s%4d %r:%r\n' % ('  ' * d, n.level, n.key.encode('hex'), n.amt, n.script))

    def new_entry (self, bytes txname, object vals):
        for index, amt, script in vals:
            self.root = tree_insert (self.root, make_key (txname, index), amt, script)
                                                        ^
------------------------------------------------------------

caesure/txfaa.pyx:308:57: Obtaining 'char *' from temporary Python value

Error compiling Cython file:
------------------------------------------------------------
...




cdef extern from *:
    ctypedef uint64_t X 'uint64_t'
            ^
------------------------------------------------------------

pair.from_py:152:13: 'uint64_t' is not a type identifier
building 'caesure.txfaa' extension
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c caesure/txfaa.cpp -o build/temp.linux-x86_64-2.7/caesure/txfaa.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
caesure/txfaa.cpp:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation.
 #error Do not use this file, it is the result of a failed Cython compilation.
  ^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Is this being maintained?

Thanks. EOM

Is this a full node

Just wonder if this is a full node or you still rely on a bitcoind node running in order to get to use this.
Whats the difference between this project and pycoind?