Coder Social home page Coder Social logo

bitcoin-abe's Introduction

Abe: a free block chain browser for Bitcoin-based currencies. https://github.com/bitcoin-abe/bitcoin-abe

Copyright(C) 2011,2012,2013 by Abe developers.
License: GNU Affero General Public License, see the file LICENSE.txt.
Portions Copyright (c) 2010 Gavin Andresen, see bct-LICENSE.txt.

Welcome to Abe!

This software reads the Bitcoin block file, transforms and loads the data into a database, and presents a web interface similar to Bitcoin Block Explorer, http://blockexplorer.com/.

Abe draws inspiration from Bitcoin Block Explorer (BBE) and BlockChain.info and seeks some level of compatibility with them but uses a completely new implementation.

Installation

Issue:

python setup.py install

This will install abe to your system. After you set up the config file and database (see below and README-.txt) you can run:

python -m Abe.abe --config myconf.conf --commit-bytes 100000 --no-serve

This will perform the initial data load and will take a long time. After it's fully synced, you can run the web server with:

python -m Abe.abe --config myconf.conf

To really get everything right see the README file for your type of database.

Abe depends on Python 2.7 (or 2.6), the pycrypto package, and an SQL database supporting ROLLBACK. Abe runs on PostgreSQL, MySQL's InnoDB engine, and SQLite. Other SQL databases may work with minor changes. Abe formerly ran on some ODBC configurations, Oracle, and IBM DB2, but we have not tested to be sure it still works. See the comments in abe.conf about dbtype for configuration examples.

Abe works with files created by the original (Satoshi) Bitcoin client. You will need a copy of the block files (blk0001.dat, blk0002.dat, etc. in your Bitcoin directory or its blocks/ subdirectory). You may let Abe read the block files while Bitcoin runs, assuming Bitcoin only appends to the file. Prior to Bitcoin v0.8, this assumption seemed safe. Abe may need some fixes to avoid skipping blocks while current and future Bitcoin versions run.

NovaCoin and CryptoCash support depends on the ltc_scrypt module available from https://github.com/CryptoManiac/bitcoin-abe (see README-SCRYPT.txt).

Hirocoin (and any other X11) support depends on the xcoin_hash module available from https://github.com/evan82/xcoin-hash.

Bitleu (a Scrypt-Jane coin) depends on the yac_scrypt module.

Copperlark (a Keccak coin) depends on the sha3 module available via "easy_install pysha3".

License

The GNU Affero General Public License (LICENSE.txt) requires whoever modifies this code and runs it on a server to make the modified code available to users of the server. You may do this by forking the Github project (if you received this code from Github.com), keeping your modifications in the new project, and linking to it in the page template. Or you may wish to satisfy the requirement by simply passing "--auto-agpl" to "python -m Abe.abe". This option makes all files in the directory containing abe.py and its subdirectories available to clients. See the comments in abe.conf for more information.

Database

For usage, run "python -m Abe.abe --help" and see the comments in abe.conf.

You will have to specify a database driver and connection arguments (dbtype and connect-args in abe.conf). The dbtype is the name of a Python module that supports your database. Known to work are psycopg2 (for PostgreSQL) and sqlite3. The value of connect-args depends on your database configuration; consult the module's documentation of the connect() method.

You may specify connect-args in any of the following forms:

  • omit connect-args to call connect() with no arguments

  • named arguments as a JSON object, e.g.: connect-args = { "database": "abe", "password": "b1tc0!n" }

  • positional arguments as a JSON array, e.g.: connect-args = ["abe", "abe", "b1tc0!n"]

  • a single string argument on one line, e.g.: connect-args = /var/lib/abe/abe.sqlite

For JSON syntax, see http://www.json.org.

Slow startup

Reading the block files takes much too long, several days or more for the main BTC block chain as of 2013. However, if you use a persistent database, Abe remembers where it stopped reading and starts more quickly the second time.

Replacing the Block File

Abe does not currently handle block file changes gracefully. If you replace your copy of the block chain, you must rebuild Abe's database or (quicker) force a rescan. To force a rescan of all data directories, run Abe once with the "--rescan" option.

Web server

By default, Abe expects to be run in a FastCGI environment. For an overview of FastCGI setup, see README-FASTCGI.txt.

To run the built-in HTTP server instead of FastCGI, specify a TCP port and network interface in abe.conf, e.g.:

port 2750
host 127.0.0.1  # or a domain name

Input

To display Namecoin, NovaCoin, or any block chain with data somewhere other than the default Bitcoin directory, specify "datadir" in abe.conf, e.g.:

datadir = /home/bitcoin/.namecoin

The datadir directive can include a new chain's basic configuration, e.g.:

datadir += [{
        "dirname": "/home/weeds/testnet",
        "chain":   "Weeds",
        "code3":   "WDS",
        "address_version": "o" }]

Note that "+=" adds to the existing datadir configuration, while "=" replaces it. For help with address_version, please open doc/FAQ.html in a web browser.

The web interface is currently unaware of name transactions, but see namecoin_dump.py in the tools directory.

More information

Please see TODO.txt for a list of what is not yet implemented but would like to be.

Forum thread: https://bitcointalk.org/index.php?topic=22785.0 Newbies: https://bitcointalk.org/index.php?topic=51139.0

Donations appreciated: 1PWC7PNHL1SgvZaN7xEtygenKjWobWsCuf (BTC) NJ3MSELK1cWnqUa6xhF2wUYAnz3RSrWXcK (NMC)

bitcoin-abe's People

Contributors

amescornish avatar bitbandi avatar bitfoo avatar crowning- avatar csyangchen avatar dermoth avatar framer99 avatar jabbervorx avatar jl2035 avatar jtobey avatar lvl4sword avatar melech avatar morinpa avatar n00bsys0p avatar recursive-rat4 avatar rixu101 avatar schinzelh avatar sigshen avatar teamdandelion avatar zackster avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

bitcoin-abe's Issues

new address types required for correct firstbits

Abe fails to parse some addresses, in particular multisig and script-hash outputs. This can result in wrong firstbits associations, where a later, parsed address gets the firstbits that should belong to the unrecognized address.

Abe must parse multiple-signature outputs. This presents a table-design challenge due to reliance on a single pubkey_id column in the txout table.

Abe must decide whether to consider addresses in multiple-signature outputs to have "appeared" for the purpose of firstbits allocation. See: https://bitcointalk.org/index.php?topic=16217.msg957933#msg957933

Abe should parse script-hash outputs and find their firstbits. This has lower priority, given that the addresses appear to use version byte 5 and will not collide with other addresses. They may lead to missing data but not wrong firstbits.

Open questions: Do examples of wrong assignments exist? Would it make sense to look for new addresses in input (scriptSig) scripts that redeem script-hash outputs? What does firstbits.com do with the new script types?

Thanks for the Quarkcoin explorer site, I left a donation(fyi addresses are reversed). We also need a phone wallet app!

I do not know how to contact the dev of the Quark block chain website( I hear its JTobey?)- The donation addresses are reversed(BTC and QRK), but also I wanted to know if anyone can make this website a little sleeker looking for the Qrk community?
If so, contact us at the Quark Foundation website (qrkfoundation.cc).
Also we are in need now of a phone wallet application for sending and receiving Quark.

YaCoin long int too large to convert to int

I tried to get Abe setup with YaCoin but it's ending up with a python error because of a overflow.

dash@rainbowdashh:~/src/yacoin-abe$ python -m Abe.abe --config abe.conf
ddl_implicit_commit=true
create_table_epilogue=''
max_varchar=4294967295
clob_type=CLOB
binary_type=buffer
int_type=str
Created silly table abe_dual
sequence_type=update
limit_style=native
Assigned chain_id 8 to YACoin
RPC failed: [Errno socket error] [Errno 111] Connection refused
catch_up_rpc: abort
Opened /home/dash/.yacoin/blk0001.dat
Exception at 72056512645693578
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': 8, 'loader': None, 'dirname': u'/home/dash/.yacoin', 'id': 1}
Traceback (most recent call last):
File "Abe/DataStore.py", line 2581, in catch_up
store.catch_up_dir(dircfg)
File "Abe/DataStore.py", line 2835, in catch_up_dir
store.import_blkdat(dircfg, ds, blkfile['name'])
File "Abe/DataStore.py", line 2940, in import_blkdat
b = store.parse_block(ds, chain_id, magic, length)
File "Abe/DataStore.py", line 2971, in parse_block
d['transactions'].append(deserialize.parse_Transaction(ds))
File "Abe/deserialize.py", line 86, in parse_Transaction
d['txIn'].append(parse_TxIn(vds))
File "Abe/deserialize.py", line 45, in parse_TxIn
d['sequence'] = vds.read_uint32()
File "Abe/BCDataStream.py", line 71, in read_uint32
def read_uint32(self): return self._read_num('<I')
File "Abe/BCDataStream.py", line 110, in _read_num
(i,) = struct.unpack_from(format, self.input, self.read_cursor)
OverflowError: long int too large to convert to int
Abe initialized.
Listening on http://0.0.0.0:2750

Can not test for DDL implicit commit

Hallo,
I have got the problem

File "Abe/DataStore.py", line 1375, in configure_ddl_implicit_commit
raise Exception("Can not test for DDL implicit commit.")
Exception: Can not test for DDL implicit commit.

when I try to run "python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve". I still got the information from "#11", but I didn't need to change something in the file "DataStore.py". It seems to be up-to-date. Can anyone help me?

P.s.: I installed MySQL, Python MySQLdb and everything on a 64 bit system (so it's a completely new database).

Thanks for your help!

Warning

ddl_implicit_commit=true
create_table_epilogue=''
Abe/DataStore.py:526: Warning: Converting column 'a' from VARCHAR to TEXT
store.cursor.execute(stmt)
Abe/DataStore.py:526: Warning: Converting column 'b' from VARCHAR to TEXT
store.cursor.execute(stmt)
max_varchar=4294967295
clob_type=LONGTEXT
Abe/DataStore.py:526: Warning: Converting column 'test_varbit' from VARCHAR to TEXT
store.cursor.execute(stmt)
binary_type=hex
int_type=int
Created silly table abe_dual
sequence_type=mysql
limit_style=native
Abe/DataStore.py:526: Warning: Converting column 'txout_scriptPubKey' from VARCHAR to TEXT
store.cursor.execute(stmt)
Abe/DataStore.py:526: Warning: Converting column 'txin_scriptSig' from VARCHAR to TEXT
store.cursor.execute(stmt)
Opened /home/mapleshadow/.chncoin/blk0001.dat
block_tx 1 1
block_tx 2 2
block_tx 3 3

Obtain chain_id for zero confirmation transactions

Hi!

I'm working with ABE capturing realtime transactions. I can see them inserted on the tx, txin and txout tables, but I'm using the same database for multiple blockchains, and want to know which is the chain_id for every transaction.

At this time I need to wait until the transactions is available on txin_detail and txout_detail, but I think there should be a way to know the blockchain of every transaction without waiting, just I'm still missing which tables or what is the right procedure to know that.

So, can you tell me which tables I should check? or what is that procedure?

Thank you!

PS: maybe the chain_id could be added to the tx table.

if tests to add support for coins with other algo's

would it be possible to do an if test to check for a setting in the config to see what algo's to use for each coin etc. this would make it possible to have multiple coins/variations on 1 block explorer such as ppc/btc/ltc/src all on 1

Possible to mark an address with a name?

would it be possible to set an address in the config with a name so that when that address appears in the block explorer it shows the name instead of the address or next to it?

exception on memory map

Ran nicely up to ~188k blocks. then:
desktop:/.bitcoin/bitcoin-abe$ python -m Abe.abe --config abe-pg.conf --commit-bytes 100000 --no-serve
Opened /home/tomer/.bitcoin/blk0001.dat
Failed to catch up {'blkfile_number': 1, 'dirname': '/.bitcoin', 'chain_id': None, 'id': Decimal('1'), 'blkfile_offset': 2097361271}
Traceback (most recent call last):
File "Abe/DataStore.py", line 2424, in catch_up
store.catch_up_dir(dircfg)
File "Abe/DataStore.py", line 2498, in catch_up_dir
next_blkfile = open_blkfile(dircfg['blkfile_number'] + 1)
File "Abe/DataStore.py", line 2465, in open_blkfile
blkfile['stream'].map_file(file, 0)
File "Abe/BCDataStream.py", line 27, in map_file
self.input = mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ)
error: [Errno 12] Cannot allocate memory

too big to fit in memory? got 2G RAM, lots of swap just in case.
fails constantly when retrying the above.

Thanks

Error thrown after 88 hour import

imported bitcoin blockchain, took 88 hours
went to site

Skipped 4969494 NUL bytes at block end
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 85, in run
    self.result = application(self.environ, self.start_response)
  File "/home/arij/bitcoin-abe/Abe/abe.py", line 205, in __call__
    handler(page)
  File "/home/arij/bitcoin-abe/Abe/abe.py", line 279, in handle_chains
    int(row[5]), abe.store.hashout_hex(row[6]), int(row[7]))
TypeError: int() argument must be a string or a number, not 'NoneType'
localhost.localdomain - - [14/Nov/2012 15:56:41] "GET / HTTP/1.1" 500 59```

Next/Prev Transaction by tx_id

Would be nice to see Next/Prev Transaction feature on transaction page based on tx_id, like block page have prev/next block.

Errors in setting up CosmosCoin block explorer

Hey jtobey,
I need your help, I'm trying to run block explorer for CosmosCoin, but I get the following errors:
root@c44181:~/CMC-BE# python -m Abe.abe --config abe-cmc.conf --commit-bytes 100000 --no-serve
RPC data not understood: block hash mismatch
catch_up_rpc: abort
Opened /root/.CosmosCoin/blk0001.dat
Exception at 157003672201461898
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('2'), 'loader': None, 'dirname': u'/root/.CosmosCoin', 'id': 6L}
Traceback (most recent call last):
File "Abe/DataStore.py", line 2624, in catch_up
store.catch_up_dir(dircfg)
File "Abe/DataStore.py", line 2882, in catch_up_dir
store.import_blkdat(dircfg, ds, blkfile['name'])
File "Abe/DataStore.py", line 3009, in import_blkdat
b = store.parse_block(ds, chain_id, magic, length)
File "Abe/DataStore.py", line 3040, in parse_block
d['transactions'].append(deserialize.parse_Transaction(ds))
File "Abe/deserialize.py", line 86, in parse_Transaction
d['txIn'].append(parse_TxIn(vds))
File "Abe/deserialize.py", line 45, in parse_TxIn
d['sequence'] = vds.read_uint32()
File "Abe/BCDataStream.py", line 71, in read_uint32
def read_uint32 (self): return self._read_num('<I')
File "Abe/BCDataStream.py", line 110, in _read_num
(i,) = struct.unpack_from(format, self.input, self.read_cursor)
error: unpack_from requires a buffer of at least 4 bytes

my abe-cmc.conf is as follows:
dbtype MySQLdb
connect-args {"user":"abe_cmc","db":"abe_cmc","passwd":"@Yuan1985@"}
upgrade
host 103.28.44.181
port 19999

datadir = [{
"dirname":"/root/.CosmosCoin",
"chain":"CosmosCoin",
"code3":"CMC",
"address_version": "\u001b" }]

And mysql server has been configured correctly.

Please help me to solve this.

Transactions values needs to be multilplied by 100

Hi! I'm using ABE to get the blockchain for bitcoin, litecoin, testnet and ppcoin into a mysql database.

I just added the process that imports the ppcoin blockchain to my database and I see that all inputs and outputs requires to be multiplied by 100.

Just check this: http://node.coinmarket.pw/address/PThKfvGaixfh6qbKDRFe8N7iRBHxSeNfxT

it should be 16.399.... and not 0.16399....

The problem is only on ppcoin, and not on the other chains.

What could be?

Thank you.

Exception after working fine for many weeks for CryptogenicBullion (CGB)

I am suddenly getting this exception even though Abe has been working fine for almost a month:

Exception at 117431433
Failed to catch up {'blkfile_number': 1, 'dirname': u'/home/mercury/.CryptogenicBullion', 'chain_id': 8, 'id': 1, 'blkfile_offset': 0}
Traceback (most recent call last):
File "Abe/DataStore.py", line 2422, in catch_up
store.catch_up_dir(dircfg)
File "Abe/DataStore.py", line 2475, in catch_up_dir
store.import_blkdat(dircfg, ds, filename[0])
File "Abe/DataStore.py", line 2599, in import_blkdat
store.import_block(b, chain_ids = chain_ids)
File "Abe/DataStore.py", line 1736, in import_block
len(b['transactions']), b['search_block_id']))
File "Abe/DataStore.py", line 462, in sql
store.cursor.execute(cached, params)
OverflowError: long too big to convert

It seems to be a issue that Abe is parsing the block and finding a large number where it suddenly doesn't expect. I got Abe working fine (forking the Novacoin version from Balthazar) without issue over a month ago and has worked smooth until today. The database downloading for Abe seems to fail at block 227545, but using http://coinplorer.com/CGB (a block explorer not using Abe, but rather their own scripting with the daemon) and inspecting the block, I didn't see anything out of the ordinary or fishy with blocks in that part of the chain. I will continue to research it, but any suggestions would be appreciated.

Primecoin (XPM) Failed to catch up - OverflowError: Python int too large to convert to C long

Hi

I've encountered a problem with Abe and Primecoin. I've applied the PPC fix in deserialize.py. (It crashes out of the box and it uses probably PPC structure) However after rerunning Abe stops at:

Opened /home/enira/.primecoin/blocks/blk00000.dat
Exception at 432400545497219875
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 100000, 'chain_id': Decimal('9'), 'loader': None, 'dirname': u'/home/enira/.primecoin', 'id': 4L}
Traceback (most recent call last):
File "Abe/DataStore.py", line 2639, in catch_up
store.catch_up_dir(dircfg)
File "Abe/DataStore.py", line 2897, in catch_up_dir
store.import_blkdat(dircfg, ds, blkfile['name'])
File "Abe/DataStore.py", line 3024, in import_blkdat
b = store.parse_block(ds, chain_id, magic, length)
File "Abe/DataStore.py", line 3055, in parse_block
d['transactions'].append(deserialize.parse_Transaction(ds))
File "Abe/deserialize.py", line 93, in parse_Transaction
d['txOut'].append(parse_TxOut(vds))
File "Abe/deserialize.py", line 67, in parse_TxOut
d['value'] = vds.read_int64()
File "Abe/BCDataStream.py", line 72, in read_int64
def read_int64 (self): return self._read_num('<q')
File "Abe/BCDataStream.py", line 110, in _read_num
(i,) = struct.unpack_from(format, self.input, self.read_cursor)
OverflowError: Python int too large to convert to C long

Any ideas on this?
Thanks in advance and knd regards

no-statistics error: Cannot delete or update a parent row: a foreign key constraint fails

On my Debian 6/mysql 5.5.30 with branch no-statistics after almost a week I receive the following error:

Exception at 50855473
Failed to catch up {'blkfile_number': 100047, 'dirname': '/home/[user]/.bitcoin', 'chain_id': None, 'id': Decimal('2'), 'blkfile_offset': 50609049}
Traceback (most recent call last):
\ File "Abe/DataStore.py", line 2331, in catch_up
store.catch_up_dir(dircfg)
File "Abe/DataStore.py", line 2396, in catch_up_dir
store.import_blkdat(dircfg, ds, blkfile['name'])
File "Abe/DataStore.py", line 2526, in import_blkdat
store.import_block(b, chain_ids = chain_ids)
File "Abe/DataStore.py", line 1815, in import_block
b['height'] - trim_depth, int(block_id)))
File "Abe/DataStore.py", line 2830, in trim_block
txins = store.delete_block_txins(block_id)
File "Abe/DataStore.py", line 2850, in delete_block_txins
(block_id,))
File "Abe/DataStore.py", line 471, in sql
store.cursor.execute(cached, params)
File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 166, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
IntegrityError: (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (abe.unlinked_txin, CONSTRAINT unlinked_txin_ibfk_1 FOREIGN KEY (txin_id) REFERENCES txin (txin_id))')
Abe initialized.

I receive this error with both:
python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve
and
python -m Abe.abe --config abe-my.conf

TypeError: int() argument must be a string or a number, not 'NoneType'

Traceback (most recent call last):
File "/usr/lib/python2.7/wsgiref/handlers.py", line 85, in run
self.result = application(self.environ, self.start_response)
File "/home/bob/bitcoin-abe/Abe/abe.py", line 249, in call
handler(page)
File "/home/bob/bitcoin-abe/Abe/abe.py", line 506, in handle_chain
ss = int(ss)
TypeError: int() argument must be a string or a number, not 'NoneType

And this happened everytime when read the last block...

Error:"No known large integer representation works"

Hi

I am trying to get Abe working on my computer but it is coming up with the following error:

"No known large integer representation works"

My Abe version is Abe-0.7.2.win32 and my PostgreSQL version is 8.3.23.

I have emailed John Tobey about it and he suggested that I upgade my PostgreSQL but unfortunately on my computer, any version of PostgreSQL above 8.3.23 has a problem whereby it does not have proper access to the data folder which causes an error at the end of the installation.

I would be very grateful if anyone can provide a solution to this issue.

Thank you very much.

Kind regards

Tim

CoinAge Problem [I0Coin] [Namecoin]

i was trying to setup every sha256 based still alive chain with ABE, namecoin and i0coin didnt succed:

abe.conf

datadir = [{
        "dirname": "/home/k1773r/.i0coin",
        "chain":   "I0Coin",
        "code3":   "I0C",
        "address_version": "\u0069" }]

datadir += [{
        "dirname": "/home/k1773r/.namecoin",
        "chain":   "NameCoin",
        "code3":   "NMC",
        "address_version": "\u0034" }]

error:

Traceback (most recent call last):
  File "/usr/lib/python2.6/wsgiref/handlers.py", line 93, in run
    self.result = application(self.environ, self.start_response)
  File "/usr/local/lib/python2.6/dist-packages/Abe/abe.py", line 205, in __call__
    handler(page)
  File "/usr/local/lib/python2.6/dist-packages/Abe/abe.py", line 459, in handle_chain
    ss = int(ss)
TypeError: int() argument must be a string or a number, not 'NoneType'

Error when importing

I'm running bitcoind version v0.8.1.0-g34d62a8-beta (from ubuntu ppa).

My block chain is currently up-to-date according to bitcoin-qt.

When I run abe I get this error:

% python -m Abe.abe --config abe.conf
no chain_id
catch_up_rpc: abort
Opened /home/jim/.bitcoin/blocks/blk00000.dat
Exception at 521845
Failed to catch up {'blkfile_offset': 521621, 'blkfile_number': 100000, 'chain_id': None, '>loader': None, 'dirname': u'/home/jim/.bitcoin', 'id': 1}
Traceback (most recent call last):
File "Abe/DataStore.py", line 2555, in catch_up
store.catch_up_dir(dircfg)
File "Abe/DataStore.py", line 2809, in catch_up_dir
store.import_blkdat(dircfg, ds, blkfile['name'])
File "Abe/DataStore.py", line 2917, in import_blkdat
store.import_block(b, chain_ids = chain_ids)
File "Abe/DataStore.py", line 1770, in import_block
len(b['transactions']), b['search_block_id']))
File "Abe/DataStore.py", line 481, in sql
store.cursor.execute(cached, params)
OverflowError: long too big to convert
Abe initialized.
Listening on http://localhost:2750

My config is:

connect-args = abe.sqlite
datadir += /home/jim/.bitcoin
dbtype = sqlite3
host localhost
port 2750

I'm sure I'm doing something dumb if noöne else is having this problem, so any advice or help would be greatly appreciated.

no chain_id catch_up_rpc: abort

My Bitcoin QT client is fully up to date and I keep it running in the background to ensure new block information is downloaded.

But every time I run abe the following errors appear below.

Heres my abe.conf:

dbtype = MySQLdb
connect-args {"user":"abe","db":"abe","passwd":"PASSWORD"}
upgrade
port 2750
host localhost
datadir += /Users/matthewlobas/Library/Application Support/Bitcoin/

[1]+ Stopped python -m Abe.abe --config abe.conf
computer$ python -m Abe.abe --config abe.conf
no chain_id
catch_up_rpc: abort
Opened /Users/matthewlobas/Library/Application Support/Bitcoin/blocks/blk00002.dat
Exception at 129792251
Failed to catch up {'blkfile_offset': 129743158, 'blkfile_number': 100002, 'chain_id': None, 'loader': None, 'dirname': '/Users/matthewlobas/Library/Application Support/Bitcoin/', 'id': Decimal('1')}
Traceback (most recent call last):
File "Abe/DataStore.py", line 2639, in catch_up
store.catch_up_dir(dircfg)
File "Abe/DataStore.py", line 2897, in catch_up_dir
store.import_blkdat(dircfg, ds, blkfile['name'])
File "Abe/DataStore.py", line 3027, in import_blkdat
store.import_block(b, chain_ids = chain_ids)
File "Abe/DataStore.py", line 1739, in import_block
tx['tx_id'] = store.import_and_commit_tx(tx, pos == 0)
File "Abe/DataStore.py", line 2261, in import_and_commit_tx
tx_id = store.import_tx(tx, is_coinbase)
File "Abe/DataStore.py", line 2185, in import_tx
store.intin(tx['lockTime']), tx['size']))
File "Abe/DataStore.py", line 510, in sql
store._execute(cached, params)
File "Abe/DataStore.py", line 487, in _execute
store.cursor.execute(stmt, params)
File "build/bdist.macosx-10.3-fat/egg/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "build/bdist.macosx-10.3-fat/egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')
Abe initialized.
Listening on http://localhost:2750

Storing address in database in User friendly way

Hi,
I really like Abe!
Why addresses not stored in db in common user-friendly way, and is there any fast and nice solution to do so, for example to store it in tx*_detail?
Another thing is there any way to store transaction comments in db or Transaction HEX Data, like we can see it here https://andarazoroflove.org/explorer/betacoin/block_crawler.php?transaction=b9684d589b29cb27e67c8c257ab2733f206788f64100c4806d94bfa9dd21c14e. (TX Message and HEX Data section) -> https://github.com/CallMeJake/BlockCrawler ?

Abe not following datadir blockfile tail (bitcoin-qt v0.8)

Hi,

So I did a full import, and it worked, however at the end I got something like this:

Skipped 12019996 NUL bytes at block end

Then I start Abe, all looks fine but the last block was a few hours ago.

I'm not sure if something changed in the v0.8 block format, but is appears Abe is writing the file's tail position in datadir.blkfile_offset, and it should instead write the real data tail offset.

The fix for now is to issue a command such as:

UPDATE datadir SET blkfile_offset=blkfile_offset-12019996 WHERE datadir_id=1 AND blkfile_number=100008;

Which resets the file position back to where is should.

As I fix I'd be tempted to use a similar update command in Abe/DataStore.py:2638 in save_blkfile_offset().

Also I noticed the way it looks for the next file in open_blkfile() is pretty rough... How you you now the client didn't write a new block in the current file between the last read and the attempt to open the next one?

I think you should do the following (at Abe/DataStore.py:2470):

  1. Check the presence of a next file
  2. Read current blockfile
  3. If next file was there in 1, move on to it, else return.

What do you think? I'm reticent making the change myself as I don't know that code very well and I don't want to break my blocks - at least not without first making a backup.

Thanks

Changing the default view

I'm wondering if there is a switch or setting that can be changed to make the chain/CHAIN page (the one showing the last 20 blocks) show when the visitor goes to the base abe url.

If not, that would be a great option.

error: unpack_from requires a buffer of at least 4 bytes

I read something about a timestamp missing.

What would cause this:

python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve

Opened /home/.Philosopherstone/blk0001.dat
Exception at 72056514054979722
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('9'), 'loader': None, 'dirname': u'/home/vnc/.Philosopherstone/', 'id': 30L}
Traceback (most recent call last):
File "Abe/DataStore.py", line 2633, in catch_up
store.catch_up_dir(dircfg)
File "Abe/DataStore.py", line 2897, in catch_up_dir
store.import_blkdat(dircfg, ds, blkfile['name'])
File "Abe/DataStore.py", line 3024, in import_blkdat
b = store.parse_block(ds, chain_id, magic, length)
File "Abe/DataStore.py", line 3055, in parse_block
d['transactions'].append(deserialize.parse_Transaction(ds))
File "Abe/deserialize.py", line 88, in parse_Transaction
d['txIn'].append(parse_TxIn(vds))
File "Abe/deserialize.py", line 46, in parse_TxIn
d['sequence'] = vds.read_uint32()
File "Abe/BCDataStream.py", line 71, in read_uint32
def read_uint32 (self): return self._read_num('<I')
File "Abe/BCDataStream.py", line 110, in _read_num
(i,) = struct.unpack_from(format, self.input, self.read_cursor)
error: unpack_from requires a buffer of at least 4 bytes

Given the Wallet Address, query the database

What is the reason for converting the address to a pubkey hash, and the pubkey to a pubkey hash and compring them? Why not store the pubkey in the form of the address so we can query the requested address directly?

SolarCoin: OverflowError: Python int too large to convert to C long

Trying to add SLR SolarCoin to bitcoin-abe getting following error. Have tried using the PPCoin branch, didn't help. Any thoughts?

block 2082 already in chain 9
Exception at 69259
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': 9, 'loader': None, 'dirname': '/home/ubuntu/.solarcoin', 'id': Decimal('30')}
Traceback (most recent call last):
File "Abe/DataStore.py", line 2639, in catch_up
store.catch_up_dir(dircfg)
File "Abe/DataStore.py", line 2897, in catch_up_dir
store.import_blkdat(dircfg, ds, blkfile['name'])
File "Abe/DataStore.py", line 3024, in import_blkdat
b = store.parse_block(ds, chain_id, magic, length)
File "Abe/DataStore.py", line 3055, in parse_block
d['transactions'].append(deserialize.parse_Transaction(ds))
File "Abe/deserialize.py", line 92, in parse_Transaction
for i in xrange(n_vout):
OverflowError: Python int too large to convert to C long

Support for altcoins with 4 character abbreviations

If I'm not mistaken Abe only allows 3 characters in code3. (web interface just shows the first 3)

There are some altcoins with 4 caracters (like DOGE, MEOW, FUCK).

Would be great, if a datadir config file entry with 4 characters would work.

Importing the testnet3 chain bugs out on a certain block

block_tx 54503 88349
Exception at 21692239
Failed to catch up {'blkfile_offset': 21640593, 'blkfile_number': 100000, 'chain_id': 8L, 'loader': None, 'dirname': u'/home/chungy/.bitcoin/testnet3', 'id': 1L}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2581, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2835, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 2943, in import_blkdat
    store.import_block(b, chain_ids = chain_ids)
  File "Abe/DataStore.py", line 1718, in import_block
    tx['tx_id'] = store.import_tx(tx, pos == 0)
  File "Abe/DataStore.py", line 2149, in import_tx
    store.binin(txout['scriptPubKey']), pubkey_id))
  File "Abe/DataStore.py", line 497, in sql
    store._execute(cached, params)
  File "Abe/DataStore.py", line 481, in _execute
    store.cursor.execute(stmt, params)
DataError: value too long for type character varying(20000)

As it might be guessed, I was trying to load this into a PostgreSQL database; the block in particular is very large. It might be worth changing the type in the schema to text rather than varying to account for unexpectedly large transactions like this.

Connect to an external MySQL with tcp/ip

Hi! I want to know the connect-args settings to connect abe to an external mysql server.

At the moment I tested adding "host" and "server" but I think I'm wrong in the parameter name.

If it is not possible at the moment, I want to ask for that support :)

Thanks.

Method to find correct address version?

I'm using Abe with a scrypt altcoin, and the address it's displaying aren't correct. The documentation should detail how to find both the magic_number and the address_version used in initial configuration - I found how the magic number, but can't find the address_version.

Fails to import Novacoin

Importing the novacoin chain fails with

Exception at 72056513014792330
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 100000, 'chain_id': 9, 'loader': None, 'dirname': u'/home/drak/.novacoin', 'id': 1}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2639, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2897, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 3024, in import_blkdat
    b = store.parse_block(ds, chain_id, magic, length)
  File "Abe/DataStore.py", line 3055, in parse_block
    d['transactions'].append(deserialize.parse_Transaction(ds))
  File "Abe/deserialize.py", line 88, in parse_Transaction
    d['txIn'].append(parse_TxIn(vds))
  File "Abe/deserialize.py", line 46, in parse_TxIn
    d['sequence'] = vds.read_uint32()
  File "Abe/BCDataStream.py", line 71, in read_uint32
    def read_uint32 (self): return self._read_num('<I')
  File "Abe/BCDataStream.py", line 110, in _read_num
    (i,) = struct.unpack_from(format, self.input, self.read_cursor)
error: unpack_from requires a buffer of at least 4 bytes

MySQL server has gone away

Hi, i am running 2 instances of abe for testnet and for the main chain (using seperate databaces, seperate clients and seperate instances of abe.

Up until block 54515 on testnet everything runs fine, however it seems that after this block i get this error, meaning i cant continue dumping blocks to sql:

C:\Users\Matthew\Desktop\bitcoin-abe-master>python -m Abe.abe --config abe-my-te
stnet.conf --no-serve
RPC failed: [Errno socket error] [Errno 10061] No connection could be made becau
se the target machine actively refused it
catch_up_rpc: abort
Opened C:/Users/Matthew/AppData/Roaming/Bitcoin/testnet3\blk0001.dat
block_tx 54514 88368
block_tx 54515 88369
Exception at 22941369
Failed to catch up {'blkfile_offset': 21942607, 'blkfile_number': 1, 'chain_id':
2, 'loader': None, 'dirname': 'C:/Users/Matthew/AppData/Roaming/Bitcoin/testnet
3', 'id': Decimal('1')}
Traceback (most recent call last):
File "Abe\DataStore.py", line 2639, in catch_up
store.catch_up_dir(dircfg)
File "Abe\DataStore.py", line 2897, in catch_up_dir
store.import_blkdat(dircfg, ds, blkfile['name'])
File "Abe\DataStore.py", line 3027, in import_blkdat
store.import_block(b, chain_ids = chain_ids)
File "Abe\DataStore.py", line 1741, in import_block
tx['tx_id'] = store.import_tx(tx, pos == 0)
File "Abe\DataStore.py", line 2205, in import_tx
store.binin(txout['scriptPubKey']), pubkey_id))
File "Abe\DataStore.py", line 510, in sql
store._execute(cached, params)
File "Abe\DataStore.py", line 487, in _execute
store.cursor.execute(stmt, params)
File "C:\Python27\lib\site-packages\MySQLdb\cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "C:\Python27\lib\site-packages\MySQLdb\connections.py", line 36, in defau
lterrorhandler
raise errorclass, errorvalue
OperationalError: (2006, 'MySQL server has gone away')
Reconnecting after rollback error: (2006, 'MySQL server has gone away')
Reconnecting to database.

The server configuration is ok as it has worked up until this point, i am forcing abe to use blkfile, however it also fails under rpc.

EDIT:
forgot to mention that under RPC i get this error:

C:\Users\Matthew\Desktop\bitcoin-abe-master>python -m Abe.abe --config abe-my-te
stnet.conf --no-serve
block_tx 54516 88372
block_tx 54517 88373
RPC service lacks full txindex
catch_up_rpc: abort
Opened C:/Users/Matthew/AppData/Roaming/Bitcoin/testnet3\blk0001.dat

Update blkfile_number

Skipping datadir /home/REDACTED/.bitcoin/blocks: [Errno 2] No such file or directory: '/home/REDACTED/.bitcoin/blocks/blk0001.dat'

new blockname is blk00001.dat in subfolder blocks of .bitcoin

Server run successfully but 0 blocks are shown at homepage

All blocks are loaded during installation:
python -m Abe.abe --config mycoin.conf --commit-bytes 100000 --no-serve

However after I run the server by:
python -m Abe.abe --config mycoin.conf

It only show block 0 and zero coin created at home page. What's wrong with my Abe settings?

Currency Code Block Time Started Age (days) Coins Created Avg Coin Age % CoinDD
Applecoin APC 0 2013-11-17 20:41:12 2013-11-17 39.8439 0

IndexError: string index out of range doing catch_up

Hello,

Everything since August was working fine. But since about 23 Oct 2013 I see some new problem:

$ PYTHONPATH=$HOME/lib/python2.7/site-packages python -m Abe.abe --config abe-pg.conf --no-serve
RPC service lacks full txindex
catch_up_rpc: abort
Opened /home/bitcoin/.bitcoin/blocks/blk00088.dat
Exception at 57956067
Failed to catch up {'blkfile_offset': 57723527, 'blkfile_number': 100088, 'chain_id': 1, 'loader': None, 'dirname': '/home/bitcoin/.bitcoin', 'id': Decimal('1')}
Traceback (most recent call last):
File "/home/bitcoin/lib/python2.7/site-packages/Abe/DataStore.py", line 2639, in catch_up
store.catch_up_dir(dircfg)
File "/home/bitcoin/lib/python2.7/site-packages/Abe/DataStore.py", line 2897, in catch_up_dir
store.import_blkdat(dircfg, ds, blkfile['name'])
File "/home/bitcoin/lib/python2.7/site-packages/Abe/DataStore.py", line 3027, in import_blkdat
store.import_block(b, chain_ids = chain_ids)
File "/home/bitcoin/lib/python2.7/site-packages/Abe/DataStore.py", line 1739, in import_block
tx['tx_id'] = store.import_and_commit_tx(tx, pos == 0)
File "/home/bitcoin/lib/python2.7/site-packages/Abe/DataStore.py", line 2261, in import_and_commit_tx
tx_id = store.import_tx(tx, is_coinbase)
File "/home/bitcoin/lib/python2.7/site-packages/Abe/DataStore.py", line 2195, in import_tx
pubkey_id = store.script_to_pubkey_id(txout['scriptPubKey'])
File "/home/bitcoin/lib/python2.7/site-packages/Abe/DataStore.py", line 2575, in script_to_pubkey_id
for opcode, data, i in deserialize.script_GetOp(script):
File "/home/bitcoin/lib/python2.7/site-packages/Abe/deserialize.py", line 247, in script_GetOp
nSize = ord(bytes[i])
IndexError: string index out of range

I know too little about python and bitcoin internals to fix it by myself. I am really surprised that I found only one similar problem: https://bitcointalk.org/index.php?PHPSESSID=0k6je1avndd9vf9o10aasu74u3&topic=22785.msg3394513#msg3394513

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.