attempt at adding ppcoin about bitcoin-abe

[jtobey]

From browsing the ppcoin source code, I think its developer changed the block format slightly by adding a "block signature" after the transactions: https://github.com/ppcoin/ppcoin/blob/09051bd53858adbb0ed5564434e88e0dc8a733cf/src/main.h#L887

This change is similar to the "auxiliary proof of work" implemented by Namecoin and others. I suspect this patch will make Abe able to load ppcoin, but it will break all the others: http://pastebin.com/qyAm5RjN

A good fix will take some work. It will probably involve a means of telling Abe that a given datadir contains blockfiles in ppcoin format. A more general solution might involve scanning forward in the block file to the next occurrence of the chain's "magic number".

[grimd34th]

still getting the same error

[jtobey]

Committed a change that may work without configuration requirements. You would have to reset the datadir coordinates before testing either fix to force a rescan: UPDATE datadir SET blkfile_number=1, blkfile_offset=0

[jtobey]

I downloaded the PPCoin blockchain (using v0.2.1ppc-beta) and found the following change necessary to load it. This renders Abe unusable for other currencies, so it can not go in as is.

diff --git a/Abe/deserialize.py b/Abe/deserialize.py
index ecef9de..781591f 100644
--- a/Abe/deserialize.py
+++ b/Abe/deserialize.py
@@ -80,6 +80,7 @@ def parse_Transaction(vds):
   d = {}
   start = vds.read_cursor
   d['version'] = vds.read_int32()
+  d['nTime'] = vds.read_uint32()
   n_vin = vds.read_compact_size()
   d['txIn'] = []
   for i in xrange(n_vin):

[grimd34th]

works, maybe an if statement for the ppcoin version number to add that in?

[c4n10]

Where would you put this code to make it work?

[jtobey]

Add a line with " d['nTime'] = vds.read_uint32()" (including the two leading spaces!) after line 82 in Abe/deserialize.py. I've created a ppcoin branch with this change, so you can get it with "git checkout ppcoin" in the bitcoin-abe directory.

[c4n10]

Thanks!

This is now producing a new error:

Exception at 103
Failed to catch up {'blkfile_number': 1, 'dirname': u'/root/.rucoin', 'chain_id': Decimal('8'), 'id': 2L, 'blkfile_offset': 0}
Traceback (most recent call last):
File "Abe/DataStore.py", line 2403, in catch_up
store.catch_up_dir(dircfg)
File "Abe/DataStore.py", line 2456, in catch_up_dir
store.import_blkdat(dircfg, ds, filename[0])
File "Abe/DataStore.py", line 2580, in import_blkdat
store.import_block(b, chain_ids = chain_ids)
File "Abe/DataStore.py", line 1656, in import_block
raise MerkleRootMismatch(b['hash'], tx_hash_array)
MerkleRootMismatch: Block header Merkle root does not match its transactions. block hash=2c3c481f87541ea4a933755aacfc0ab0bf54b693047de94be34aef8800000000

We are trying to do a block explorer for RUC if that helps any...

[jtobey]

Perhaps RUC has a different block header structure from Bitcoin. Please find the definition of class CBlockHeader in the RUC sources, and post a link. In Bitcoin, it starts in src/main.h at line 1257.

If the field types or the number or order of READWRITE lines differs in RUC, we may have only to copy the differences to parse_BlockHeader in deserialize.py.

If this keeps up, I guess we could develop a JSON format for the structure and put it in abe.conf.

[c4n10]

Ok, cblockheader looks like:

class CBlockHeader
{
public:
// header
static const int CURRENT_VERSION=2;
int nVersion;
uint256 hashPrevBlock;
uint256 hashMerkleRoot;
unsigned int nTime; 
unsigned int nBits;
unsigned int nNonce;

CBlockHeader()
{
SetNull();
}

IMPLEMENT_SERIALIZE
(
READWRITE(this->nVersion);
nVersion = this->nVersion;
READWRITE(hashPrevBlock);
READWRITE(hashMerkleRoot);
READWRITE(nTime);
READWRITE(nBits);
READWRITE(nNonce);
)

One of the RuCoin devs told us:

"when parsing blockhain file blk*.dat you need to skip additional data, that goes after vtxs: aux and blocksig streams, detected by this version flags:

// Block version bits
enum
{
// primary block version
BLOCK_VERSION_END = (7 << 0),

// modifiers
BLOCK_VERSION_AUXPOW = (1 << 8),
BLOCK_VERSION_STAKE = (1 << 9),
BLOCK_VERSION_TRUSTED = (1 << 10),
BLOCK_VERSION_STOCK = (1 << 11),

BLOCK_VERSION_SIGNED = (1 << 12),
BLOCK_VERSION_EXTENDED = (1 << 13),
BLOCK_VERSION_RETARGET = (1 << 15),
// bits allocated for chain ID
BLOCK_VERSION_CHAIN_START = (1 << 16),
BLOCK_VERSION_CHAIN_END = (1 << 30)
};

and he said:

"Skip the aux data like as in namecoin parser and signature like in ppcoin parser. I try to mod the abe myself, but later. ...also, you may skip all this data simply by searching next blockstart magic..."

[jtobey]

Thanks for the detailed info.

Searching for the next occurrence of the magic number would be a nice feature and not too hard to add.

But I doubt that that will completely solve this problem. According to the output you quoted, the error occurred 103 bytes into the file, and Abe had parsed what looked like a complete block, including the transaction array, vtxs. (That occurs before the Merkle root check, which raised the exception.) It could not have been the second block, because each block is over 80 bytes long, and 103 < 2*80. Given a 4-byte magic number, 4-byte block length, 80-byte header, and 1-byte nTransactions, I count only 14 bytes for the coinbase transaction, whose format is described here. Given 10 bytes of overhead within the tx, that leaves 4 bytes, not enough for even one transaction input or output. This leads me to think that the format changed somewhere before the end of vtxs. Namecoin aux data is not after vtxs as the developer states but between the block header and vtxs. I guess this is true of RUC's new structures, and Abe will have to detect and skip over them, too.

I would like to look more closely but can not promise to do so soon.