bitcoinjs / indexd Goto Github PK

I see a ton of

zmq 866 messages lost +133ms
zmq 227 messages lost +1s
zmq 534 messages lost +152ms
zmq 205 messages lost +128ms
zmq 523 messages lost +7ms
zmq 976 messages lost +136ms
zmq 996 messages lost +329ms
zmq 425 messages lost +495ms
zmq 721 messages lost +4ms
zmq 1015 messages lost +147ms
zmq 486 messages lost +146ms
zmq 157 messages lost +294ms
zmq 1072 messages lost +478ms
zmq 624 messages lost +4ms
zmq 1178 messages lost +141ms
zmq 818 messages lost +145ms
zmq 824 messages lost +340ms
zmq 301 messages lost +166ms
zmq 231 messages lost +139ms
zmq 739 messages lost +3ms
zmq 856 messages lost +126ms

When its sycning up, I assume this is handled and is just for debug purposes or are the indexes incomplete if this appears?

We rely on ZMQ at the moment to trigger a resync, but, we could additionally encourage a poll w/ a time interval.

Otherwise, if the ZMQ socket goes down, the indexd node will cease to synchronize at all.

Easily extended... either way, its at the example/ level, not part of the module itself, but I think everyone uses example as the reference implementation at the moment as it is non-trivial.

a) Install and start bitcoind and indexd for the first time (both were syncing from scratch)
b) Last night SSH sessions to these systems dropped because notebook went to sleep mode. bitocoind was running in the background, so it didn't get interrupted.
c) This morning I connected again and started indexd (yesterday it was running in foreground so it stopped when SSH connection dropped). As shown above, indexd API is now working and showing the current, correct chainBlock, but blocksBehind is increasing (i.e. indexing is not working, blockIndex is stuck at 1211212)

root@indexd:/data/indexd/indexd-testnet# curl http://localhost:18432/status
{"chainBlock":1211215,"indexBlock":1211212,"blocksBehind":3,"ready":false}

indexd LOG file:

# tail -f LOG
2017/10/28-05:54:54.264780 7f6eb9cca700 Recovering log #15703
2017/10/28-05:55:11.497125 7f6eb9cca700 Level-0 table #15714: started
2017/10/28-05:55:16.870921 7f6eb9cca700 Level-0 table #15714: 475144539 bytes OK
2017/10/28-05:55:16.908894 7f6eb9cca700 Delete type=3 #9
2017/10/28-05:55:16.909430 7f6eb9cca700 Delete type=0 #15703

Bitcoin daemon status:

root@indexd:/data/indexd/indexd-testnet# bitcoin-cli getblockchaininfo
{
  "chain": "test",
  "blocks": 1211215,
  "headers": 1211215,
  "bestblockhash": "000000001eb9ec8f83980220f52bff577c06a0798ea77a6a2133cb2f1f212322",
  "difficulty": 1,
  "mediantime": 1509166914,
  "verificationprogress": 0.9999985607027567,
  "chainwork": "0000000000000000000000000000000000000000000000318e74502323fed131",
  "pruned": false,
  "softforks": [
    {
      "id": "bip34",
      "version": 2,
      "reject": {
        "status": true
      }
    },
    {
      "id": "bip66",
      "version": 3,
      "reject": {
        "status": true
      }
    },
    {
      "id": "bip65",
      "version": 4,
      "reject": {
        "status": true
      }
    }
  ],
  "bip9_softforks": {
    "csv": {
      "status": "active",
      "startTime": 1456790400,
      "timeout": 1493596800,
      "since": 770112
    },
    "segwit": {
      "status": "active",
      "startTime": 1462060800,
      "timeout": 1493596800,
      "since": 834624
    }
  }
}

I checked to make sure that I have just one indexd running.

# ps -ef | grep index
root      8930  8920  0 05:54 ?        00:00:00 node /usr/bin/forever index.js
root      8940  8930  0 05:54 ?        00:00:24 /usr/bin/node /indexd/index.js
root      9244  8901  0 07:18 ?        00:00:00 grep --color=auto index

Is this normal?

I stopped and started indexd and it caught up. I'll now leave it running in foreground and hopefully duplicate this problem.

root@indexd:/indexd# nodejs --version
v6.11.5
root@indexd:/indexd# npm --version
3.10.10

In some cases such as this one below, certain components use a different date format. Probably something from upstream, in which case feel free to close.

2017-10-30T14:35:27.153Z zmq:tx hashtx ef108ac19e62d4b2c3c0adfb4a8dde787a41abde0                                                                                     6e02bcab907980c41b9821f
Mon, 30 Oct 2017 14:35:27 GMT mempool { transactions: 284, inputs: 450, outputs: 505 }

Is there any code or documentation around block reorgs and how indexd handles them? I'm referring to the types of block reorgs illustrated in this technical explanation.

I'm just starting to look through the code and get the indexer running. If there are specific places in the code that handle re-orgs, please drop some links.

I did some testing and found I get slower, but still acceptable performance with very minimalist settings: just 1 RPC concurrent and only 32 RPC requests per batch (compared to 32 and 500, respectively, in https://github.com/bitcoinjs/indexd/blob/v0.8.2/example/.env#L5).

These tests from the table were done as follows:
a) Start bitcoind on testnet and let it catch up with the network
b) Delete indexd data, start indexd and stop it when it catches up
I used bitcoin 0.15.1rc1 and indexd 0.8.2. This system has plenty of resources and uses SSD.

In prior days (not on the chart) using the default indexd .env settings with bitcoind adjusted to 32 RPC threads and queue of 500, I'd get (address) index built in as little as 70 minutes, but with a lot more dropped messages and (IMO) unnecessarily high stress on bitcoind.

With my minimalist settings I also got the lowest number of missed ZMQ messages. Your mileage may vary (I didn't have any application (from indexd clients) workload) but IMO indexd is an important, but not critical service, and my lowest settings work just fine while not placing bitcoind at risk (while we know that aggressive settings can kill bitcoind).

I would suggest to change the defaults to values that aren't higher than the default related bitcoind settings, so that those do not have to be changed (so, not more than 4 RPC threads on bitcoind). I'm not sure how batch sizes are sized, but if bitcoind's default rpcworkqueue is 16, maybe RPCBATCHSIZE should be 16 as well (maybe even 8 or 4 would work well - because the initial values were so high, I spent a lot of time until I reached low values from the chart, so I haven't had a chance to try even lower values).

I'm getting the following error using the example code when starting indexd:
/root/.bitcoin/chainstate/LOCK: Resource temporarily unavailable

I'm on centos 7, same configuration has worked for me on an ubuntu server a while back. Anyone got an idea?

Probably best suited to an external module

With orphans retained to enable non-SPV clients to roll-back if required

Can someone created a simple Getting Started section in README?

I assume I need to install NodeJS, NPM, then this, and create a custom https://github.com/bitcoinjs/indexd/blob/master/test/index.js-like config file? Does that config file replace the main index.js file, or is it supposed to exist in a separate directory like here?

How to execute?
How to query the index?

Running indexd master (7c0c125) against testnet:

indexd:rpc getblock [ '00000000003ba174fd77250a04eba0e15134da0b4edfb9996bfedffdd235f97c',
  2 ] RangeError [ERR_INVALID_OPT_VALUE]: The value "250000" is invalid for option "value"
    at checkInt (buffer.js:1275:11)
    at Buffer.writeUInt16LE (buffer.js:1337:7)
    at Object.encode (/home/adam/indexd/node_modules/varstruct/lib/numbers.js:27:11)
    at Object.encode (/home/adam/indexd/node_modules/varstruct/lib/varbuffer.js:22:18)
    at /home/adam/indexd/node_modules/varstruct/lib/object.js:36:19
    at Array.forEach (<anonymous>)
    at Object.encode (/home/adam/indexd/node_modules/varstruct/lib/object.js:33:13)
    at ChainedBatch.put (/home/adam/indexd/dbwrapper.js:42:35)
    at outs.forEach (/home/adam/indexd/indexes/txo.js:52:14)
    at Array.forEach (<anonymous>) +43ms
  service RangeError [ERR_INVALID_OPT_VALUE]: The value "250000" is invalid for option "value"
  service     at checkInt (buffer.js:1275:11)
  service     at Buffer.writeUInt16LE (buffer.js:1337:7)
  service     at Object.encode (/home/adam/indexd/node_modules/varstruct/lib/numbers.js:27:11)
  service     at Object.encode (/home/adam/indexd/node_modules/varstruct/lib/varbuffer.js:22:18)
  service     at /home/adam/indexd/node_modules/varstruct/lib/object.js:36:19
  service     at Array.forEach (<anonymous>)
  service     at Object.encode (/home/adam/indexd/node_modules/varstruct/lib/object.js:33:13)
  service     at ChainedBatch.put (/home/adam/indexd/dbwrapper.js:42:35)
  service     at outs.forEach (/home/adam/indexd/indexes/txo.js:52:14)
  service     at Array.forEach (<anonymous>) +1m

I think it's choking on tx 22e08a53aff6cb0680758036e09f792dac56e4813de81c9be1b4b1ae3cda0ab0 which seems to be some sort of crazy non-standard tx. I couldn't decode it with bitcoin-cli

The RPC is the limiting factor, we could trivially de-linearize the 1st order connect calls, provided tip doesn't get confused.
tip is the only data that is "UPDATE"d.

If an attached listener service throws an exception, it should not impact other potential listeners.

Partially resolved with 22994cf, but only in terms of it does not halt resync now.

e.g https://www.npmjs.com/package/async-eventemitter

I ran out of disk space while building index, which partially corrupted my Bitcoin data.

It'd be nice if indexd had a "low disk space" setting so that we could have it auto-shutdown before it gets a chance to do damage.

What are exact steps needed to launch indexd daemon on ubuntu machine with bitcoind installed?

Just hitting some of the API endpoints in private-bitcoin to see what happens and ran into this when I visited /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block:

  express:router dispatching GET /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +1m                                                                                                   
  express:router query  : /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms                                                                                                         
  express:router expressInit  : /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms                                                                                                   
  express:router corsMiddleware  : /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms                                                                                                
  express:router router  : /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +1ms                                                                                                        
  express:router dispatching GET /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms                                                                                                  
  express:router <anonymous>  : /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms                                                                                                   
  express:router <anonymous>  : /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms                                                                                                   
  express:router trim prefix (/1) from url /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms                                                                                        
  express:router router /1 : /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms              
  express:router dispatching GET /t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms            
  express:router <anonymous>  : /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms
  api Error: Expected Object, got String "eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489"                
  api     at typeforce (/home/adam/indexd/node_modules/typeforce/index.js:198:11)                                                                                                                                  
  api     at LevelDOWN.get (/home/adam/indexd/dbwrapper.js:25:3)                                                           
  api     at TxIndex.heightBy (/home/adam/indexd/indexes/tx.js:65:6)                                     
  api     at Indexd.blockIdByTransactionId (/home/adam/indexd/index.js:260:19)                                             
  api     at router.get (/home/adam/private-bitcoin/routes/1.js:122:14)                                                    
  api     at Layer.handle [as handle_request] (/home/adam/private-bitcoin/node_modules/express/lib/router/layer.js:95:5)
  api     at next (/home/adam/private-bitcoin/node_modules/express/lib/router/route.js:137:13)                             
  api     at hexWare (/home/adam/private-bitcoin/routes/1.js:106:5)                                                                                                                                                
  api     at Layer.handle [as handle_request] (/home/adam/private-bitcoin/node_modules/express/lib/router/layer.js:95:5)   
  api     at next (/home/adam/private-bitcoin/node_modules/express/lib/router/route.js:137:13) +0ms      
  api GET /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block 400 Bad Request 3ms 0->98B +3ms  

From #11

We should instead maintain two indexd chains, one for 1st order data, and the other chain, for the 2nd order data...

This would allow us to independently roll-back the chains in the event of failure.

This would also mean we can run a direct import using fast-dat-parser and allow the fee-data to catch up.

Hello,

from the issues I read I found out perhaps I am not using the indexd module correctly. There is a standalone index server as I could read, is this a module for it?

In the previous commit (53506f0) I could run node test/index.js and the index leveldb would be populated with debug output visible on the console.

When using the latest commit I can do queries navigating to express example page, but I do not see the db being populated anymore.

Would like to ask your kind help in setting it up, or pointing me to a readme where it is noted how to run it.

I noticed this negative number of lost messages. Maybe the formula isn't completely correct.

root@indexd:/indexd# forever index.js
warn:    --minUptime not set. Defaulting to: 1000ms
warn:    --spinSleepTime not set. Your script will exit if it does not stay up for at least 1000ms
2017-10-28T12:08:15.260Z index Initializing blockchain connection (for testnet)
2017-10-28T12:08:15.350Z index starting API server
2017-10-28T12:08:15.357Z index App listening on port 18432
2017-10-28T12:23:43.160Z zmq -494 messages lost
2017-10-28T12:23:43.451Z zmq 493 messages lost

If a system failure occurs during writing of the 2nd order data (e.g out of memory/space), then upon resume the 2nd order data is skipped due to the tip update not rolling back.

index Error: No such mempool or blockchain transaction. Use gettransaction for wallet transactions.

These are reacting to bitcoind events... so not sure how that could happen.

... to a 3rd party module.

Note that a .Value type can be used now instead of the .prefix
Added in d3e6fe9

@Runn1ng I assume this is what you mean by timestamp index?
I'll push up this, as its only in my local fork of indexd atm...

Hello, my bitcoin node (v0.16.99.0-e057589) crashes without throwing an error while syncing indexd. It happens after a few hours and is really frustrating. I'm using the latest release in this server but I figured it probably is an indexd issue: https://github.com/CounterpartyXCP/indexd-server

bitcoin.conf

listen=1
rpcuser=xxxxxxxxxxxxx
rpcpassword=xxxxxxxxxxxx
rpcport=8332
rpcconnect=127.0.0.1
rpcallowip=222.22.22.22
rpcallowip=127.0.0.1
rpcallowip=22.222.222.22
zmqpubhashtx=tcp://127.0.0.1:38832
zmqpubhashblock=tcp://127.0.0.1:38832
server=1
txindex=1
addrindex=1
walletbroadcast=0
daemon=1
rpcthreads=16
rpcworkqueue=32

Thoughts @fanatid? @jprichardson

This is an alternative to insight or the like, in that it uses the mainline https://github.com/bitcoin/bitcoin RPC and software, not a modified version.

• Script Index (ala, address index)
• Spents Index (ala, inverse of unspents)
• TxOuts Index
• TxHeight Index

It still uses the txindex in bitcoin-core (as moving that out would duplicate a lot of data), and we can either use the soon-to-be-added unspents index as in bitcoin/bitcoin#9806, or we can trivially add our own (probably ideal tbh).

Hello,

I am working on a project that has many private keys that have balances in them. I do not want to have to go out to a public API to query for security concerns.

I see there is a way to get the unspent amounts for an address in the example. However, I am not seeing a way to connect the private key to it's address. I believe this is what I would have to do to determine how much BTC the private key is linked to.

Is the data structure in indexd able to perform this type of query against the blockchain?

Thanks in advance!

May or may not be suitable, but it probably could work to be honest...
Provided you only communicate with a trusted node - indexd doesn't do any blockchain validation.

Then it'd just be a matter of -addnode to connect up rather than RPC.

If your tooling uses addresses... you probably want to transcode addresses to their scId (sha256(script)):

    let bitcoin = require('bitcoinjs-lib')

    // ...
    let tasks = {}

    addrs.forEach((address) => {
      let script = bitcoin.address.toOutputScript(address)
      let scId = bitcoin.crypto.sha256(script).toString('hex')

      tasks[address] = (next) => indexd.adapter.transactionsByScriptId(scId, height, next)
    })

Then you can easily map the task results back to each address involved...

List in README? Or not necessary?

The only "extra" information returned is value, useful, but, could probably do better.
scId too?

Is that even useful without a mapping?
Do we want:

scId -> script

For reverse mapping that information?

Partially resolved by 46361a4

Unfortunately, that query doesn't return the maximum height achieved so you can re-continue the query later without starting from 0.

That should be added somehow.
For now, this prevents DoS.

Tested with https://blockchain.info/popular-addresses

This would mean we could run indexd on a pruning node, provided it is fully synchronized to start.

Just a fast question

Would it be possible to use decoderawtransaction RPC call to decode transaction instead of using bitcoin.js parsing?

The only reason why I am asking this is that I would love something that's coin-agnostic, and decoderawtransaction is in all bitcoin clones, but bitcoin.js has to be sometimes specifically ported...

TXID | SEEN_HEIGHT | UTC probably

• Change Adapter.prototype.connectBlock and Adapter.prototype.disconnectBlock transaction iteration to be done asynchronously, allowing for inner-loop asynchronous flow
• Aggregate input values for each transaction (asynchronously)
• Aggregate output values for each transaction
• let fee = inAccum - outAccum
• Uncomment fee type

Work in progress branch: https://github.com/dcousens/indexd/tree/feeindex