External bticoind not tracking new blocks about samourai-dojo

[dmp1ce]

@4NobleTruths In my external bitcoind I have rpcbind=0.0.0.0 set in bitcoin.conf. I couldn't find a way around that, but my bitcoind RPC port is only exposed on my LAN so I don't think it is a problem.

I also have internal processes from a Docker Compose connecting to bitcoind using the IP 127.0.0.1.

[LaurentMT]

The fact that Dojo doesn't seem able to process the mempool while being able to sync the blocks when it's launched suggests an issue with the configuration of ZeroMQ notifications, either in your bitcoin.conf or in your docker-bitcoind.conf.

Can you check the values set for these 2 properties in your bitcoin.conf file?

• zmqpubhashblock
• zmqpubrawtx

If I'm correct, you should have something like this:

• zmqpubhashblock=tcp://172.17.0.1:18501
• zmqpubrawtx=tcp://172.17.0.1:18502

Another possible cause is a firewall blocking the port 18502.

[dmp1ce]

This is what I thought, but I've double checked bitcoind and it looks like the ports are right. I don't think there is a firewall or routing issue because RPC port 8332 is accessible.

Here is what bitcoin-cli gives me:

$ bitcoin-cli getzmqnotifications
[
  {
    "type": "pubrawblock",
    "address": "tcp://127.0.0.1:18501",
    "hwm": 1000
  },
  {
    "type": "pubrawtx",
    "address": "tcp://127.0.0.1:18502",
    "hwm": 1000
  }
]

Is there a way I can test the ZMQ ports are working?

[dmp1ce]

In my bitcoin.conf I had

zmqpubrawblock=tcp://127.0.0.1:18501
zmqpubrawtx=tcp://127.0.0.1:18502

but I needed

zmqpubrawblock=tcp://0.0.0.0:18501
zmqpubrawtx=tcp://0.0.0.0:18502

because I'm exposing the ports outside of my docker container. Networking issues get me every time! I'm surprised Whirlpool and Samourai worked as well as they did without any new transaction or block information!

[dmp1ce]

After about 1 hour of processing I again got Processing inactive Mempool (0 transactions)! Why would this happen? Here is what my logs looked like.

[20191217 20:59:36.383 091 MiB] Processing active Mempool (41 transactions)
[20191217 20:59:46.384 096 MiB] Processing active Mempool (33 transactions)
[20191217 20:59:56.388 084 MiB] Processing active Mempool (38 transactions)
[20191217 21:00:06.389 086 MiB] Processing active Mempool (32 transactions)
[20191217 21:00:16.387 083 MiB] Processing active Mempool (74 transactions)
[20191217 21:00:26.388 084 MiB] Processing active Mempool (41 transactions)
[20191217 21:00:36.393 088 MiB] Processing active Mempool (39 transactions)
[20191217 21:00:46.397 092 MiB] Processing active Mempool (41 transactions)
[20191217 21:00:56.398 094 MiB] Processing active Mempool (29 transactions)
[20191217 21:01:06.403 084 MiB] Processing active Mempool (36 transactions)
[20191217 21:01:16.405 084 MiB] Processing active Mempool (30 transactions)
[20191217 21:01:26.406 084 MiB] Processing active Mempool (42 transactions)
[20191217 21:01:36.407 084 MiB] Processing active Mempool (20 transactions)
[20191217 21:01:46.407 084 MiB] Processing active Mempool (35 transactions)
[20191217 21:01:56.409 084 MiB] Processing active Mempool (29 transactions)
[20191217 21:02:05.672 086 MiB] Processing unconfirmed transactions
[20191217 21:02:05.696 087 MiB]  Finished processing unconfirmed transactions 0.0s, 4 tx, 6ms/tx
[20191217 21:02:06.409 087 MiB] Processing active Mempool (31 transactions)
[20191217 21:02:16.407 089 MiB] Processing active Mempool (43 transactions)
[20191217 21:02:26.409 091 MiB] Processing active Mempool (32 transactions)
[20191217 21:02:36.408 093 MiB] Processing active Mempool (30 transactions)
[20191217 21:02:46.417 095 MiB] Processing active Mempool (24 transactions)
[20191217 21:02:56.421 085 MiB] Processing active Mempool (23 transactions)
[20191217 21:03:06.422 085 MiB] Processing active Mempool (42 transactions)
[20191217 21:03:16.424 085 MiB] Processing active Mempool (43 transactions)
[20191217 21:03:26.422 085 MiB] Processing active Mempool (41 transactions)
[20191217 21:03:36.429 085 MiB] Processing active Mempool (29 transactions)
[20191217 21:03:46.430 086 MiB] Processing active Mempool (38 transactions)
[20191217 21:03:56.430 089 MiB] Processing active Mempool (30 transactions)
[20191217 21:04:06.431 092 MiB] Processing inactive Mempool (2935 transactions)
[20191217 21:04:16.435 154 MiB] Processing inactive Mempool (0 transactions)
[20191217 21:04:26.435 154 MiB] Processing inactive Mempool (0 transactions)
[20191217 21:04:36.433 154 MiB] Processing inactive Mempool (0 transactions)
[20191217 21:04:46.439 154 MiB] Processing inactive Mempool (0 transactions)
[20191217 21:04:56.439 154 MiB] Processing inactive Mempool (0 transactions)
[20191217 21:05:06.441 154 MiB] Processing inactive Mempool (0 transactions)
[20191217 21:05:16.450 076 MiB] Processing inactive Mempool (0 transactions)
[20191217 21:05:26.449 076 MiB] Processing inactive Mempool (0 transactions)
[20191217 21:05:36.449 076 MiB] Processing inactive Mempool (0 transactions)
[20191217 21:05:46.450 076 MiB] Processing inactive Mempool (0 transactions)
[20191217 21:05:56.448 076 MiB] Processing inactive Mempool (0 transactions)
[20191217 21:06:06.451 076 MiB] Processing inactive Mempool (0 transactions)
[20191217 21:06:16.451 076 MiB] Processing inactive Mempool (0 transactions)
[20191217 21:06:26.454 077 MiB] Processing inactive Mempool (0 transactions)
[20191217 21:06:36.457 076 MiB] Processing inactive Mempool (0 transactions)

What should I do to try and debug this?

[dmp1ce]

It ran for about one and a half hours last night and then back to inactive mempool. I'm going to restart again now.

[dmp1ce]

After restarting this morning, I got inactive mempool after only about 40 minutes of processing transactions. I would like to know why this is happening. I'm happy to debug further on my setup if you can give me some advice.

[dmp1ce]

I think I finally figured you my issue. I was missing zmqpubhashblock! I had zmqpubrawblock instead.

Make sure the logs show Processing active Mempool (xx transactions) and Block #xxxxxx 0000000xxxxxxxxxxx from time to time, or the dojo is probably are not listening to the correct ZMQ ports.

[4NobleTruths]

Thanks for these tips @dmp1ce !
What should the rpcbind in bitcoin.conf be for this to work, if there are already other rpc clients using rpcbind=127.0.0.1?
Did you put rpcbind=172.17.0.1 also?

Is the docker0 ip not supposed to work for the zmq also, and has to be set 0.0.0.0?

[Oberfeldwedler]

I ran into the same Issue. FULL NODE and INDEXER stay synced while TRACKER and DOJO DB fall behind.
My Issue was that I had

BITCOIND_ZMQ_RAWTXS=<IP>:18502
BITCOIND_ZMQ_BLK_HASH=<IP>:18501

instead of

BITCOIND_ZMQ_RAWTXS=18502
BITCOIND_ZMQ_BLK_HASH=18501

Also in my bitcoin.conf i had

zmqpubrawblock=<IP>:18501

instead of

zmqpubrawblock=tcp://<IP>:18501

Just wanted to leave this here in case somebody has the same issue.