cosmos-gaminghub / testnets Goto Github PK

Summary of Bug

nibiru version
neuron-1.1

• cosmos-sdk v0.44.2
• tendermint v0.34.12

Environment
ubuntu 20.04
6 vCore/16 GB

What happened

• My node crashed with the error below soon after restarting node
• Restart succeeded only when the block height was under 1000 with no peer info in address.json
• Even using same version peer (seeds and persistent_peers), still crashed
• Using synced data from other node, it still crashed after the command nibirud start
• Only the solution for this issue is re sync from scratch with the command nibirud unsafe-reset-all

Logs

panic: Failed to process committed block (15253:313A220A98EB569C01B7BCEF20F9E25AAB761C586D9E17617FCEF95CE8EF3A12): wrong Block.Header.LastResultsHash.  Expected 1E5CCA645833276B4ED18012F98634C8A7CC7661F5D4D1C0796703E9B843F2DD, got 455F0B1ADF97CF95C59C1D6D037B4B6A9160ED3CBF209F12FB517658291FE20F
Oct 25 04:46:48 neuron-dev-2 nibirud[4410]: goroutine 163 [running]:
Oct 25 04:46:48 neuron-dev-2 nibirud[4410]: github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc0002348c0, 0x0)
Oct 25 04:46:48 neuron-dev-2 nibirud[4410]:         /root/go/pkg/mod/github.com/tendermint/[email protected]/blockchain/v0/reactor.go:401 +0x1265
Oct 25 04:46:48 neuron-dev-2 nibirud[4410]: created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
Oct 25 04:46:48 neuron-dev-2 nibirud[4410]:         /root/go/pkg/mod/github.com/tendermint/[email protected]/blockchain/v0/reactor.go:110 +0x85
Oct 25 04:46:48 neuron-dev-2 systemd[1]: nibirud.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

Possible cause

This error display when call replay block, after that try to apply this block and get case that block.LastResultsHash not equal with state.LastResultsHash: see detail: https://github.com/tendermint/tendermint/blob/d030cddca01c0c3ff0ce41e051e123cdc9872a4e/consensus/replay.go#L404-L424

Possible solution

Rollback to [the crashed height - 1] block in both app state side and tendermint state side discussed in the related issues.

Related Issues

Tendermint
Add command to roll-back a single block #3845
Cosmos-SDK
Add rollback support in the event of an incorrect hash #10281
Terra
[BUG] terrad cannot start after Columbus 5 Upgrade #582

Effect to the testnet

Due to the difficulty of restart node, Neuron Incentivized Testnet Mission 6 (Upgrade Node) will be skipped.

How to configure validator node setting without restart in the rest of testnet period
In order to reflect the change in config.toml, it is necessary to restart the node. However, restarting the node causes it to crash, which means that the node needs to be resynchronized from the beginning, leading to missing the signatures.

One idea is to move the private key of the original validator from server A to another server B, rewrite server B's config.toml to synchronize the blocks until just before it catches up with the latest block, and stop the process of server A's node at that time.　Obviously, this involves the risk of double signatures, and should be done at your own risk.

Effect to the mainnet

Currently the rollback feature is implemented only tendermint(v0.34.14) side. In order for the rollback to work flawlessly. we have to wait until this issue resolved in app state side.

{"body":{"messages":[{"@type":"/cosmos.staking.v1beta1.MsgCreateValidator","description":{"moniker":"Gunslair-Stakersnodes","identity":"","website":"","security_contact":"","details":""},"commission":{"rate":"0.100000000000000000","max_rate":"1.000000000000000000","max_change_rate":"0.100000000000000000"},"min_self_delegation":"1","delegator_address":"game156w7czrdhy97nwsxx0evw93akmxg650eqjvraz","validator_address":"gamevaloper156w7czrdhy97nwsxx0evw93akmxg650esmyw9c","pubkey":{"@type":"/cosmos.crypto.ed25519.PubKey","key":"/yNYuDVE91/+7evEeOSc5aJatMEUxW2diyC16nD/ceQ="},"value":{"denom":"ugame","amount":"100000000000"}}],"memo":"[email protected]:26656","timeout_height":"0","extension_options":[],"non_critical_extension_options":[]},"auth_info":{"signer_infos":[{"public_key":{"@type":"/cosmos.crypto.secp256k1.PubKey","key":"A6um3sJwpjC2YhzOnFPQ2PlFPBg2monRHrds4EW3axbe"},"mode_info":{"single":{"mode":"SIGN_MODE_DIRECT"}},"sequence":"0"}],"fee":{"amount":[],"gas_limit":"200000","payer":"","granter":""}},"signatures":["xK3/K5FuZHKp+urC/qI+7C+Rg50r5sOt+eztkFHphoR9gE2FaasAU0JcMI/TPOCgbsdBLGS/SOYYXqycMq1Rhw=="]}

overview

add docs about nibiru-3000

• testnet purpose
• testnet timeline
• gentx process

Overview

nibiru-2000 was planned to start from August 28, 2021 11:00 GMT, but some gentxs has wrong contents so that chain crashed with the error message below.

panic: signature verification failed; please verify account number (0) and chain-id (nibiru-2000): unauthorized

Steps to Reproduce

Create Gentx

When creating gentx, it must include correct chain-id, or the panic will occur.

# gentx with wrong chain-id
nibirud gentx eg 100000000000game --commission-rate=0.1 --commission-max-rate=1 --commission-max-change-rate=0.1 --pubkey $(nibirud tendermint show-validator) --chain-id=testchain

{
  "body": {
    "messages": [
      {
        "@type": "/cosmos.staking.v1beta1.MsgCreateValidator",
        "description": {
          "moniker": "eg",
          "identity": "",
          "website": "",
          "security_contact": "",
          "details": ""
        },
        "commission": {
          "rate": "0.100000000000000000",
          "max_rate": "1.000000000000000000",
          "max_change_rate": "0.100000000000000000"
        },
        "min_self_delegation": "1",
        "delegator_address": "nibiru1ltn3338k29t8gcmmysmfq3e3mfc246p0n038cz",
        "validator_address": "nibiruvaloper1ltn3338k29t8gcmmysmfq3e3mfc246p0wpy4eq",
        "pubkey": {
          "@type": "/cosmos.crypto.ed25519.PubKey",
          "key": "r7vI8FmhWJcN3W+GedAaAtx8VSmFvB5CxLlN57QxoxA="
        },
        "value": {
          "denom": "game",
          "amount": "100000000000"
        }
      }
    ],
    "memo": "[email protected]:26656",
    "timeout_height": "0",
    "extension_options": [],
    "non_critical_extension_options": []
  },
  "auth_info": {
    "signer_infos": [
      {
        "public_key": {
          "@type": "/cosmos.crypto.secp256k1.PubKey",
          "key": "A+9jKJn7mg9V3fLJqFhq1STtYhUnzfLej6NeYSGdCf2j"
        },
        "mode_info": {
          "single": {
            "mode": "SIGN_MODE_DIRECT"
          }
        },
        "sequence": "0"
      }
    ],
    "fee": {
      "amount": [],
      "gas_limit": "200000",
      "payer": "",
      "granter": ""
    }
  },
  "signatures": [
    "AABcMs3CPeE2ynAvtYolXZ1wlNjYFyeW55R3GLNY5EskXKhkeMLC2ZDsfg4vL1FzfZ6HrZmDqFkpveeEVELiZw=="
  ]
}

# gentx with correct chain-id
nibirud gentx eg 100000000000game --commission-rate=0.1 --commission-max-rate=1 --commission-max-change-rate=0.1 --pubkey $(nibirud tendermint show-validator) --chain-id=nibiru-2000

{
  "body": {
    "messages": [
      {
        "@type": "/cosmos.staking.v1beta1.MsgCreateValidator",
        "description": {
          "moniker": "eg",
          "identity": "",
          "website": "",
          "security_contact": "",
          "details": ""
        },
        "commission": {
          "rate": "0.100000000000000000",
          "max_rate": "1.000000000000000000",
          "max_change_rate": "0.100000000000000000"
        },
        "min_self_delegation": "1",
        "delegator_address": "nibiru1ltn3338k29t8gcmmysmfq3e3mfc246p0n038cz",
        "validator_address": "nibiruvaloper1ltn3338k29t8gcmmysmfq3e3mfc246p0wpy4eq",
        "pubkey": {
          "@type": "/cosmos.crypto.ed25519.PubKey",
          "key": "r7vI8FmhWJcN3W+GedAaAtx8VSmFvB5CxLlN57QxoxA="
        },
        "value": {
          "denom": "game",
          "amount": "100000000000"
        }
      }
    ],
    "memo": "[email protected]:26656",
    "timeout_height": "0",
    "extension_options": [],
    "non_critical_extension_options": []
  },
  "auth_info": {
    "signer_infos": [
      {
        "public_key": {
          "@type": "/cosmos.crypto.secp256k1.PubKey",
          "key": "A+9jKJn7mg9V3fLJqFhq1STtYhUnzfLej6NeYSGdCf2j"
        },
        "mode_info": {
          "single": {
            "mode": "SIGN_MODE_DIRECT"
          }
        },
        "sequence": "0"
      }
    ],
    "fee": {
      "amount": [],
      "gas_limit": "200000",
      "payer": "",
      "granter": ""
    }
  },
  "signatures": [
    "8p4AR91tlNk4ZH3idx7Dmk4xBWXYN3JySoGNHA2DkdE3YUxHDo+Z8aE/Myd8ChmZ5yASdXdEpr1pOx/L0uBDmw=="
  ]
}

Only "signatures" value is different!

Create genesis.json and Start

NETWORK=nibiru-2000
DAEMON=nibirud
HOME_DIR=~/.nibiru
CONFIG=~/.nibiru/config
TOKEN_DENOM=game

$DAEMON init $NETWORK --chain-id $NETWORK

rm -rf $CONFIG/gentx && mkdir $CONFIG/gentx

sed -i "s/\"stake\"/\"$TOKEN_DENOM\"/g" $HOME_DIR/config/genesis.json

for i in $NETWORK/gentxs/*.json; do
  echo $i
  $DAEMON add-genesis-account $(jq -r '.body.messages[0].delegator_address' $i) 100000000000$TOKEN_DENOM
  cp $i $CONFIG/gentx/
done

$DAEMON collect-gentxs

$DAEMON validate-genesis

$DAEMON start

Result

if there is gentx with wrong chain-id, then that panic will occur. Every gentx should include correct chain-id for the genesis launch.

Improvement

• Remove invalid gentx
  I checked all gentx one by one. And rm the gentxs. If you are the one, please submit again.
• Add test code for merging gentx PR
  Use the github workflow to automatically check the gentx is correct.
• Check genesis.json over and over again
  After the end of gentx collecting period, we should leave 1 day before genesis time.