[Klaytn v1.7.1+86d60fe05b testnet] BAD BLOCK, invalid merkle root about klaytn

[aidan-kwon]

@jeongkyun-oh Please, take a look.

[manbobo2002]

any update?

[aidan-kwon]

@manbobo2002 I need more information to analyze your problem. Can you answer for the following questions?

• How did download a chaindata snapshot from https://packages.klaytn.net/baobab/chaindata/ or sync from the genesis block. If you download the snapshot, which date snapshot did you use?
• Did you operate the EN only with v1.7.1? If you use other versions before, what was the version?

[manbobo2002]

This is my config file:

# Configuration file for the kend

# cypress, baobab is only available if you don't specify NETWORK_ID.
# cypress is mainnet while baobab is testnet
NETWORK=baobab
# if you specify NETWORK_ID, a private network is created.
NETWORK_ID=

PORT=32323

SERVER_TYPE="fasthttp"
SYNCMODE="full"
VERBOSITY=3
MAXCONNECTIONS=10

# txpool options setting
TXPOOL_EXEC_SLOTS_ALL=4096
TXPOOL_NONEXEC_SLOTS_ALL=4096
TXPOOL_EXEC_SLOTS_ACCOUNT=4096
TXPOOL_NONEXEC_SLOTS_ACCOUNT=4096
TXPOOL_LIFE_TIME="30m"

# rpc options setting
RPC_ENABLE=1 # if this is set, the following options will be used
RPC_API="klay" # available apis: admin,debug,klay,miner,net,personal,rpc,txpool,web3
RPC_PORT=8551
RPC_ADDR="0.0.0.0"
RPC_CORSDOMAIN="*"
RPC_VHOSTS="*"

# Raw options e.g) "--txpool.nolocals"
ADDITIONAL=""

DATA_DIR={{ klaytn_config_dir }}
LOG_DIR={{ klaytn_log_dir }}

I did not download any snapshot file, just sync from zero to the latest.

yes, we just use v1.7.1, and one day it suddenly has bad block. From previous experience, when u release a patch and I update to the latest version will fix, but this time u do not have any new release

[jeongkyun-oh]

@manbobo2002 Could you please provide more logs right before the bad block is created?

[manbobo2002]

@jeongkyun-oh I am not sure this is what u want, I just restart the klaytn node and capture the logging:

########## BAD BLOCK #########
Chain config: {ChainID: 1001 IstanbulCompatibleBlock: 75373312 SubGroupSize: 22 UnitPrice: 25000000000 DeriveShaImpl: 2 Engine: istanbul}

Number: 75513601
Hash: 0x18ec244645447ab1c85b520efc51a2fe7089009d3f94fb4f977bbfe12564548c


Error: invalid merkle root (remote: 6b542929c92338f02f70e318141931631a33a4bd72ec580a9827cbed4c83f213 local: 6ecdb26bc39e8388680f37a9394b4672633c2c9031b00282f1840bdfd158d164)
##############################
 
WARN[12/02,01:48:39 Z] [28] Synchronisation failed, dropping peer     peer=26e7de5192e61953 err="retrieved hash chain is invalid: invalid merkle root (remote: 6b542929c92338f02f70e318141931631a33a4bd72ec580a9827cbed4c83f213 local: 6ecdb26bc39e8388680f37a9394b4672633c2c9031b00282f1840bdfd158d164)"
WARN[12/02,01:48:39 Z] [33] ProtocolManager failed to read msg        id=26e7de5192e61953 conn=trusted-staticdial err=EOF
ERROR[12/02,01:48:40 Z] [44] failed to update stakingInfo              blockNum=75513601 staking block number=75427200 err="failed to make a state for interval block. blockNum: 75427200, root err: missing trie node 434d12d6040b4e5a8f981cbf55035e871afffec321b3083b62b039ccc90e1fd6 (path )"
ERROR[12/02,01:48:40 Z] [5] 
########## BAD BLOCK #########
Chain config: {ChainID: 1001 IstanbulCompatibleBlock: 75373312 SubGroupSize: 22 UnitPrice: 25000000000 DeriveShaImpl: 2 Engine: istanbul}

Number: 75513601
Hash: 0x18ec244645447ab1c85b520efc51a2fe7089009d3f94fb4f977bbfe12564548c


Error: invalid merkle root (remote: 6b542929c92338f02f70e318141931631a33a4bd72ec580a9827cbed4c83f213 local: 6ecdb26bc39e8388680f37a9394b4672633c2c9031b00282f1840bdfd158d164)
##############################
 
WARN[12/02,01:48:40 Z] [28] Synchronisation failed, dropping peer     peer=026bf3c2c005375f err="retrieved hash chain is invalid: invalid merkle root (remote: 6b542929c92338f02f70e318141931631a33a4bd72ec580a9827cbed4c83f213 local: 6ecdb26bc39e8388680f37a9394b4672633c2c9031b00282f1840bdfd158d164)"
WARN[12/02,01:48:40 Z] [33] ProtocolManager failed to read msg        id=026bf3c2c005375f conn=inbound            err=EOF
INFO[12/02,01:48:47 Z] [33] Added a single channel P2P Peer           id=f10844d29dc00d91 conn=inbound            peerID=f10844d29dc00d91
ERROR[12/02,01:48:49 Z] [44] failed to update stakingInfo              blockNum=75513601 staking block number=75427200 err="failed to make a state for interval block. blockNum: 75427200, root err: missing trie node 434d12d6040b4e5a8f981cbf55035e871afffec321b3083b62b039ccc90e1fd6 (path )"
ERROR[12/02,01:48:49 Z] [5] 
########## BAD BLOCK #########
Chain config: {ChainID: 1001 IstanbulCompatibleBlock: 75373312 SubGroupSize: 22 UnitPrice: 25000000000 DeriveShaImpl: 2 Engine: istanbul}

Number: 75513601
Hash: 0x18ec244645447ab1c85b520efc51a2fe7089009d3f94fb4f977bbfe12564548c


Error: invalid merkle root (remote: 6b542929c92338f02f70e318141931631a33a4bd72ec580a9827cbed4c83f213 local: 6ecdb26bc39e8388680f37a9394b4672633c2c9031b00282f1840bdfd158d164)
##############################
 
WARN[12/02,01:48:49 Z] [28] Synchronisation failed, dropping peer     peer=f10844d29dc00d91 err="retrieved hash chain is invalid: invalid merkle root (remote: 6b542929c92338f02f70e318141931631a33a4bd72ec580a9827cbed4c83f213 local: 6ecdb26bc39e8388680f37a9394b4672633c2c9031b00282f1840bdfd158d164)"
WARN[12/02,01:48:49 Z] [33] ProtocolManager failed to read msg        id=f10844d29dc00d91 conn=inbound            err=EOF
INFO[12/02,01:49:00 Z] [33] [Dial] Add dial candidate from static nodes  id=132d7e00f49b36bc NodeType=2 ip=15.164.82.77  mainPort=32323 port=[32323]
INFO[12/02,01:49:00 Z] [33] Added a single channel P2P Peer           id=132d7e00f49b36bc conn=trusted-staticdial peerID=132d7e00f49b36bc
ERROR[12/02,01:49:04 Z] [44] failed to update stakingInfo              blockNum=75513601 staking block number=75427200 err="failed to make a state for interval block. blockNum: 75427200, root err: missing trie node 434d12d6040b4e5a8f981cbf55035e871afffec321b3083b62b039ccc90e1fd6 (path )"
ERROR[12/02,01:49:04 Z] [5] 
########## BAD BLOCK #########
Chain config: {ChainID: 1001 IstanbulCompatibleBlock: 75373312 SubGroupSize: 22 UnitPrice: 25000000000 DeriveShaImpl: 2 Engine: istanbul}

Number: 75513601
Hash: 0x18ec244645447ab1c85b520efc51a2fe7089009d3f94fb4f977bbfe12564548c


Error: invalid merkle root (remote: 6b542929c92338f02f70e318141931631a33a4bd72ec580a9827cbed4c83f213 local: 6ecdb26bc39e8388680f37a9394b4672633c2c9031b00282f1840bdfd158d164)
##############################
 
WARN[12/02,01:49:04 Z] [28] Synchronisation failed, dropping peer     peer=132d7e00f49b36bc err="retrieved hash chain is invalid: invalid merkle root (remote: 6b542929c92338f02f70e318141931631a33a4bd72ec580a9827cbed4c83f213 local: 6ecdb26bc39e8388680f37a9394b4672633c2c9031b00282f1840bdfd158d164)"
WARN[12/02,01:49:04 Z] [33] ProtocolManager failed to read msg        id=132d7e00f49b36bc conn=trusted-staticdial err=EOF
INFO[12/02,01:49:04 Z] [33] [Dial] Add dial candidate from static nodes  id=26e7de5192e61953 NodeType=2 ip=15.164.78.160 mainPort=32323 port=[32323]
INFO[12/02,01:49:04 Z] [33] Added a single channel P2P Peer           id=26e7de5192e61953 conn=trusted-staticdial peerID=26e7de5192e61953
ERROR[12/02,01:49:13 Z] [44] failed to update stakingInfo              blockNum=75513601 staking block number=75427200 err="failed to make a state for interval block. blockNum: 75427200, root err: missing trie node 434d12d6040b4e5a8f981cbf55035e871afffec321b3083b62b039ccc90e1fd6 (path )"
ERROR[12/02,01:49:13 Z] [5] 
########## BAD BLOCK #########
Chain config: {ChainID: 1001 IstanbulCompatibleBlock: 75373312 SubGroupSize: 22 UnitPrice: 25000000000 DeriveShaImpl: 2 Engine: istanbul}

Number: 75513601
Hash: 0x18ec244645447ab1c85b520efc51a2fe7089009d3f94fb4f977bbfe12564548c


Error: invalid merkle root (remote: 6b542929c92338f02f70e318141931631a33a4bd72ec580a9827cbed4c83f213 local: 6ecdb26bc39e8388680f37a9394b4672633c2c9031b00282f1840bdfd158d164)
##############################
 
WARN[12/02,01:49:13 Z] [28] Synchronisation failed, dropping peer     peer=26e7de5192e61953 err="retrieved hash chain is invalid: invalid merkle root (remote: 6b542929c92338f02f70e318141931631a33a4bd72ec580a9827cbed4c83f213 local: 6ecdb26bc39e8388680f37a9394b4672633c2c9031b00282f1840bdfd158d164)"
WARN[12/02,01:49:13 Z] [33] ProtocolManager failed to read msg        id=26e7de5192e61953 conn=trusted-staticdial err=EOF
INFO[12/02,01:49:16 Z] [33] Added a single channel P2P Peer           id=a1368465cc501e19 conn=inbound            peerID=a1368465cc501e19
WARN[12/02,01:49:22 Z] [33] ProtocolManager failed to read msg        id=a1368465cc501e19 conn=inbound            err=EOF
INFO[12/02,01:49:24 Z] [33] Added a single channel P2P Peer           id=2861a76387a8e3f6 conn=inbound            peerID=2861a76387a8e3f6
INFO[12/02,01:49:26 Z] [18] Got interrupt, shutting down...          
INFO[12/02,01:49:26 Z] [40] HTTP endpoint closed                      url=http://0.0.0.0:8551
INFO[12/02,01:49:26 Z] [40] IPC endpoint closed                       endpoint=/var/kend/data/klay.ipc
INFO[12/02,01:49:26 Z] [41] Stopping Klaytn protocol                 
WARN[12/02,01:49:26 Z] [33] ProtocolManager failed to read msg        id=53895263da2e3606 conn=inbound            err=EOF
WARN[12/02,01:49:26 Z] [33] ProtocolManager failed to read msg        id=5096494436fb172c conn=inbound            err=EOF
WARN[12/02,01:49:26 Z] [33] ProtocolManager failed to read msg        id=2861a76387a8e3f6 conn=inbound            err=EOF
INFO[12/02,01:49:26 Z] [29] Terminated insertWorker                  
WARN[12/02,01:49:26 Z] [33] ProtocolManager failed to read msg        id=9506de2482cf5468 conn=inbound            err=EOF
WARN[12/02,01:49:26 Z] [28] Synchronisation failed, dropping peer     peer=2861a76387a8e3f6 err="retrieved hash chain is invalid: syncing canceled (requested)"
WARN[12/02,01:49:26 Z] [33] ProtocolManager failed to read msg        id=30fe411ed65ad556 conn=inbound            err=EOF
WARN[12/02,01:49:26 Z] [33] ProtocolManager failed to read msg        id=8167dacb5051bed3 conn=inbound            err=EOF
WARN[12/02,01:49:26 Z] [33] ProtocolManager failed to read msg        id=5b8a303894313857 conn=inbound            err=EOF
INFO[12/02,01:49:26 Z] [41] Klaytn protocol stopped                  
INFO[12/02,01:49:26 Z] [5] Transaction pool stopped                 
INFO[12/02,01:49:26 Z] [5] Writing cached state to disk              block=75513600 hash=7bdc0f…1ae580 root=15b287…207871
INFO[12/02,01:49:26 Z] [47] Persisted trie from memory database       blockNum=75513600 updated nodes=0 updated nodes size=0.00B time=4.657µs gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO[12/02,01:49:26 Z] [5] Blockchain manager stopped               
INFO[12/02,01:49:26 Z] [46] Database closed                           path=/var/kend/data/klay/chaindata/misc
INFO[12/02,01:49:26 Z] [46] Database closed                           path=/var/kend/data/klay/chaindata/header
INFO[12/02,01:49:26 Z] [46] Database closed                           path=/var/kend/data/klay/chaindata/body
INFO[12/02,01:49:26 Z] [46] Database closed                           path=/var/kend/data/klay/chaindata/receipts
INFO[12/02,01:49:26 Z] [46] Database closed                           path=/var/kend/data/klay/chaindata/statetrie/0
INFO[12/02,01:49:26 Z] [46] Database closed                           path=/var/kend/data/klay/chaindata/statetrie/1
INFO[12/02,01:49:26 Z] [46] Database closed                           path=/var/kend/data/klay/chaindata/statetrie/2
INFO[12/02,01:49:26 Z] [46] Database closed                           path=/var/kend/data/klay/chaindata/statetrie/3
INFO[12/02,01:49:26 Z] [46] Database closed                           path=/var/kend/data/klay/chaindata/txlookup
INFO[12/02,01:49:26 Z] [46] Database closed                           path=/var/kend/data/klay/chaindata/bridgeservice
WARN[12/02,01:49:26 Z] [34] UDP read error                            err="read udp [::]:32323: use of closed network connection"
INFO[12/02,01:49:26 Z] [18] Setting connection type                   nodetype=en conntype=1
INFO[12/02,01:49:26 Z] [18] Baobab bootnodes are set                 
INFO[12/02,01:49:26 Z] [18] Bootnode - Add Seed                       Node="kni://779d766628247ebda5f3e108e9303bd8efdb8eba9fd8d6c529e2614aec7207ebf6614fe7e61d0d99b75e8b23dd3a679b112fd0de7e4e71a7008f0718710da48f@52.79.223.199:32323?ntype=bn"
INFO[12/02,01:49:26 Z] [18] Setting MaxPhysicalConnections            MaxPhysicalConnections=10
INFO[12/02,01:49:26 Z] [18] Baobab network ID is set                  networkid=1001
INFO[12/02,01:49:26 Z] [18] Raised fd limit to process's maximum value  fd=1048576
INFO[12/02,01:49:26 Z] [18] Baobab network ID is set                  networkid=1001
INFO[12/02,01:49:26 Z] [18] Archiving mode of this node               isArchiveMode=false
INFO[12/02,01:49:26 Z] [40] Starting peer-to-peer node                instance=Klaytn/v1.7.1+86d60fe05b/linux-amd64/go1.15.7
INFO[12/02,01:49:26 Z] [46] Non-single database is used for persistent storage  DBType=LevelDB
INFO[12/02,01:49:26 Z] [46] LevelDB configurations                    path=/var/kend/data/klay/chaindata/misc levelDBCacheSize=16 openFilesLimit=10485 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[12/02,01:49:26 Z] [46] LevelDB configurations                    path=/var/kend/data/klay/chaindata/header levelDBCacheSize=38 openFilesLimit=26214 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[12/02,01:49:26 Z] [46] LevelDB configurations                    path=/var/kend/data/klay/chaindata/body   levelDBCacheSize=38 openFilesLimit=26214 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[12/02,01:49:26 Z] [46] LevelDB configurations                    path=/var/kend/data/klay/chaindata/receipts levelDBCacheSize=38 openFilesLimit=26214 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[12/02,01:49:26 Z] [46] LevelDB configurations                    path=/var/kend/data/klay/chaindata/statetrie/0 levelDBCacheSize=76 openFilesLimit=52428 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[12/02,01:49:26 Z] [46] LevelDB configurations                    path=/var/kend/data/klay/chaindata/statetrie/1 levelDBCacheSize=76 openFilesLimit=52428 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[12/02,01:49:26 Z] [46] LevelDB configurations                    path=/var/kend/data/klay/chaindata/statetrie/2 levelDBCacheSize=76 openFilesLimit=52428 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[12/02,01:49:27 Z] [46] LevelDB configurations                    path=/var/kend/data/klay/chaindata/statetrie/3 levelDBCacheSize=76 openFilesLimit=52428 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[12/02,01:49:27 Z] [46] Created a sharded database                dbType=statetrie numShards=4
INFO[12/02,01:49:27 Z] [46] LevelDB configurations                    path=/var/kend/data/klay/chaindata/txlookup    levelDBCacheSize=16 openFilesLimit=10485 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[12/02,01:49:27 Z] [46] LevelDB configurations                    path=/var/kend/data/klay/chaindata/bridgeservice levelDBCacheSize=16 openFilesLimit=5242  useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[12/02,01:49:27 Z] [41] Initialised chain configuration           config="{ChainID: 1001 IstanbulCompatibleBlock: 75373312 SubGroupSize: 22 UnitPrice: 25000000000 DeriveShaImpl: 2 Engine: istanbul}"
INFO[12/02,01:49:27 Z] [30] Successfully loaded governance state from database  blockNumber=75512832
INFO[12/02,01:49:27 Z] [41] Initialising Klaytn protocol              versions=[64] network=1001
INFO[12/02,01:49:27 Z] [5] Using DeriveShaConcat!                   
INFO[12/02,01:49:27 Z] [47] Initializing local trie node cache (fastCache)  MaxMiB=9411 FilePath=/var/kend/data/fastcache
INFO[12/02,01:49:27 Z] [47] Initialized local trie node cache (fastCache)  LoadedMiB=0 LoadedEntries=0 elapsed=2.142742ms
INFO[12/02,01:49:27 Z] [5] Loaded most recent local header           number=75513600 hash=7bdc0f…1ae580 td=75513601 age=2w1d9h
INFO[12/02,01:49:27 Z] [5] Loaded most recent local full block       number=75513600 hash=7bdc0f…1ae580 td=75513601 age=2w1d9h
INFO[12/02,01:49:27 Z] [5] Loaded most recent local fast block       number=75513600 hash=7bdc0f…1ae580 td=75513601 age=2w1d9h
INFO[12/02,01:49:27 Z] [5] prefetchTxWorkers are started             num=32
INFO[12/02,01:49:27 Z] [5] Loaded local transaction journal          transactions=1 dropped=0
INFO[12/02,01:49:27 Z] [5] Regenerated local transaction journal     transactions=1 accounts=1
INFO[12/02,01:49:27 Z] [40] Starting P2P networking                  
INFO[12/02,01:49:29 Z] [33] Create udp                                config="{NetworkID:1001 PrivateKey:<nil> AnnounceAddr:[::]:32323 NodeDBPath:/var/kend/data/klay/nodes NetRestrict:<nil> Bootnodes:[kni://779d766628247ebda5f3e108e9303bd8efdb8eba9fd8d6c529e2614aec7207ebf6614fe7e61d0d99b75e8b23dd3a679b112fd0de7e4e71a7008f0718710da48f@52.79.223.199:32323?ntype=bn] Unhandled:<nil> Id:3a5986743908791f31f6c004f6a463d4c38c5dc5e36799ab4e0d6cb8f4ded6dfb933aeaccbe7283ffd9266354b2d7af2e621a9145b9bcb228013401457c13135 Addr:[::]:32323 udp:<nil> Conn:0xc002998048 NodeType:3 MaxNeighborsNode:0 AuthorizedNodes:[]}"
INFO[12/02,01:49:29 Z] [34] UDP listener up                           self="kni://3a5986743908791f31f6c004f6a463d4c38c5dc5e36799ab4e0d6cb8f4ded6dfb933aeaccbe7283ffd9266354b2d7af2e621a9145b9bcb228013401457c13135@[::]:32323?ntype=en"
INFO[12/02,01:49:29 Z] [40] Started P2P server                        id=3a5986743908791f multichannel=false
INFO[12/02,01:49:29 Z] [40] RLPx listener up                          self="kni://3a5986743908791f31f6c004f6a463d4c38c5dc5e36799ab4e0d6cb8f4ded6dfb933aeaccbe7283ffd9266354b2d7af2e621a9145b9bcb228013401457c13135@[::]:32323?ntype=en"
INFO[12/02,01:49:29 Z] [44] Start listening chain head event to update stakingInfoCache. 
INFO[12/02,01:49:29 Z] [40] IPC endpoint opened                       url=/var/kend/data/klay.ipc
INFO[12/02,01:49:29 Z] [40] FastHTTP endpoint opened                  url=http://0.0.0.0:8551     cors=* vhosts=*
ERROR[12/02,01:49:29 Z] [44] failed to update stakingInfo              blockNum=75513601 staking block number=75427200 err="failed to make a state for interval block. blockNum: 75427200, root err: missing trie node 434d12d6040b4e5a8f981cbf55035e871afffec321b3083b62b039ccc90e1fd6 (path )"
INFO[12/02,01:49:30 Z] [33] Added a single channel P2P Peer           id=5c3c5ce9550617e1 conn=inbound peerID=5c3c5ce9550617e1
INFO[12/02,01:49:33 Z] [33] [Dial] Add dial candidate from static nodes  id=132d7e00f49b36bc NodeType=2 ip=15.164.82.77 mainPort=32323 port=[32323]
INFO[12/02,01:49:33 Z] [33] [Dial] Add dial candidate from static nodes  id=26e7de5192e61953 NodeType=2 ip=15.164.78.160 mainPort=32323 port=[32323]
INFO[12/02,01:49:33 Z] [33] Added a single channel P2P Peer           id=26e7de5192e61953 conn=trusted-staticdial peerID=26e7de5192e61953
INFO[12/02,01:49:33 Z] [33] Added a single channel P2P Peer           id=132d7e00f49b36bc conn=trusted-staticdial peerID=132d7e00f49b36bc
INFO[12/02,01:49:34 Z] [33] Added a single channel P2P Peer           id=27f00addbd9ee5d0 conn=inbound            peerID=27f00addbd9ee5d0
INFO[12/02,01:49:35 Z] [28] Block synchronisation started            
INFO[12/02,01:49:35 Z] [49] Mining aborted due to sync               
INFO[12/02,01:49:35 Z] [28] Downloader queue stats                    receiptTasks=0 blockTasks=358 itemSize=780.21B throttle=8192
ERROR[12/02,01:49:35 Z] [44] failed to update stakingInfo              blockNum=75513601 staking block number=75427200 err="failed to make a state for interval block. blockNum: 75427200, root err: missing trie node 434d12d6040b4e5a8f981cbf55035e871afffec321b3083b62b039ccc90e1fd6 (path )"
ERROR[12/02,01:49:35 Z] [5] 
########## BAD BLOCK #########
Chain config: {ChainID: 1001 IstanbulCompatibleBlock: 75373312 SubGroupSize: 22 UnitPrice: 25000000000 DeriveShaImpl: 2 Engine: istanbul}

Number: 75513601
Hash: 0x18ec244645447ab1c85b520efc51a2fe7089009d3f94fb4f977bbfe12564548c


Error: invalid merkle root (remote: 6b542929c92338f02f70e318141931631a33a4bd72ec580a9827cbed4c83f213 local: 6ecdb26bc39e8388680f37a9394b4672633c2c9031b00282f1840bdfd158d164)
##############################
 
WARN[12/02,01:49:35 Z] [28] Synchronisation failed, dropping peer     peer=26e7de5192e61953 err="retrieved hash chain is invalid: invalid merkle root (remote: 6b542929c92338f02f70e318141931631a33a4bd72ec580a9827cbed4c83f213 local: 6ecdb26bc39e8388680f37a9394b4672633c2c9031b00282f1840bdfd158d164)"
WARN[12/02,01:49:35 Z] [33] ProtocolManager failed to read msg        id=26e7de5192e61953 conn=trusted-staticdial err=EOF
ERROR[12/02,01:49:36 Z] [44] failed to update stakingInfo              blockNum=75513601 staking block number=75427200 err="failed to make a state for interval block. blockNum: 75427200, root err: missing trie node 434d12d6040b4e5a8f981cbf55035e871afffec321b3083b62b039ccc90e1fd6 (path )"
ERROR[12/02,01:49:36 Z] [5] 
########## BAD BLOCK #########
Chain config: {ChainID: 1001 IstanbulCompatibleBlock: 75373312 SubGroupSize: 22 UnitPrice: 25000000000 DeriveShaImpl: 2 Engine: istanbul}

Number: 75513601
Hash: 0x18ec244645447ab1c85b520efc51a2fe7089009d3f94fb4f977bbfe12564548c


Error: invalid merkle root (remote: 6b542929c92338f02f70e318141931631a33a4bd72ec580a9827cbed4c83f213 local: 6ecdb26bc39e8388680f37a9394b4672633c2c9031b00282f1840bdfd158d164)
##############################
 
WARN[12/02,01:49:36 Z] [28] Synchronisation failed, dropping peer     peer=132d7e00f49b36bc err="retrieved hash chain is invalid: invalid merkle root (remote: 6b542929c92338f02f70e318141931631a33a4bd72ec580a9827cbed4c83f213 local: 6ecdb26bc39e8388680f37a9394b4672633c2c9031b00282f1840bdfd158d164)"
WARN[12/02,01:49:36 Z] [33] ProtocolManager failed to read msg        id=132d7e00f49b36bc conn=trusted-staticdial err=EOF
WARN[12/02,01:49:36 Z] [28] Synchronisation failed, dropping peer     peer=27f00addbd9ee5d0 err="action from bad peer ignored"
WARN[12/02,01:49:36 Z] [33] ProtocolManager failed to read msg        id=27f00addbd9ee5d0 conn=inbound            err=EOF
ERROR[12/02,01:49:39 Z] [44] failed to update stakingInfo              blockNum=75513601 staking block number=75427200 err="failed to make a state for interval block. blockNum: 75427200, root err: missing trie node 434d12d6040b4e5a8f981cbf55035e871afffec321b3083b62b039ccc90e1fd6 (path )"
ERROR[12/02,01:49:39 Z] [5] 
########## BAD BLOCK #########
Chain config: {ChainID: 1001 IstanbulCompatibleBlock: 75373312 SubGroupSize: 22 UnitPrice: 25000000000 DeriveShaImpl: 2 Engine: istanbul}

Number: 75513601
Hash: 0x18ec244645447ab1c85b520efc51a2fe7089009d3f94fb4f977bbfe12564548c


Error: invalid merkle root (remote: 6b542929c92338f02f70e318141931631a33a4bd72ec580a9827cbed4c83f213 local: 6ecdb26bc39e8388680f37a9394b4672633c2c9031b00282f1840bdfd158d164)
##############################
 
WARN[12/02,01:49:39 Z] [28] Synchronisation failed, dropping peer     peer=5c3c5ce9550617e1 err="retrieved hash chain is invalid: invalid merkle root (remote: 6b542929c92338f02f70e318141931631a33a4bd72ec580a9827cbed4c83f213 local: 6ecdb26bc39e8388680f37a9394b4672633c2c9031b00282f1840bdfd158d164)"
WARN[12/02,01:49:39 Z] [33] ProtocolManager failed to read msg        id=5c3c5ce9550617e1 conn=inbound            err=EOF

[jeongkyun-oh]

@manbobo2002 Thanks for the logs. We expect data corruption, but the given logs are not enough to check unfortunately. So, if it is possible, could you please upload the logs from when the bad block is created at the very first till the recent? It would be better if you give a google drive URL or something else. Thank you for your help in advance.

[manbobo2002]

@jeongkyun-oh the above log already included stop and start, but I am not sure when it stucks, could you please give us any suggestion?

[jeongkyun-oh]

@manbobo2002 Yes, I meant when the very first bad block is created, not when the node is stopped and restarted. I want to know why the data is corrupted. Or, can you provide the full log file via google drive if possible?

[manbobo2002]

@jeongkyun-oh I just rebuild a new testnet node and works fine, closed this ticket

[jeongkyun-oh]

@manbobo2002 Thanks for reporting. But, we would like to investigate your issue, so please attach the log file or link if possible. Thank you.

[manbobo2002]

@jeongkyun-oh Unfortunately we did not keep those records