0glabs / 0g-storage-node Goto Github PK

Upgrading or install fresh does not fix the problem

After running for a couple of minutes the blocks does not proceed and continuously spam this message

2024-05-24T20:21:35.165437Z WARN log_entry_sync::sync_manager::log_entry_fetcher: send LogFetchProgress failed: SendError { .. }
2024-05-24T20:21:35.165483Z DEBUG log_entry_sync::sync_manager::log_entry_fetcher: log sync gets entries without progress? old_progress=289193
2024-05-24T20:21:35.667057Z DEBUG log_entry_sync::sync_manager::log_entry_fetcher: from block number 289193, latest block number 289245, confirmation delay

This is Albert from BlockPI Network. We just found that there is a bug with the storage node. Resulting in the error is “Failed to start zgs node”,
Here is the log:

2024-05-10T08:27:46.110481Z ERROR network::discovery: Error getting mapping to ENR multiaddr=/ip4/54.219.26.22/udp/1234/p2p/16Uiu2HAmPxGNWu9eVAQPJww79J32pTJL KGcpjRMb4Qb8xxKkyuG1 error=Timeout
2024-05-10T08:27:46.110485Z ERROR network::discovery: Error getting mapping to ENR multiaddr=/ip4/52.52.127.117/udp/1234/p2p/16Uiu2HAm93Hd5azfhkGBbkx1zero3nY HvfjQYM2NtiW4R3r5bE2g error=Timeout
2024-05-10T08:27:46.110489Z DEBUG network::discovery: Starting a peer discovery request target_peers=16
2024-05-10T08:27:46.110700Z  INFO network::service: Listening established address=/ip4/0.0.0.0/tcp/12092/p2p/16Uiu2HAm8jKGN6jpceAJWZbQfHXBTbBQxAC4xN7pBBYnDRa s52Wb
2024-05-10T08:27:46.110722Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/find_file/ssz_snappy
2024-05-10T08:27:46.110725Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/find_chunks/ssz_snappy
2024-05-10T08:27:46.110726Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/announce_file/ssz_snappy
2024-05-10T08:27:46.110728Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/announce_chunks/ssz_snappy
2024-05-10T08:27:46.110729Z  INFO network::service: Subscribed to topics topics=[FindFile, FindChunks, AnnounceFile, AnnounceChunks]
2024-05-10T08:27:46.110757Z DEBUG sync::service: Starting sync service
2024-05-10T08:27:46.112216Z DEBUG miner::miner_id: Checking miner id on chain...
2024-05-10T08:27:46.114062Z ERROR zgs_node: Failed to start zgs node reason=Fail to query miner id information: AbiError(DecodingError(InvalidName("please en sure the contract and method you're calling exist! failed to decode empty bytes. if you're using jsonrpc this is likely due to jsonrpc returning `0x` in case  contract or method don't exist")))
2024-05-10T08:27:46.114081Z  INFO discv5::service: Discv5 Service shutdown
2024-05-10T08:27:46.114084Z  INFO zgs_node::client::environment: Internal shutdown received reason="Failed to start zgs node"
2024-05-10T08:27:46.114093Z  INFO zgs_node: Shutting down... reason=Failure("Failed to start zgs node")

My config

zgs_uploadSegments() method just deployed but cannot publish message because of lack of peers

2024-06-03T14:02:31.301648Z DEBUG rpc::zgs::r#impl: zgs_uploadSegments()
2024-06-03T14:02:31.301703Z DEBUG rpc::zgs::r#impl: putSegment root=0x50a3…fc86 index=0
2024-06-03T14:02:31.302035Z DEBUG rpc::types: data len: 262144, expected len: 262144
2024-06-03T14:02:31.313268Z DEBUG chunk_pool::mem_pool::chunk_pool_inner: Begin to write segment, root=0x50a3…fc86, segment_size=262144, segment_index=0
2024-06-03T14:02:31.319072Z DEBUG chunk_pool::mem_pool::chunk_write_control: Succeeded to write segment, root=0x50a3…fc86, seg_index=0, total_writings=0
2024-06-03T14:02:31.319452Z DEBUG rpc::zgs::r#impl: putSegment root=0x50a3…fc86 index=1
2024-06-03T14:02:31.319788Z DEBUG rpc::types: data len: 262144, expected len: 262144
2024-06-03T14:02:31.325015Z DEBUG chunk_pool::mem_pool::chunk_pool_inner: Begin to write segment, root=0x50a3…fc86, segment_size=262144, segment_index=1
2024-06-03T14:02:31.328299Z DEBUG chunk_pool::mem_pool::chunk_write_control: Succeeded to write segment, root=0x50a3…fc86, seg_index=1, total_writings=0
2024-06-03T14:02:31.328342Z DEBUG rpc::zgs::r#impl: putSegment root=0x50a3…fc86 index=2
2024-06-03T14:02:31.328484Z DEBUG rpc::types: data len: 262144, expected len: 262144
2024-06-03T14:02:31.331884Z DEBUG chunk_pool::mem_pool::chunk_pool_inner: Begin to write segment, root=0x50a3…fc86, segment_size=262144, segment_index=2
2024-06-03T14:02:31.335622Z DEBUG chunk_pool::mem_pool::chunk_write_control: Succeeded to write segment, root=0x50a3…fc86, seg_index=2, total_writings=0
2024-06-03T14:02:31.335662Z DEBUG rpc::zgs::r#impl: putSegment root=0x50a3…fc86 index=3
2024-06-03T14:02:31.335828Z DEBUG rpc::types: data len: 262144, expected len: 262144
2024-06-03T14:02:31.344837Z DEBUG chunk_pool::mem_pool::chunk_pool_inner: Begin to write segment, root=0x50a3…fc86, segment_size=262144, segment_index=3
2024-06-03T14:02:31.349595Z DEBUG chunk_pool::mem_pool::chunk_write_control: Succeeded to write segment, root=0x50a3…fc86, seg_index=3, total_writings=0
2024-06-03T14:02:31.349640Z DEBUG chunk_pool::mem_pool::chunk_pool_inner: Queue to finalize transaction for file 0x50a3…fc86
2024-06-03T14:02:31.349671Z DEBUG jsonrpsee_http_server::server: [service_fn] sending back: "{\"jsonrpc\":\"2.0\",\"result\":null,\"id\":71}"
2024-06-03T14:02:31.350732Z DEBUG chunk_pool::handler: Received task to finalize transaction id=FileID { root: 0x50a34963d1108469d60febff9c6a016d7587c65bc03d8eff40d692c8aca7fc86, tx_id: TxID { seq: 24927, hash: 0xaf79a296bf9d9bfd9d053d33890a3ba82c2d012f16d53ed86b6479283222ba9e } }
2024-06-03T14:02:31.351383Z DEBUG storage::log_store::log_manager: finalize_tx_with_hash: tx=Transaction { stream_ids: [], data: [], data_merkle_root: 0x50a34963d1108469d60febff9c6a016d7587c65bc03d8eff40d692c8aca7fc86, merkle_nodes: [(13, 0x50a34963d1108469d60febff9c6a016d7587c65bc03d8eff40d692c8aca7fc86)], start_entry_index: 150384640, size: 1048576, seq: 24927 }
2024-06-03T14:02:31.351426Z DEBUG storage::log_store::log_manager: segments_for_proof: 4, last_segment_size_for_proof: 1024
2024-06-03T14:02:31.351429Z DEBUG storage::log_store::log_manager: segments_for_file: 4, last_segment_size_for_file: 1024
2024-06-03T14:02:31.351436Z DEBUG storage::log_store::log_manager: Padding size: 0
2024-06-03T14:02:31.353849Z DEBUG chunk_pool::handler: Transaction finalized id=FileID { root: 0x50a34963d1108469d60febff9c6a016d7587c65bc03d8eff40d692c8aca7fc86, tx_id: TxID { seq: 24927, hash: 0xaf79a296bf9d9bfd9d053d33890a3ba82c2d012f16d53ed86b6479283222ba9e } }
2024-06-03T14:02:31.353950Z DEBUG router::service: Received new message msg=AnnounceLocalFile { tx_id: TxID { seq: 24927, hash: 0xaf79a296bf9d9bfd9d053d33890a3ba82c2d012f16d53ed86b6479283222ba9e } }
2024-06-03T14:02:31.354558Z DEBUG router::service: Received new message msg=Publish { messages: [AnnounceFile(SignedMessage { inner: AnnounceFile { tx_id: TxID { seq: 24927, hash: 0xaf79a296bf9d9bfd9d053d33890a3ba82c2d012f16d53ed86b6479283222ba9e }, peer_id: WrappedPeerId(PeerId("16Uiu2HAmDRe6P2e56QBrmQwaBCXcYdeHAk85ngaYRqBFzQGevEQY")), at: WrappedMultiaddr("/ip4/172.19.0.1/tcp/1234"), timestamp: 1717423351 }, signature: [48, 68, 2, 32, 22, 175, 255, 30, 158, 71, 232, 159, 123, 38, 65, 155, 164, 166, 61, 78, 9, 45, 21, 75, 22, 136, 3, 72, 3, 21, 222, 68, 5, 136, 41, 146, 2, 32, 23, 39, 35, 95, 243, 146, 248, 148, 108, 1, 7, 49, 222, 225, 199, 163, 116, 152, 55, 196, 106, 98, 53, 66, 30, 27, 145, 232, 105, 3, 13, 120], resend_timestamp: 1717423351 })] }
2024-06-03T14:02:31.354679Z DEBUG router::service: Sending pubsub messages count=1 topics=[AnnounceFile]
2024-06-03T14:02:31.355096Z  WARN network::behaviour: Could not publish message error=InsufficientPeers

i just found some errors in my public node logs. the errors seem strange since I freshly deployed the server for the public node, and no other chain has been installed on this server before. besides that, the sync is still working normally, but the errors seem like a bug. my preliminary analysis is that this error might be caused by many people using the RPC from my public node.

public rpc: http://lightnode-0g.grandvalleys.com:26657
public json-rpc: http://lightnode-0g.grandvalleys.com:8545

7:00AM ERR RPC method eth_gasPrice crashed: failed to parse minimum gas price coin (0.15uinit,0.01uusdc): invalid decimal coin expression: 0.15uinit,0.01uusdc
goroutine 62995293 [running]:
github.com/ethereum/go-ethereum/rpc.(*callback).call.func1()
        github.com/ethereum/[email protected]/rpc/service.go:200 +0x85
panic({0x24351a0?, 0xc09a007890?})
        runtime/panic.go:770 +0x132
github.com/cosmos/cosmos-sdk/server/config.(*Config).GetMinGasPrices(0xc004be30a0?)
        github.com/cosmos/[email protected]/server/config/config.go:269 +0x1a5
github.com/evmos/ethermint/rpc/backend.(*Backend).RPCMinGasPrice(0xc004be8008)
        github.com/evmos/[email protected]/rpc/backend/node_info.go:347 +0x5f
github.com/evmos/ethermint/rpc/backend.(*Backend).GasPrice(0xc004be8008)
        github.com/evmos/[email protected]/rpc/backend/call_tx.go:411 +0x85
github.com/evmos/ethermint/rpc/namespaces/ethereum/eth.(*PublicAPI).GasPrice(0xc004b2ccf0)
        github.com/evmos/[email protected]/rpc/namespaces/ethereum/eth/api.go:317 +0x4d
reflect.Value.call({0xc003fab4f0?, 0xc003857b28?, 0x30?}, {0x2828a4e, 0x4}, {0xc00c6027e0, 0x1, 0xc00c6027e0?})
        reflect/value.go:596 +0xce5
reflect.Value.Call({0xc003fab4f0?, 0xc003857b28?, 0x52336e?}, {0xc00c6027e0?, 0x5519bc?, 0xc036fdd7c0?})
        reflect/value.go:380 +0xb9
github.com/ethereum/go-ethereum/rpc.(*callback).call(0xc0071e11a0, {0x359f8a8, 0xc09cb6f860}, {0xc0447b8444, 0xc}, {0x0, 0x0, 0x4e5a4f?})
        github.com/ethereum/[email protected]/rpc/service.go:206 +0x36d
github.com/ethereum/go-ethereum/rpc.(*handler).runMethod(0x0?, {0x359f8a8?, 0xc09cb6f860?}, 0xc037ffc3f0, 0x0?, {0x0?, 0xc045569f80?, 0x10?})
        github.com/ethereum/[email protected]/rpc/handler.go:389 +0x3c
github.com/ethereum/go-ethereum/rpc.(*handler).handleCall(0xc02f763c20, 0xc00c6026f0, 0xc037ffc3f0)
        github.com/ethereum/[email protected]/rpc/handler.go:337 +0x22f
github.com/ethereum/go-ethereum/rpc.(*handler).handleCallMsg(0xc02f763c20, 0xc00c6026f0, 0xc037ffc3f0)
        github.com/ethereum/[email protected]/rpc/handler.go:298 +0xbd
github.com/ethereum/go-ethereum/rpc.(*handler).handleMsg.func1(0xc00c6026f0)
        github.com/ethereum/[email protected]/rpc/handler.go:139 +0x2f
github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc.func1()
        github.com/ethereum/[email protected]/rpc/handler.go:226 +0xbe
created by github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc in goroutine 62995253
        github.com/ethereum/[email protected]/rpc/handler.go:222 +0x79
 module=geth```

I don't know if this is already possible or not, the docs around log_config are scarce. but the process should be able to directly output logs into STDOUT too. this way log management is way easier with Docker or systemd

Version: v0.3.1-testnet

1、Problem

because of add peer to the local routing table failed, it will throw error "ENR has no UDP socket to connect to" in storage node discovery service.

2024-06-05T10:47:27.342571Z DEBUG network::service: Loaded network key from disk.
2024-06-05T10:47:27.347435Z DEBUG network::discovery::enr: ENR loaded from disk file="network/enr.dat"
2024-06-05T10:47:27.350018Z INFO network::nat: UPnP Attempting to initialise routes
2024-06-05T10:47:27.350025Z INFO network::service: Libp2p Starting peer_id=16Uiu2HAmV1thBhRkR4UxeTsZvAf8TPYjaAKYat52uhEXj8sPHRUD bandwidth_config=3-Average
2024-06-05T10:47:27.350046Z DEBUG network::service: Attempting to open listening ports address=0.0.0.0 tcp_port=1234 udp_port=1234
2024-06-05T10:47:27.370063Z INFO network::discovery: ENR Initialised enr=enr:-Hy4QHLFbV8sgroNoC-98OKLalUx77VkpSqEPi3zWEz5ifAyammhtQ72BXkjNBWol1nqDJBeRBQLrfVtr1BwHmS9S9ABgmlkgnY0iXNlY3AyNTZrMaED8xXKnGEjsaedRV7svbQn_bCXih-xmdtFpcnqsxVz4WqDdGNwggTS seq=1 id=0x1c57..c52d ip=None udp=None tcp=Some(1234) udp4_socket=None
2024-06-05T10:47:27.375388Z DEBUG network::discovery: Discovery service started
2024-06-05T10:47:27.375396Z INFO network::discovery: Contacting Multiaddr boot-nodes for their ENR
2024-06-05T10:47:27.375451Z DEBUG discv5::handler: Handler Starting
2024-06-05T10:47:27.375456Z DEBUG discv5::socket::recv: Recv handler starting
2024-06-05T10:47:27.375473Z DEBUG discv5::socket::send: Send handler starting
2024-06-05T10:47:27.375498Z INFO discv5::service: Discv5 Service started
2024-06-05T10:47:27.377093Z DEBUG discv5::service: Sending RPC Request: id: 541cedb93aed7b3d: FINDNODE Request: distance: [0] to node: Node: 0xc909..1134, addr: 54.219.26.22:1234
2024-06-05T10:47:27.377117Z DEBUG discv5::service: Sending RPC Request: id: d93684dc33f63c9f: FINDNODE Request: distance: [0] to node: Node: 0x3ec9..8916, addr: 52.52.127.117:1234
2024-06-05T10:47:27.461357Z DEBUG discv5::service: Received RPC response: NODES: total: 1, Nodes: [ENR: NodeId: 0xc909..1134, IpV4 Socket: None IpV6 Socket: None] to request: FINDNODE Request: distance: [0] from: Node: 0xc909..1134, addr: 54.219.26.22:1234
2024-06-05T10:47:27.461455Z DEBUG network::discovery: Adding node to routing table node_id=0xc909..1134 peer_id=16Uiu2HAmTVDGNhkHD98zDnJxQWu3i1FL1aFYeh9wiQTNu4pDCgps ip=None udp=None tcp=Some(1234)
2024-06-05T10:47:27.461479Z WARN discv5::discv5: ENR attempted to be added without a UDP socket has been ignored
2024-06-05T10:47:27.461485Z ERROR network::discovery: Could not add peer to the local routing table addr=/ip4/54.219.26.22/udp/1234/p2p/16Uiu2HAmTVDGNhkHD98zDnJxQWu3i1FL1aFYeh9wiQTNu4pDCgps error=ENR has no UDP socket to connect to
2024-06-05T10:47:27.461617Z DEBUG discv5::service: Received RPC response: NODES: total: 1, Nodes: [ENR: NodeId: 0x3ec9..8916, IpV4 Socket: None IpV6 Socket: None] to request: FINDNODE Request: distance: [0] from: Node: 0x3ec9..8916, addr: 52.52.127.117:1234
2024-06-05T10:47:27.461682Z DEBUG network::discovery: Adding node to routing table node_id=0x3ec9..8916 peer_id=16Uiu2HAkzRjxK2gorngB1Xq84qDrT4hSVznYDHj6BkbaE4SGx9oS ip=None udp=None tcp=Some(1234)
2024-06-05T10:47:27.461697Z WARN discv5::discv5: ENR attempted to be added without a UDP socket has been ignored
2024-06-05T10:47:27.461700Z ERROR network::discovery: Could not add peer to the local routing table addr=/ip4/52.52.127.117/udp/1234/p2p/16Uiu2HAkzRjxK2gorngB1Xq84qDrT4hSVznYDHj6BkbaE4SGx9oS error=ENR has no UDP socket to connect to

2、Config info

• modified: blockchain_rpc_endpoint value with local validator rpc endpoint
• others with default value

3、Enviroment

• 0.2.0 branch

1. Problem
   v0.2.0 binary version is old one.

Command

zgs_node --version

Result

zgs_node 0.1.0

it should be 0.2.0

2. Environment
   OS: Ubuntu22.04LTS
   cargo 1.78.0 (54d8815d0 2024-03-26)
   rustc 1.78.0 (9b00956e5 2024-04-29)
   rustup 1.27.1 (54dd3d00f 2024-04-24)

i use my own validator node rpc for the storage node. my validator node rpc is synced well. but my storage node can't sync. here's the logs:

2024-05-30T12:30:20.605336Z INFO zgs_node::client::environment: Internal shutdown received reason="log sync failure"
2024-05-30T12:30:20.605356Z INFO zgs_node: Shutting down... reason=Failure("log sync failure")
2024-05-30T12:30:39.469803Z INFO zgs_node: Starting services...
2024-05-30T12:30:39.519665Z DEBUG storage::log_store::log_manager: LogManager::new() with chunk_list_len=0 start_tx_seq=None last_chunk=0
2024-05-30T12:30:39.520453Z DEBUG network::service: Loaded network key from disk.
2024-05-30T12:30:39.521387Z DEBUG network::discovery::enr: ENR loaded from disk file="network/enr.dat"
2024-05-30T12:30:39.521745Z INFO network::service: Libp2p Starting peer_id=16Uiu2HAm9i66Bwa2rNpHBzVAzLi248gRQQqbfXG1KAiCeHkkEJGB bandwidth_config=3-Average
2024-05-30T12:30:39.521778Z DEBUG network::service: Attempting to open listening ports address=0.0.0.0 tcp_port=1234 udp_port=1234
2024-05-30T12:30:39.521966Z INFO network::nat: UPnP Attempting to initialise routes
2024-05-30T12:30:39.522568Z INFO network::discovery: ENR Initialised enr=enr:-Hy4QBIoXiK2jg5cplb6ykQ7uKwbAb35BJoXkBzFjT9XYS0sSQUPuPzRLtP8ecVOSYYCWdJL3aQasUxkqd9kf9u3XskBgmlkgnY0iXNlY3AyNTZrMaEC1Dgr5q7Ptu2lhCAkr2gQRiqqeIZjdNudydjE9DNkm-yDdGNwggTS seq=1 id=0xa8af..5bd2 ip=None udp=None tcp=Some(1234) udp4_socket=None
2024-05-30T12:30:39.523679Z DEBUG network::discovery: Discovery service started
2024-05-30T12:30:39.523694Z INFO network::discovery: Contacting Multiaddr boot-nodes for their ENR
2024-05-30T12:30:39.523826Z INFO discv5::service: Discv5 Service started
2024-05-30T12:30:39.523893Z DEBUG discv5::service: Sending RPC Request: id: 6e492ab7c0a72d47: FINDNODE Request: distance: [0] to node: Node: 0xc909..1134, addr: 54.219.26.22:1234
2024-05-30T12:30:39.523906Z DEBUG discv5::service: Sending RPC Request: id: 592c653507584bcd: FINDNODE Request: distance: [0] to node: Node: 0x3ec9..8916, addr: 52.52.127.117:1234
2024-05-30T12:30:39.523956Z DEBUG discv5::handler: Handler Starting
2024-05-30T12:30:39.523987Z DEBUG discv5::socket::send: Send handler starting
2024-05-30T12:30:39.524018Z DEBUG discv5::socket::recv: Recv handler starting
2024-05-30T12:30:39.883850Z DEBUG discv5::service: Received RPC response: NODES: total: 1, Nodes: [ENR: NodeId: 0xc909..1134, IpV4 Socket: None IpV6 Socket: None] to request: FINDNODE Request: distance: [0] from: Node: 0xc909..1134, addr: 54.219.26.22:1234
2024-05-30T12:30:39.883988Z DEBUG network::discovery: Adding node to routing table node_id=0xc909..1134 peer_id=16Uiu2HAmTVDGNhkHD98zDnJxQWu3i1FL1aFYeh9wiQTNu4pDCgps ip=None udp=None tcp=Some(1234)
2024-05-30T12:30:39.884005Z WARN discv5::discv5: ENR attempted to be added without a UDP socket has been ignored
2024-05-30T12:30:39.884009Z ERROR network::discovery: Could not add peer to the local routing table addr=/ip4/54.219.26.22/udp/1234/p2p/16Uiu2HAmTVDGNhkHD98zDnJxQWu3i1FL1aFYeh9wiQTNu4pDCgps error=ENR has no UDP socket to connect to
2024-05-30T12:30:39.900506Z DEBUG discv5::service: Received RPC response: NODES: total: 1, Nodes: [ENR: NodeId: 0x3ec9..8916, IpV4 Socket: None IpV6 Socket: None] to request: FINDNODE Request: distance: [0] from: Node: 0x3ec9..8916, addr: 52.52.127.117:1234
2024-05-30T12:30:39.901136Z DEBUG network::discovery: Adding node to routing table node_id=0x3ec9..8916 peer_id=16Uiu2HAkzRjxK2gorngB1Xq84qDrT4hSVznYDHj6BkbaE4SGx9oS ip=None udp=None tcp=Some(1234)
2024-05-30T12:30:39.901170Z WARN discv5::discv5: ENR attempted to be added without a UDP socket has been ignored
2024-05-30T12:30:39.901181Z ERROR network::discovery: Could not add peer to the local routing table addr=/ip4/52.52.127.117/udp/1234/p2p/16Uiu2HAkzRjxK2gorngB1Xq84qDrT4hSVznYDHj6BkbaE4SGx9oS error=ENR has no UDP socket to connect to
2024-05-30T12:30:39.901193Z DEBUG network::discovery: Starting a peer discovery request target_peers=16
2024-05-30T12:30:39.901537Z INFO network::service: Listening established address=/ip4/0.0.0.0/tcp/1234/p2p/16Uiu2HAm9i66Bwa2rNpHBzVAzLi248gRQQqbfXG1KAiCeHkkEJGB
2024-05-30T12:30:39.901573Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/find_file/ssz_snappy
2024-05-30T12:30:39.901578Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/find_chunks/ssz_snappy
2024-05-30T12:30:39.901580Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/announce_file/ssz_snappy
2024-05-30T12:30:39.901584Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/announce_chunks/ssz_snappy
2024-05-30T12:30:39.901586Z INFO network::service: Subscribed to topics topics=[FindFile, FindChunks, AnnounceFile, AnnounceChunks]
2024-05-30T12:30:39.901712Z DEBUG sync::service: Starting sync service
2024-05-30T12:30:40.743002Z WARN log_entry_sync::sync_manager: unable to get finalized block: Err(JsonRpcClientError(JsonRpcError(JsonRpcError { code: -32601, message: "Method not found", data: None })))
2024-05-30T12:30:40.884943Z DEBUG discv5::service: RPC Request timed out. id: 79e8a4df90a30d6b
2024-05-30T12:30:40.902271Z DEBUG discv5::service: RPC Request timed out. id: 4ae6ab6e4372a3bc
2024-05-30T12:30:41.068135Z ERROR zgs_node: Failed to start zgs node reason=Unable to get chain_id: JsonRpcClientError(JsonRpcError(JsonRpcError { code: -32601, message: "Method not found", data: None }))
2024-05-30T12:30:41.068220Z INFO discv5::service: Discv5 Service shutdown
2024-05-30T12:30:41.068423Z INFO zgs_node::client::environment: Internal shutdown received reason="Failed to start zgs node"
2024-05-30T12:30:41.068453Z INFO zgs_node: Shutting down... reason=Failure("Failed to start zgs node")
2024-05-30T12:30:41.068478Z DEBUG discv5::socket::recv: Recv handler shutdown
2024-05-30T12:30:41.068488Z DEBUG discv5::socket::send: Send handler shutdown
2024-05-30T12:30:41.068514Z DEBUG task_executor: Async task shutdown, exit received task="log_reload"
2024-05-30T12:30:59.974765Z INFO zgs_node: Starting services...
2024-05-30T12:31:00.025068Z DEBUG storage::log_store::log_manager: LogManager::new() with chunk_list_len=0 start_tx_seq=None last_chunk=0
2024-05-30T12:31:00.025603Z DEBUG network::service: Loaded network key from disk.
2024-05-30T12:31:00.026704Z DEBUG network::discovery::enr: ENR loaded from disk file="network/enr.dat"
2024-05-30T12:31:00.027085Z INFO network::service: Libp2p Starting peer_id=16Uiu2HAm9i66Bwa2rNpHBzVAzLi248gRQQqbfXG1KAiCeHkkEJGB bandwidth_config=3-Average
2024-05-30T12:31:00.027121Z DEBUG network::service: Attempting to open listening ports address=0.0.0.0 tcp_port=1234 udp_port=1234
2024-05-30T12:31:00.027190Z INFO network::nat: UPnP Attempting to initialise routes
2024-05-30T12:31:00.027789Z INFO network::discovery: ENR Initialised enr=enr:-Hy4QBIoXiK2jg5cplb6ykQ7uKwbAb35BJoXkBzFjT9XYS0sSQUPuPzRLtP8ecVOSYYCWdJL3aQasUxkqd9kf9u3XskBgmlkgnY0iXNlY3AyNTZrMaEC1Dgr5q7Ptu2lhCAkr2gQRiqqeIZjdNudydjE9DNkm-yDdGNwggTS seq=1 id=0xa8af..5bd2 ip=None udp=None tcp=Some(1234) udp4_socket=None
2024-05-30T12:31:00.028959Z DEBUG network::discovery: Discovery service started
2024-05-30T12:31:00.028969Z INFO network::discovery: Contacting Multiaddr boot-nodes for their ENR
2024-05-30T12:31:00.029129Z INFO discv5::service: Discv5 Service started
2024-05-30T12:31:00.029150Z DEBUG discv5::service: Sending RPC Request: id: 4d44aa7a4e1c05cc: FINDNODE Request: distance: [0] to node: Node: 0xc909..1134, addr: 54.219.26.22:1234
2024-05-30T12:31:00.029163Z DEBUG discv5::service: Sending RPC Request: id: c1c982c3e6f18684: FINDNODE Request: distance: [0] to node: Node: 0x3ec9..8916, addr: 52.52.127.117:1234
2024-05-30T12:31:00.029191Z DEBUG discv5::socket::recv: Recv handler starting
2024-05-30T12:31:00.029199Z DEBUG discv5::handler: Handler Starting
2024-05-30T12:31:00.029264Z DEBUG discv5::socket::send: Send handler starting
2024-05-30T12:31:00.389140Z DEBUG discv5::service: Received RPC response: NODES: total: 1, Nodes: [ENR: NodeId: 0xc909..1134, IpV4 Socket: None IpV6 Socket: None] to request: FINDNODE Request: distance: [0] from: Node: 0xc909..1134, addr: 54.219.26.22:1234
2024-05-30T12:31:00.389826Z DEBUG network::discovery: Adding node to routing table node_id=0xc909..1134 peer_id=16Uiu2HAmTVDGNhkHD98zDnJxQWu3i1FL1aFYeh9wiQTNu4pDCgps ip=None udp=None tcp=Some(1234)
2024-05-30T12:31:00.389853Z WARN discv5::discv5: ENR attempted to be added without a UDP socket has been ignored
2024-05-30T12:31:00.389915Z ERROR network::discovery: Could not add peer to the local routing table addr=/ip4/54.219.26.22/udp/1234/p2p/16Uiu2HAmTVDGNhkHD98zDnJxQWu3i1FL1aFYeh9wiQTNu4pDCgps error=ENR has no UDP socket to connect to
2024-05-30T12:31:00.406628Z DEBUG discv5::service: Received RPC response: NODES: total: 1, Nodes: [ENR: NodeId: 0x3ec9..8916, IpV4 Socket: None IpV6 Socket: None] to request: FINDNODE Request: distance: [0] from: Node: 0x3ec9..8916, addr: 52.52.127.117:1234
2024-05-30T12:31:00.407218Z DEBUG network::discovery: Adding node to routing table node_id=0x3ec9..8916 peer_id=16Uiu2HAkzRjxK2gorngB1Xq84qDrT4hSVznYDHj6BkbaE4SGx9oS ip=None udp=None tcp=Some(1234)
2024-05-30T12:31:00.407236Z WARN discv5::discv5: ENR attempted to be added without a UDP socket has been ignored
2024-05-30T12:31:00.407245Z ERROR network::discovery: Could not add peer to the local routing table addr=/ip4/52.52.127.117/udp/1234/p2p/16Uiu2HAkzRjxK2gorngB1Xq84qDrT4hSVznYDHj6BkbaE4SGx9oS error=ENR has no UDP socket to connect to
2024-05-30T12:31:00.407260Z DEBUG network::discovery: Starting a peer discovery request target_peers=16
2024-05-30T12:31:00.407617Z INFO network::service: Listening established address=/ip4/0.0.0.0/tcp/1234/p2p/16Uiu2HAm9i66Bwa2rNpHBzVAzLi248gRQQqbfXG1KAiCeHkkEJGB
2024-05-30T12:31:00.407657Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/find_file/ssz_snappy
2024-05-30T12:31:00.407662Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/find_chunks/ssz_snappy
2024-05-30T12:31:00.407665Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/announce_file/ssz_snappy
2024-05-30T12:31:00.407668Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/announce_chunks/ssz_snappy
2024-05-30T12:31:00.407670Z INFO network::service: Subscribed to topics topics=[FindFile, FindChunks, AnnounceFile, AnnounceChunks]
2024-05-30T12:31:00.407781Z DEBUG sync::service: Starting sync service
2024-05-30T12:31:01.390090Z DEBUG discv5::service: RPC Request timed out. id: a3fbb53a306e072a
2024-05-30T12:31:01.408304Z DEBUG discv5::service: RPC Request timed out. id: 9f1bdf9bb6309ef4
2024-05-30T12:31:01.640498Z ERROR zgs_node: Failed to start zgs node reason=Unable to get chain_id: JsonRpcClientError(JsonRpcError(JsonRpcError { code: -32601, message: "Method not found", data: None }))
2024-05-30T12:31:01.640556Z INFO discv5::service: Discv5 Service shutdown
2024-05-30T12:31:01.640842Z DEBUG discv5::socket::recv: Recv handler shutdown
2024-05-30T12:31:01.640821Z INFO zgs_node::client::environment: Internal shutdown received reason="Failed to start zgs node"
2024-05-30T12:31:01.640858Z INFO zgs_node: Shutting down... reason=Failure("Failed to start zgs node")
2024-05-30T12:31:20.476222Z INFO zgs_node: Starting services...
2024-05-30T12:31:20.515538Z DEBUG storage::log_store::log_manager: LogManager::new() with chunk_list_len=0 start_tx_seq=None last_chunk=0
2024-05-30T12:31:20.516087Z DEBUG network::service: Loaded network key from disk.
2024-05-30T12:31:20.517042Z DEBUG network::discovery::enr: ENR loaded from disk file="network/enr.dat"
2024-05-30T12:31:20.517320Z INFO network::service: Libp2p Starting peer_id=16Uiu2HAm9i66Bwa2rNpHBzVAzLi248gRQQqbfXG1KAiCeHkkEJGB bandwidth_config=3-Average
2024-05-30T12:31:20.517343Z DEBUG network::service: Attempting to open listening ports address=0.0.0.0 tcp_port=1234 udp_port=1234
2024-05-30T12:31:20.517380Z INFO network::nat: UPnP Attempting to initialise routes
2024-05-30T12:31:20.517909Z INFO network::discovery: ENR Initialised enr=enr:-Hy4QBIoXiK2jg5cplb6ykQ7uKwbAb35BJoXkBzFjT9XYS0sSQUPuPzRLtP8ecVOSYYCWdJL3aQasUxkqd9kf9u3XskBgmlkgnY0iXNlY3AyNTZrMaEC1Dgr5q7Ptu2lhCAkr2gQRiqqeIZjdNudydjE9DNkm-yDdGNwggTS seq=1 id=0xa8af..5bd2 ip=None udp=None tcp=Some(1234) udp4_socket=None
2024-05-30T12:31:20.518751Z DEBUG network::discovery: Discovery service started
2024-05-30T12:31:20.518763Z INFO network::discovery: Contacting Multiaddr boot-nodes for their ENR
2024-05-30T12:31:20.518897Z INFO discv5::service: Discv5 Service started
2024-05-30T12:31:20.518897Z DEBUG discv5::handler: Handler Starting
2024-05-30T12:31:20.518915Z DEBUG discv5::service: Sending RPC Request: id: 94a73e18bb0199da: FINDNODE Request: distance: [0] to node: Node: 0xc909..1134, addr: 54.219.26.22:1234
2024-05-30T12:31:20.518923Z DEBUG discv5::service: Sending RPC Request: id: 77a12d98f549a1a5: FINDNODE Request: distance: [0] to node: Node: 0x3ec9..8916, addr: 52.52.127.117:1234
2024-05-30T12:31:20.518935Z DEBUG discv5::socket::recv: Recv handler starting
2024-05-30T12:31:20.518967Z DEBUG discv5::socket::send: Send handler starting
2024-05-30T12:31:20.878559Z DEBUG discv5::service: Received RPC response: NODES: total: 1, Nodes: [ENR: NodeId: 0xc909..1134, IpV4 Socket: None IpV6 Socket: None] to request: FINDNODE Request: distance: [0] from: Node: 0xc909..1134, addr: 54.219.26.22:1234
2024-05-30T12:31:20.878661Z DEBUG network::discovery: Adding node to routing table node_id=0xc909..1134 peer_id=16Uiu2HAmTVDGNhkHD98zDnJxQWu3i1FL1aFYeh9wiQTNu4pDCgps ip=None udp=None tcp=Some(1234)
2024-05-30T12:31:20.878677Z WARN discv5::discv5: ENR attempted to be added without a UDP socket has been ignored
2024-05-30T12:31:20.878681Z ERROR network::discovery: Could not add peer to the local routing table addr=/ip4/54.219.26.22/udp/1234/p2p/16Uiu2HAmTVDGNhkHD98zDnJxQWu3i1FL1aFYeh9wiQTNu4pDCgps error=ENR has no UDP socket to connect to
2024-05-30T12:31:20.896223Z DEBUG discv5::service: Received RPC response: NODES: total: 1, Nodes: [ENR: NodeId: 0x3ec9..8916, IpV4 Socket: None IpV6 Socket: None] to request: FINDNODE Request: distance: [0] from: Node: 0x3ec9..8916, addr: 52.52.127.117:1234
2024-05-30T12:31:20.896802Z DEBUG network::discovery: Adding node to routing table node_id=0x3ec9..8916 peer_id=16Uiu2HAkzRjxK2gorngB1Xq84qDrT4hSVznYDHj6BkbaE4SGx9oS ip=None udp=None tcp=Some(1234)
2024-05-30T12:31:20.896837Z WARN discv5::discv5: ENR attempted to be added without a UDP socket has been ignored
2024-05-30T12:31:20.896853Z ERROR network::discovery: Could not add peer to the local routing table addr=/ip4/52.52.127.117/udp/1234/p2p/16Uiu2HAkzRjxK2gorngB1Xq84qDrT4hSVznYDHj6BkbaE4SGx9oS error=ENR has no UDP socket to connect to
2024-05-30T12:31:20.896881Z DEBUG network::discovery: Starting a peer discovery request target_peers=16
2024-05-30T12:31:20.897260Z INFO network::service: Listening established address=/ip4/0.0.0.0/tcp/1234/p2p/16Uiu2HAm9i66Bwa2rNpHBzVAzLi248gRQQqbfXG1KAiCeHkkEJGB
2024-05-30T12:31:20.897294Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/find_file/ssz_snappy
2024-05-30T12:31:20.897299Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/find_chunks/ssz_snappy
2024-05-30T12:31:20.897301Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/announce_file/ssz_snappy
2024-05-30T12:31:20.897305Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/announce_chunks/ssz_snappy
2024-05-30T12:31:20.897307Z INFO network::service: Subscribed to topics topics=[FindFile, FindChunks, AnnounceFile, AnnounceChunks]
2024-05-30T12:31:20.897407Z DEBUG sync::service: Starting sync service
2024-05-30T12:31:21.753827Z WARN log_entry_sync::sync_manager: unable to get finalized block: Err(JsonRpcClientError(JsonRpcError(JsonRpcError { code: -32601, message: "Method not found", data: None })))
2024-05-30T12:31:21.880854Z DEBUG discv5::service: RPC Request timed out. id: 003ce4b893ae5033
2024-05-30T12:31:21.898164Z DEBUG discv5::service: RPC Request timed out. id: acfc4418baf750c1
2024-05-30T12:31:22.163848Z ERROR log_entry_sync::sync_manager: log sync failure: e=(code: -32601, message: Method not found, data: None)

Stack backtrace:
0: anyhow::error::<impl core::convert::From for anyhow::Error>::from
1: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
2: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
3: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
4: <futures_util::future::future::flatten::Flatten<Fut,::Output> as core::future::future::Future>::poll
5: tokio::runtime::task::core::Core<T,S>::poll
6: tokio::runtime::task::harness::Harness<T,S>::poll
7: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
8: tokio::runtime::scheduler::multi_thread::worker::Context::run
9: tokio::runtime::context::scoped::Scoped::set
10: tokio::runtime::context::runtime::enter_runtime
11: tokio::runtime::scheduler::multi_thread::worker::run
12: <tokio::runtime::blocking::task::BlockingTask as core::future::future::Future>::poll
13: tokio::runtime::task::core::Core<T,S>::poll
14: tokio::runtime::task::harness::Harness<T,S>::poll
15: tokio::runtime::blocking::pool::Inner::run
16: std::sys_common::backtrace::__rust_begin_short_backtrace
17: core::ops::function::FnOnce::call_once{{vtable.shim}}
18: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/alloc/src/boxed.rs:2007:9
19: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/alloc/src/boxed.rs:2007:9
20: std::sys::unix::thread::Thread::new::thread_start
at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/std/src/sys/unix/thread.rs:108:17
21:
22:

A error occured when i start storage node.

Config config.toml

# RPC endpoint to sync event logs on EVM compatible blockchain.
#blockchain_rpc_endpoint = "https://rpc-testnet.0g.ai"
blockchain_rpc_endpoint="http://localhost:26657"

Error when startup:

2024-05-18T10:01:49.574455Z  INFO zgs_node: Starting services...
2024-05-18T10:01:50.066414Z DEBUG storage::log_store::log_manager: LogManager::new() with chunk_list_len=0 start_tx_seq=Some(0) last_chunk=22
2024-05-18T10:01:50.168410Z DEBUG network::service: Loaded network key from disk.
2024-05-18T10:01:50.193779Z  WARN log_entry_sync::sync_manager: unable to get finalized block: Err(JsonRpcClientError(JsonRpcError(JsonRpcError { code: -32601, message: "Method not found", data: None })))
2024-05-18T10:01:50.198436Z DEBUG network::discovery::enr: ENR loaded from disk file="network/enr.dat"
2024-05-18T10:01:50.198684Z  INFO network::nat: UPnP Attempting to initialise routes
2024-05-18T10:01:50.198722Z  INFO network::service: Libp2p Starting peer_id=16Uiu2HAmF52b7fjW6W2E651Cqr2pcBCxecCwQi4WQ1wT5vD4dNCX bandwidth_config=3-Average
2024-05-18T10:01:50.219336Z DEBUG network::service: Attempting to open listening ports address=0.0.0.0 tcp_port=1234 udp_port=1234
2024-05-18T10:01:50.224224Z  INFO network::discovery: ENR Initialised enr=enr:-Hy4QIe3R67ny0z_jpySrA2HlPW4l9d6uF6Hm2-CCGTZtFe4UVq7dVS5IJG9C-qFa1YdV8NOdxvB0DVQ6yn5jbe33rkBgmlkgnY0iXNlY3AyNTZrMaEDI9_YHknFhCr1msvR-3U2Hd75VGCF-swAU1jRVWpngiCDdGNwggTS seq=1 id=0x5675..9878 ip=None udp=None tcp=Some(1234) udp4_socket=None
2024-05-18T10:01:50.245293Z DEBUG discv5::handler: Handler Starting
2024-05-18T10:01:50.245313Z DEBUG discv5::socket::send: Send handler starting
2024-05-18T10:01:50.245312Z DEBUG discv5::socket::recv: Recv handler starting
2024-05-18T10:01:50.269505Z DEBUG network::discovery: Discovery service started
2024-05-18T10:01:50.269518Z  INFO network::discovery: Contacting Multiaddr boot-nodes for their ENR
2024-05-18T10:01:50.269604Z  INFO discv5::service: Discv5 Service started
2024-05-18T10:01:50.294429Z DEBUG discv5::service: Sending RPC Request: id: 41a42b23207dc9ce: FINDNODE Request: distance: [0] to node: Node: 0xc909..1134, addr: 54.219.26.22:1234
2024-05-18T10:01:50.294448Z DEBUG discv5::service: Sending RPC Request: id: eadc02f03c26edc0: FINDNODE Request: distance: [0] to node: Node: 0x3ec9..8916, addr: 52.52.127.117:1234
2024-05-18T10:01:50.373567Z DEBUG discv5::service: Received RPC response: NODES: total: 1, Nodes: [ENR: NodeId: 0xc909..1134, IpV4 Socket: None IpV6 Socket: None] to request: FINDNODE Request: distance: [0] from: Node: 0xc909..1134, addr: 54.219.26.22:1234
2024-05-18T10:01:50.373631Z DEBUG network::discovery: Adding node to routing table node_id=0xc909..1134 peer_id=16Uiu2HAmTVDGNhkHD98zDnJxQWu3i1FL1aFYeh9wiQTNu4pDCgps ip=None udp=None tcp=Some(1234)
2024-05-18T10:01:50.373641Z  WARN discv5::discv5: ENR attempted to be added without a UDP socket has been ignored
2024-05-18T10:01:50.373646Z ERROR network::discovery: Could not add peer to the local routing table addr=/ip4/54.219.26.22/udp/1234/p2p/16Uiu2HAmTVDGNhkHD98zDnJxQWu3i1FL1aFYeh9wiQTNu4pDCgps error=ENR has no UDP socket to connect to
2024-05-18T10:01:50.211120Z ERROR log_entry_sync::sync_manager: log sync failure: e=(code: -32601, message: Method not found, data: None)

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
   1: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
   2: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
   3: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
   4: <futures_util::future::future::flatten::Flatten<Fut,<Fut as core::future::future::Future>::Output> as core::future::future::Future>::poll
   5: tokio::runtime::task::core::Core<T,S>::poll
   6: tokio::runtime::task::harness::Harness<T,S>::poll
   7: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
   8: tokio::runtime::scheduler::multi_thread::worker::Context::run
   9: tokio::runtime::context::scoped::Scoped<T>::set
  10: tokio::runtime::context::runtime::enter_runtime
  11: tokio::runtime::scheduler::multi_thread::worker::run
  12: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  13: tokio::runtime::task::core::Core<T,S>::poll
  14: tokio::runtime::task::harness::Harness<T,S>::poll
  15: tokio::runtime::blocking::pool::Inner::run
  16: std::sys_common::backtrace::__rust_begin_short_backtrace
  17: core::ops::function::FnOnce::call_once{{vtable.shim}}
  18: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2007:9
  19: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2007:9
  20: std::sys::unix::thread::Thread::new::thread_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys/unix/thread.rs:108:17
  21: <unknown>
  22: <unknown>
2024-05-18T10:01:50.406093Z  INFO zgs_node::client::environment: Internal shutdown received reason="log sync failure"
2024-05-18T10:01:50.406111Z  INFO zgs_node: Shutting down... reason=Failure("log sync failure")
2024-05-18T10:01:50.406135Z DEBUG task_executor: Async task shutdown, exit received task="log_reload"

1. Problem: The discovery query might be incorrect

2024-05-24T10:49:31.088750Z INFO network::peer_manager: Peer statistics connected=0 target=50 outbound=0
2024-05-24T10:49:31.088789Z DEBUG network::peer_manager: Starting a new peer discovery query connected=0 target=50 outbound=0 wanted=16
2024-05-24T10:49:31.088810Z DEBUG network::discovery: Starting a peer discovery request target_peers=16
2024-05-24T10:49:31.786256Z WARN discv5::service: No known_closest_peers found. Return empty result without sending query.
2024-05-24T10:49:31.786331Z DEBUG network::discovery: Discovery query yielded no results.

2. Environment
   OS: Ubuntu 22.04.4 LTS
   cargo: 1.75.0
   rustc 1.75.0 (82e1608df 2023-12-21)

i've used my full-node json-rpc and this happened. also no false configuration

2024-06-25T05:15:12.801243Z  INFO discv5::service: Discv5 Service shutdown
2024-06-25T05:15:12.801260Z  INFO zgs_node::client::environment: Internal shutdown received reason="Failed to start zgs node"
2024-06-25T05:15:12.801273Z  INFO zgs_node: Shutting down... reason=Failure("Failed to start zgs node")
2024-06-25T05:15:12.801302Z DEBUG task_executor: Async task shutdown, exit received task="log_reload"

As the blockchain grows (indefinitely), nodes can manage the amount of storage disk space required by pruning old logs

1. The CI workflow requries sudo when downloading evmosd in each single test, even if the file owners in tests/tmp are not root.
2. After the first failure because of this issue, when I rerun the test script, test_all.py reports

Traceback (most recent call last):
  File "/home/chenxing/vscode-workspace/0g-storage-node/tests/test_all.py", line 8, in <module>
    run_all(
  File "/home/chenxing/vscode-workspace/0g-storage-node/tests/utility/run_all.py", line 64, in run_all
    build_bsc(tmp_dir)
  File "/home/chenxing/vscode-workspace/0g-storage-node/tests/utility/build_binary.py", line 50, in build_bsc
    assert downloaded, "Cannot download download binary from github [%s]" % BSC_BINARY
AssertionError: Cannot download download binary from github [get]

@

12:21PM INF failed attempting to add vote err="expected 761450/1/2, but got 761450/0/2: unexpected step" module=consensus server=node
12:21PM ERR failed to process message err="error adding vote" height=761451 module=consensus msg_type=*consensus.VoteMessage peer=89b355dad04e1532dd71f24fb1a2da49066b223a round=0 server=node
12:21PM INF failed attempting to add vote err="expected 761450/1/2, but got 761450/0/2: unexpected step" module=consensus server=node
12:21PM ERR failed to process message err="error adding vote" height=761451 module=consensus msg_type=*consensus.VoteMessage peer=89b355dad04e1532dd71f24fb1a2da49066b223a round=0 server=node
12:21PM INF failed attempting to add vote err="expected 761450/1/2, but got 761450/0/2: unexpected step" module=consensus server=node
12:21PM ERR failed to process message err="error adding vote" height=761451 module=consensus msg_type=*consensus.VoteMessage peer=89b355dad04e1532dd71f24fb1a2da49066b223a round=0 server=node
12:21PM INF failed attempting to add vote err="expected 761450/1/2, but got 761450/0/2: unexpected step" module=consensus server=node
12:21PM ERR failed to process message err="error adding vote" height=761451 module=consensus msg_type=*consensus.VoteMessage peer=89b355dad04e1532dd71f24fb1a2da49066b223a round=0 server=node
12:21PM INF failed attempting to add vote err="expected 761450/1/2, but got 761450/0/2: unexpected step" module=consensus server=node
12:21PM ERR failed to process message err="error adding vote" height=761451 module=consensus msg_type=*consensus.VoteMessage peer=89b355dad04e1532dd71f24fb1a2da49066b223a round=0 server=node
12:21PM INF failed attempting to add vote err="expected 761450/1/2, but got 761450/0/2: unexpected step" module=consensus server=node
12:21PM ERR failed to process message err="error adding vote" height=761451 module=consensus msg_type=*consensus.VoteMessage peer=89b355dad04e1532dd71f24fb1a2da49066b223a round=0 server=node

I think the problem is with the software, as my miner ID and miner key are in the correct format.
There are also funds to pay the commission at the address associated with the miner key.

Fail to query miner id information: AbiError(DecodingError(InvalidName("please ensure the contract and method you're calling exist! failed to decode empty bytes. if you're using jsonrpc this is likely due to jsonrpc returning 0x in case contract or method don't exist")))

2024-05-08T19:58:28.145772Z  INFO zgs_node: Starting services...
2024-05-08T19:58:28.761285Z DEBUG storage::log_store::log_manager: LogManager::new() with chunk_list_len=0 start_tx_seq=None last_chunk=0
2024-05-08T19:58:28.778283Z DEBUG network::service: Loaded network key from disk.
2024-05-08T19:58:28.778912Z DEBUG network::discovery::enr: ENR loaded from disk file="network/enr.dat"
2024-05-08T19:58:28.779032Z  INFO network::service: Libp2p Starting peer_id=16Uiu2HAmArnuVcV9zYAW3HQ9Ym2srKcte8WWzd6XLSgf4ZpTdHDV bandwidth_config=3-Average
2024-05-08T19:58:28.779048Z DEBUG network::service: Attempting to open listening ports address=0.0.0.0 tcp_port=1234 udp_port=1234
2024-05-08T19:58:28.779087Z  INFO network::nat: UPnP Attempting to initialise routes
2024-05-08T19:58:28.779362Z  INFO network::discovery: ENR Initialised enr=enr:-Hy4QFb5KofEPmZ8J2SwlaCc41_LZznhaqkSLMJeXxE5wgu8fmup4ytSd0pqpNjLKOsCqpU-zY58GRW1lAYRNeSd6yEBgmlkgnY0iXNlY3AyNTZrMaEC5U6nvTtGB_ZNk-NflD3380SyJ2KatZbGEDfqVrPnLjSDdGNwggTS seq=1 id=0xc2d7..95f0 ip=None udp=None tcp=Some(1234) udp4_socket=None
2024-05-08T19:58:28.779721Z DEBUG network::discovery: Discovery service started
2024-05-08T19:58:28.779724Z  INFO network::discovery: Contacting Multiaddr boot-nodes for their ENR
2024-05-08T19:58:28.779791Z  INFO discv5::service: Discv5 Service started
2024-05-08T19:58:28.779789Z DEBUG discv5::handler: Handler Starting
2024-05-08T19:58:28.779790Z DEBUG discv5::socket::recv: Recv handler starting
2024-05-08T19:58:28.779803Z DEBUG discv5::socket::send: Send handler starting
2024-05-08T19:58:28.779804Z DEBUG discv5::service: Sending RPC Request: id: eff0861c79a5b374: FINDNODE Request: distance: [0] to node: Node: 0x0b9e..9bef, addr: 54.219.26.22:1234
2024-05-08T19:58:28.779811Z DEBUG discv5::service: Sending RPC Request: id: dab1d3a1520dd2fa: FINDNODE Request: distance: [0] to node: Node: 0x702b..6b45, addr: 52.52.127.117:1234
2024-05-08T19:58:29.462866Z DEBUG log_entry_sync::sync_manager: current start block number 0, block hash 0x1541…6169, finalized block number 40155
2024-05-08T19:58:29.463183Z DEBUG log_entry_sync::sync_manager::log_entry_fetcher: start_watch starts, start=40156
2024-05-08T19:58:29.463303Z DEBUG log_entry_sync::sync_manager::log_entry_fetcher: start_recover starts, start=0 end=40155
2024-05-08T19:58:29.634212Z DEBUG log_entry_sync::sync_manager::log_entry_fetcher: from block number 40156, latest block number 40155, confirmation delay 12
2024-05-08T19:58:29.634237Z DEBUG log_entry_sync::sync_manager::log_entry_fetcher: log sync gets entries without progress? old_progress=40156
2024-05-08T19:58:29.781690Z DEBUG discv5::service: RPC Request timed out. id: eff0861c79a5b374
2024-05-08T19:58:29.781709Z DEBUG discv5::service: RPC Request timed out. id: dab1d3a1520dd2fa
2024-05-08T19:58:29.781725Z ERROR network::discovery: Error getting mapping to ENR multiaddr=/ip4/54.219.26.22/udp/1234/p2p/16Uiu2HAmPxGNWu9eVAQPJww79J32pTJLKGcpjRMb4Qb8xxKkyuG1 error=Timeout
2024-05-08T19:58:29.781728Z ERROR network::discovery: Error getting mapping to ENR multiaddr=/ip4/52.52.127.117/udp/1234/p2p/16Uiu2HAm93Hd5azfhkGBbkx1zero3nYHvfjQYM2NtiW4R3r5bE2g error=Timeout
2024-05-08T19:58:29.781735Z DEBUG network::discovery: Starting a peer discovery request target_peers=16
2024-05-08T19:58:29.781952Z  INFO network::service: Listening established address=/ip4/0.0.0.0/tcp/1234/p2p/16Uiu2HAmArnuVcV9zYAW3HQ9Ym2srKcte8WWzd6XLSgf4ZpTdHDV
2024-05-08T19:58:29.781973Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/find_file/ssz_snappy
2024-05-08T19:58:29.781977Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/find_chunks/ssz_snappy
2024-05-08T19:58:29.781979Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/announce_file/ssz_snappy
2024-05-08T19:58:29.781982Z DEBUG network::behaviour: Subscribed to topic topic=/eth2/announce_chunks/ssz_snappy
2024-05-08T19:58:29.781984Z  INFO network::service: Subscribed to topics topics=[FindFile, FindChunks, AnnounceFile, AnnounceChunks]
2024-05-08T19:58:29.782029Z DEBUG sync::service: Starting sync service
2024-05-08T19:58:30.289929Z DEBUG miner::miner_id: Checking miner id on chain...
2024-05-08T19:58:30.304651Z DEBUG log_entry_sync::sync_manager::log_entry_fetcher: from block number 40156, latest block number 40155, confirmation delay 12
2024-05-08T19:58:30.304663Z DEBUG log_entry_sync::sync_manager::log_entry_fetcher: log sync gets entries without progress? old_progress=40156
2024-05-08T19:58:30.459927Z ERROR zgs_node: Failed to start zgs node reason=Fail to query miner id information: AbiError(DecodingError(InvalidName("please ensure the contract and method you're calling exist! failed to decode empty bytes. if you're using jsonrpc this is likely due to jsonrpc returning `0x` in case contract or method don't exist")))
2024-05-08T19:58:30.459960Z  INFO zgs_node::client::environment: Internal shutdown received reason="Failed to start zgs node"
2024-05-08T19:58:30.459965Z  INFO discv5::service: Discv5 Service shutdown
2024-05-08T19:58:30.459971Z  INFO zgs_node: Shutting down... reason=Failure("Failed to start zgs node")

There seems to be problem getting peers when behind NAT. I tried even using UPNP it still does not seems to be able to get peers connected.