protocol support > 14 about tzindex

[echa]

Hi, the store.metadata_not_found error looks like the node is not an archive node and has pruned older blocks already (this happens with rolling nodes, they only keep block metadata for 5 cycles). If it is an archive node it is probably not running with --metadata-size-limit unlimited. In that case there's not much to do other than resync or temporarily use another archive node like rpc.tzstats.com

Oh the Lima upgrade, sorry for the lag. Our PRO version is updated, but there was no time and no funding to also update the open source release yet. I see if I can get this done early next week. Mumbai work is waiting for funding too.

[tmeinlschmidt]

Thanks, @echa will check once again our tezos nodes

[tmeinlschmidt]

@echa,
hmm, so checked - we're running with --network mainnet --history-mode archive --metadata-size-limit unlimited. Is there any other way I can validate the data and metadata in tezos itself?

when I try octez rpc get /chains/main/checkpoint I get

{ "block":
   { "level": 3047425, "proto": 15,
     "predecessor": "BLT6NBvmGXF7wvMtCrxzitnmVc9stZobjbx2zcxHBDRYkfYa9rG",
     "timestamp": "2023-01-11T06:27:14Z", "validation_pass": 4,
     "operations_hash":
       "LLoadpZD3TVk6ZQyos12NRS43tBaaD4hp5WDtnFN3gsETNHwo95EP",
     "fitness": [ "02", "002e8001", "", "ffffffff", "00000000" ],
     "context": "CoVSu2ruJvYmoqaCF5JK4gnWRrLPBUgFUfsWui4QquGWTxziTPGS",
     "protocol_data":
       "dfba531e7820aafc63b539562a895ed18d7c46ea9ff796577264b11e2643aef400000000763259c59f7002000000ffa3a61e5506db123d56ae5b81e84e8d38e23521cb8fe4c5988b9923c96656555935d1db585eb82f9de149ab318921221123b3137f54c99c71bcfada3672a104" },
 "savepoint": 0, "caboose": 0, "history_mode": "archive" }

[echa]

Can you successfully fetch blocks 2793471 BKjV8TaCuygYuh5jiSSG8bpX1nhsNHA1xwhdoMgG9pE6zakgKgc and 2793472 BLqfmdk3he2DoiaoR8U8CPubefpFq4EkaP9VmhK8P9NjVGKJaQ1 (they should contain metadata). You can also append ?metadata=always to the URL (don't know about octez-client) to make the node reconstruct pruned/missing metadata for that block and its operations.

[tmeinlschmidt]

{ "protocol": "PtKathmankSpLLDALzWw7CGD2j2MtyveTwboEYokqUCP4a1LxMg",
  "chain_id": "NetXdQprcVkpaWU",
  "hash": "BKjV8TaCuygYuh5jiSSG8bpX1nhsNHA1xwhdoMgG9pE6zakgKgc",
  "header":
    { "level": 2793471, "proto": 14,
      "predecessor": "BMDMopLtrwUTSG3PCBKryKRoQgQGKZJi27KRW1pjyqgvq8fJWwZ",
      "timestamp": "2022-10-14T00:46:14Z", "validation_pass": 4,
      "operations_hash":
        "LLob34Tbeu32x3eVYAvZ3ZL3T81h8eGEUuCDh1WeWG2AjG7eWBcTB",
      "fitness": [ "02", "002a9fff", "", "ffffffff", "00000000" ],
      "context": "CoVf7AicBNaAYDFqGm4Yv5buqPrFtRp6VqoS7yfTczDJ7fxPhasd",
      "payload_hash": "vh2oVz4oBdXkH3RTTZejGvRuzqsHp7dYqL6nkyhb65chYv9XzyaQ",
      "payload_round": 0, "proof_of_work_nonce": "4ca331941c230100",
      "liquidity_baking_toggle_vote": "pass",
      "signature":
        "sigb2xSZQwZSFhoritnbAMmPjF1WTG7RoWn2SuZ86bSEkjtqRtkrzbwG1BuNRUKsga4Grc6GWRnRmDKCNZnvZP2M1mcv5P5Y" },
  "metadata":
    { "protocol": "PtKathmankSpLLDALzWw7CGD2j2MtyveTwboEYokqUCP4a1LxMg",
      "next_protocol": "PtKathmankSpLLDALzWw7CGD2j2MtyveTwboEYokqUCP4a1LxMg",
      "test_chain_status": { "status": "not_running" },
      "max_operations_ttl": 120, "max_operation_data_length": 32768,
      "max_block_header_length": 289,
      "max_operation_list_length":
        [ { "max_size": 4194304, "max_op": 2048 }, { "max_size": 32768 },
          { "max_size": 135168, "max_op": 132 }, { "max_size": 524288 } ],

and

{ "protocol": "PtKathmankSpLLDALzWw7CGD2j2MtyveTwboEYokqUCP4a1LxMg",
  "chain_id": "NetXdQprcVkpaWU",
  "hash": "BLqfmdk3he2DoiaoR8U8CPubefpFq4EkaP9VmhK8P9NjVGKJaQ1",
  "header":
    { "level": 2793472, "proto": 14,
      "predecessor": "BKjV8TaCuygYuh5jiSSG8bpX1nhsNHA1xwhdoMgG9pE6zakgKgc",
      "timestamp": "2022-10-14T00:46:44Z", "validation_pass": 4,
      "operations_hash":
        "LLob82DESKwKeZUgFsAAgmggosSHGhoKXe7iwT6nJGpzgdd6G947r",
      "fitness": [ "02", "002aa000", "", "ffffffff", "00000000" ],
      "context": "CoVKFuKrXvWnp3tPN8H6ouenziAktdWsPc5B4eFiqhUDF6fnRF3G",
      "payload_hash": "vh3LxXXS4F2hkeC5Famuzq8pjSpiR3K4UW8QiLuyErUo8mERoqri",
      "payload_round": 0, "proof_of_work_nonce": "073ae29554340200",
      "seed_nonce_hash":
        "nceVVvpcouW8ZibtayKQkk6wDxZrEmPM1ime3aF1D7ZZ4cXtdRRF8",
      "liquidity_baking_toggle_vote": "off",
      "signature":
        "sigcf21V6pQS4Lc6whsejJ55PB93DsjN42srvY4bVwPayWpzGRF2V7gbc7T7dCmfCr7fYqCzBxfPLQptru2Mj78d2mxWrisz" },
  "metadata":
    { "protocol": "PtKathmankSpLLDALzWw7CGD2j2MtyveTwboEYokqUCP4a1LxMg",
      "next_protocol": "PtKathmankSpLLDALzWw7CGD2j2MtyveTwboEYokqUCP4a1LxMg",
      "test_chain_status": { "status": "not_running" },
      "max_operations_ttl": 120, "max_operation_data_length": 32768,
      "max_block_header_length": 289,
      "max_operation_list_length":
        [ { "max_size": 4194304, "max_op": 2048 }, { "max_size": 32768 },
          { "max_size": 135168, "max_op": 132 }, { "max_size": 524288 } ],

so that worked (both octez/curl, including metadata), restarted tzindex, started from 2793473 - with the same error, so I tried via curl go further and ended up with block 2793474

[
  {
    "kind": "permanent",
    "id": "store.metadata_not_found",
    "block_metadata_not_found": "BLcZXFe3mrEchBVJmUzrWBMJKEEkdnnQZ2MBnEA5n42xMhy2cqx"
  }
]

[echa]

Argh. TzIndex v14 already calls with /chains/main/blocks/2793474?metadata=always, so it looks like the node has corrupt storage. This happened to some of our archive nodes before when they went out of disk space during a database merge (the new Octez versions since v15 or v14 seem to no longer gracefully handle this case).

Your best paths going forward are

• reconstruct the node off a tarball from https://xtz-shots.io/ as resync from scratch will take weeks
• sync tzindex against https://rpc.tzstats.com then switch to your local rolling node

Please keep in mind TzTindex v14 will stop at the Lima migration as it does not know the new protocol hash. I'll squeeze in some time Monday to make the v15 release ready.

[tmeinlschmidt]

We didn't run ouf of space (few TBs left). Node was started from archive tarball (redownloaded it yesterday once again, 600GB+), started and let it sync, but ended up on the same. tzindex stopped with 2793473, curl to 2793474 is the same about missing metadata.

About v14 it's all clear, will wait to your spare time. But I need to resolve that weird issue - we have three nodes with the same result, all from archive tarball...

-- edit --

found that three of our six nodes restored from latest snapshot (3xxxxxx block) are corrupted, remaining 3 nodes running from snapshot 2760849 are correct.

[echa]

The Lima version is available now. Thanks for your patience.

[tmeinlschmidt]

Thanks!