blockwatch-cc / tzindex Goto Github PK
View Code? Open in Web Editor NEWTezos Blockchain Indexer
License: MIT License
tzindex's People
Contributors
Stargazers
Watchers
Forkers
ledgerhq os2357 superclassiceth reyawongkraso1 alexocool theblockchaincoder markysha pinkdiamond1 nazt tmeinlschmidt mavryk-network captaketzindex's Issues
out of sync 6.0.3
stopped at an altitude: 984006
config.json
{ "server": { "addr": "0.0.0.0", "port": 8000, "scheme": "http", "host": "127.0.0.1", "workers": 50, "read_timeout": "5s", "header_timeout": "10s", "write_timeout": "1000s", "keepalive": "90s", "shutdown_timeout": "15s", "max_list_count": 50000, "default_explore_count": 20, "max_explore_count": 100, "cors_enable": false, "cors_origin": "*", "cors_allow_headers": "Authorization, Accept, Content-Type, X-Api-Key, X-Requested-With", "cors_expose_headers": "Date, X-Runtime, X-Request-Id, X-Api-Version", "cors_methods": "GET, OPTIONS", "cors_maxage": "86400", "cors_credentials": "true", "cache_enable": false, "cache_control": "public" }, "crawler": { "queue": 1000, "snapshot_path": "./db/xtz/snapshots", "snapshot_blocks": [], "snapshot_interval": 0 }, "database": { "path": "./db/xtz", "engine": "bolt", "gc_interval": 50000000, "gc_ratio": 1.0, "log_slow_queries": "2s", "account": { "cache_size": 4 }, "account_index":{ "cache_size": 8 }, "contract": { "cache_size": 2 }, "contract_index":{ "cache_size": 2 }, "op": { "cache_size": 4 }, "op_index":{ "cache_size": 128 }, "block": { "cache_size": 8 }, "block_index":{ "cache_size": 64 }, "flow": { "cache_size": 4 }, "supply": { "cache_size": 2 }, "chain": { "cache_size": 2 }, "right": { "cache_size": 2 }, "income": { "cache_size": 2 } }, "rpc": { "host": "127.0.0.1", "port": 8732, "threads": 10, "disable_tls": true, "dial_timeout": "20s", "keepalive": "30m", "idle_timeout": "30m", "response_timeout": "60s", "continue_timeout": "60s", "idle_conns": 50 }, "logging": { "backend": "stdout", "flags": "date,time,micro,utc", "level": "info", "blockchain": "info", "database": "info", "rpc": "info", "server": "debug", "report": "info", "micheline": "info" } }
There is something not clear in the logs
363436327451000231000c000000383130303030303030303000000435353700440000000230002600000062747a314c6b4579426139375568795a6a485059505532705648714472353770524c654161000231000b0000003630303630303030303000000435353800440000000230002600000062747a314c6b475259664d6957783768747161505937676738743278516679744567514b44000231000b0000003231373837353030303000000435353900430000000230002600000062747a314c6b4a35614155344662476e664a4a7459766a6e7839473662624d5872536e5843000231000a00000035323530303030303000000435363000440000000230002600000062747a314c6b4e764c5a3171446475727370716d4d595a5a763861357a747155505a733847000231000b0000003330303030303030303000000435363100440000000230002600000062747a314c6b504e7a4457484d6376337846695a697669704247684b587a466d7758484878000231000b0000003234323539353432333000000435363200440000000230002600000062747a314c6b50727831586f346e6568684771694d576b616f774831545a4d35686b45484c000231000b0000003231363730393537303500000435363300440000000230002600000062747a314c6b5331787278664b786470326a486a6237594b78585a5a453458416859444137000231000b0000003437393530333530303000000435363400440000000230002600000062747a314c6b54555065326974486654653456694d47755a3541615275644863455974356e000231000b0000003630303030303030303000000435363500440000000230002600000062747a314c6b55535741696a41785a41423477315a565246375942384a566f614c75524643000231000b0000003537333835303030303000000435363600430000000230002600000062747a314c6b556a39666831774874436e314d4d584d766e37383852793247546f61323377000231000a00000038363930343636383300000435363700430000000230002600000062747a314c6b564c7956673535433945686e384d4d785668435a66416a6762563876677469000231000a00000035353335383736393200000435363800430000000230002600000062747a314c6b566e734341397867316e577334346a774268643865565371466f507a666161000231000a00000035373630343333343000000435363900440000000230002600000062747a314c6b5853363337596179453342484c4861685757354a46797877766457484b546e000231000b0000003133383039323939343000000435373000440000000230002600000062747a314c6b594773736f4645657237794c5a665367516a635a4166663347447a52733277000231000b0000003132313934313537363000000435373100440000000230002600000062747a314c6b63745731666b6a333373534e7347567755326a4e435143337a455265346b51000231000b0000003632363031343336323000000435373200420000000230002600000062747a314c6b67344c6379326e426e78334433624376763578757161386245615670454a710002310009000000363338343539393700000435373300440000000230002600000062747a314c6b6869624c4b6175724746563358487766474b685833324e4869525264727674000231000b0000003231363935393931303000000435373400430000000230002600000062747a314c6b69425336794b7976576667357052734236386d79696f66525633536375436f000231000a00000034313937373530303000000435373500430000000230002600000062747a314c6b6a6755344a4362666d6834483932717154534b55697337767147456a4e6539000231000a00000035303030303030303000000435373600430000000230002600000062747a314c6b6b375266364e71686a375739636d526b62787166536b367757637669557231000231000a00000036383631333832323000000435373700440000000230002600000062747a314c6b6d4a5a664433565239595842635363316f7a62384232643677626e70457647000^C
ERROR: Repository not found.
When trying to build the binary I get the following error:
go: blockwatch.cc/[email protected]: invalid version: git fetch -f https://github.com/blockwatch-cc/tzwatch refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /home/marc/go/pkg/mod/cache/vcs/d5d86cc6751cf0e24701351e42caf2573fed5966931c6e0742de0adb8869746b: exit status 128:
ERROR: Repository not found.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
balance 0 is smaller than outgoing amount 374 on ithacanet
when running tzindex on ithacanet, the following error is raised:
2022/04/08 12:41:50.882666 ERRO BLOC Processing block 312914 BMbfdJAZYPj9s1XJ6fA8LdHKjHPytkFjJJqQQz1bqg5xK7Rij8z: build stage 4: acc.update id 147589 tz2UPSgvCW2keRj8fF3SzCdYzJZHzVAsX5DS balance 0 is smaller than outgoing amount 374
2022/04/08 12:41:50.882985 INFO BLOC Stopping blockchain sync due to too many errors at 312913.
2022/04/08 12:41:51.831792 INFO BLOC Exiting monitor loop on cancel.
2022/04/08 12:41:51.832694 INFO BLOC Stopping blockchain ingest on cancel.
This is with tzindex on master.
Cannot access tzindex server
When I launch tzindex, I cannot access it:
As such I amnot able to connect tzstats to it. Is a further config required?
Contract NamedBigmaps method returns invalid result
Hi,
I've noticed that since the commit Jakarta release the method in etl/model/contract.go#NamedBigmaps (line 329) has changed :
func (c *Contract) NamedBigmaps(m []*BigmapAlloc) map[string]int64 {
if len(m) == 0 {
return nil
}
_, sTyp, err := c.LoadType()
if err != nil {
return nil
}
named := make(map[string]int64)
// find bigmap typedefs in script
bigmaps, _ := sTyp.FindOpCodes(micheline.T_BIG_MAP)
// unpack micheline types into tzgo types for matching
// this resolves ambiguities from different comb pair expressions
types := make([]micheline.Typedef, len(bigmaps))
for i, v := range bigmaps {
types[i] = micheline.Type{v}.Typedef("")
}
// match bigmap allocs to type annotations using type comparison
for i, v := range m {
kt, vt := v.GetKeyType().Typedef(""), v.GetValueType().Typedef("")
var name string
for _, typ := range types {
if !typ.Left().Equal(kt) {
continue
}
if !typ.Right().Equal(vt) {
continue
}
name = typ.Name
// some bigmap types may be reused (different bigmap use the same type)
// so be carful not overwriting existing matches
if _, ok := named[name]; !ok {
break
}
}
// generate a unique name when annots are missing
if name == "" {
name = "bigmap_" + strconv.Itoa(i)
}
// make sure name is not a duplicate
if _, ok := named[name]; ok {
var c int
for {
n := name + "_" + strconv.Itoa(c)
if _, ok := named[n]; !ok {
name = n
break
}
c++
}
}
named[name] = v.BigmapId
}
return named
}More precisely, for an example, I've deployed the KT1RAdg1ckDd3kgyVcAXbdXGoSvMUxbwV2ty smart contract, containing some bigmaps.
I've see that the method will return a map where all the keys are ledger, ledger_0, ledger_1 instead of the effective names of the bigmap declared in the smart contract
While debugging I can provide more details:
- The first bigmap is named
ledger - Then the loop
for _, typ := range typesalways enters in the continue:if !typ.Left().Equal(kt) { continue }orif !typ.Right().Equal(kt) { continue }, so the name var keeps the previous name and an index is added by// make sure name is not a duplicate
Furthermore I don't get the intention about // match bigmap allocs to type annotations using type comparison (but I miss some understanding about type and .Typedef("")):
- given a smart contract containing two bigmaps with the same key type and value type
- when this match is done,
- then is there a risk that the method returns the first bigmap name matching instead of the expecting one ?
I have noticed it by working on custom feature I've made on a fork of your project.
I would be glad to be able to organize a meeting to introduce you this feature and see if the implementation can be integrated in this repository. I could add a pull request, but as I said previously I will be glad to talk you about the feature within a meeting.
Despite of this issue, thanks for your job !
I really appreciate !
Sébastien
Warning/Errors in tzindex logs
Hi, I've been running a set of tzindex instances for several weeks now and I first want to thank you for this nice software (some negociation is underway to buy the appropriate licence from you but that's not my part). I just noticed some recurring warning and error messages like :
2019/12/06 17:30:25.442348 WARN DATA query: op.api.list_account_ops_sent compile=16.003µs journal=56.982µs index=1.908µs scan=35.179963726s total=35.1800287s matched=44 rows, scheduled=249 packs, scanned=249 packs, searched=0 index rows
2019/12/06 17:28:24.387723 WARN DATA query: account.api.list_all_delegates compile=25.381µs journal=168.531µs index=1.129µs scan=2.011291699s total=2.01146634s matched=3960 rows, scheduled=12 packs, scanned=12 packs, searched=0 index rows
or
2019/12/06 17:32:43.266086 ERRO SRVR 863fcfcd443bcda370c33199d40a9684 404 (1305) GET /explorer/account/tz1hobi4RsAeswm9JA7mHB96vqc1eAWRzxDc/op?limit=100 HTTP/1.1 - ReadAccountOps failed (no such account): account not indexed
2019/12/06 17:36:39.710247 ERRO SRVR 7007f203f9493855def955548f26a297 404 (1305) GET /explorer/account/tz1hobi4RsAeswm9JA7mHB96vqc1eAWRzxDc/op?limit=100 HTTP/1.1 - ReadAccountOps failed (no such account): account not indexed
2019/12/06 17:36:39.850212 ERRO SRVR 00607fe14d131124cd2690f24102db57 404 (1305) GET /explorer/account/tz1hobi4RsAeswm9JA7mHB96vqc1eAWRzxDc HTTP/1.1 - ReadAccount failed (no such account): account not indexed
Despite these messages, the program looks fine and keeps indexing new data. Can you please tell me if these messages can be safely ignored or if we should consider them as a symptom of some issue somewhere. Thanks.
API endpoint for current and historical prices of FA1.2 and FA2 tokens
Hi I I would like to add my "idea" for an API endpoint for tezos token prices.
So for FA2 and FA1.2 tokens there is no endpoint to get historical price or "live" price.
The API endpoint should allow to query tokens using the contract address which is unique to a given token.
Knowing the contract address for youves uUSD
KT1XRPEPXbZK25r3Htzp2o1x7xdMMmfocKNW
I should be able to query with Tzstats and get the current prices
Now the question some might ask is "But there are more decentralised exchanges so there are different prices, how to do that?"
A very good example is how coingecko does it. They have a separate object for every exchange they gather/track the data.
I dont say Tzstats api should track every dex. But for example the top 2-3?
then you would have something like this:
{
Quipu:{
"base": "uUSD",
"target": "XTZ",
"lastprice": 1,
"volume": "157824452",
"bid_ask_spread_percentage": "0.2",
"timestamp": "2022-04-27T21:10:16+00:00",
"last_trade_at": "2022-04-26T21:10:16+00:00",
"trade_url": "quipuswap dot com/pair"
}
Spicy:{
"base": "uUSD",
"target": "XTZ",
"lastprice": 1,
"volume": "157824452",
"bid_ask_spread_percentage": "0.2",
"timestamp": "2022-04-27T21:10:16+00:00",
"last_trade_at": "2022-04-26T21:10:16+00:00",
"trade_url": "spicyswap dot com/pair"
}
}```
Then people can easily get the data and build on top of it.
Since a year people are hoping for coingecko, cmc api but lets dont wait and integrate this for Tzstats. 🙂
My example screenshot is taken from this example here.
https://api.coingecko.com/api/v3/coins/ethereum/contract/0x0d8775f648430679a709e98d2b0cb6250d2887ef
The screenshot
Support for edo protocol
Hello, I would like to ask, when will be release new docker with 008 edo protocol support,
now it fails like this on edonet:
2021/01/21 21:15:35.415749 ERRO BLOC Processing block 184 BLLT8xpajb4hPVU64VsmD7iFhHRmBLDYb27wRoBrATjk5GVLXdP: build stage 3: origination op [3:0:1]: patch bigmap: bigmap patch unknown contract KT1A4mDg25qvC3tcwyNFy87kcCpLNyfvop2M
2021/01/21 21:15:35.420797 INFO BLOC Stopping blockchain sync due to too many errors at 183.
Micheline INT prim
Hello.
Looks like Micheline INT prim for some numbers is broken. For example if I PACK { "int": "33"} by node RPC I got 0021 as result. When MarshalBinary by tzIndex Prim result is 00a100 and node Could not decode bytes
Prim example:
prim := &micheline.Prim{
Type: micheline.PrimInt,
OpCode: micheline.T_INT,
Int: big.NewInt(33),
}
API
I am trying to calculate the part of each delegator in the reward of a specific baker for some cycle.
I was using tzscan for this aim, and now I want to change to tzstats. However, the features that I need either are still missing or are not as straightforward documented as I thought. I need mainly two features:
- The possibility of getting a list of delegators of a baking address for some cycle according to the corresponding snapshot block. The only thing I get from the API is the number of delegators, but not their addresses and respective delegated balance.
- The possibility of getting more precise results for the rewards/income. What I get now is multiples of XTZ (e.g. 124 XTZ) but what I need is e.g. 124.500243 XTZ. This is indeed implemented for staking balances, but not for income info.
I hope I did not oversee these features in the API, otherwise I would be thankful to be able to get these info using tzstats API.
Not all transactions in block returned, due to default limit
Query: https://api.tzstats.com/explorer//block/882580/op
Expected result: Return all ops associated with block
Actual result Only 20 out of 31 returned
Desire solution: Return all ops by default (there not that many anyway)
Current hack: Supply limit=<wahtever_big_numebr_in_mind>
I understand reason to apply limit when query something like https://api.tzstats.com/explorer/account/tz1WDujRWCYjLBDfZieXW6insg5EUbg1rCRK/op?order=desc&type=transaction,delegation&limit=24
@echa What do you think ?
BuildType in primitives.go
I was re-using this code. Thanks a lot. great work!
Some of the cases in the BuildType function in primitives.go do not correctly pass values.
E.g. PrimInt, PrimString, and PrimBytes. Should be something like:
case PrimInt:
t.OpCode = p.Type.TypeCode()
t.Type = PrimNullary
t.Int = p.Int
case PrimString:
t.OpCode = p.Type.TypeCode()
t.Type = PrimNullary
t.String = p.String
case PrimBytes:
t.OpCode = p.Type.TypeCode()
t.Type = PrimNullary
t.Bytes = p.Bytes
"since" parameter is off by one on `/explorer/contract/ID/calls`
https://api.carthagenet.tzstats.com/explorer/contract/KT1CUTjTqf4UMf6c9A8ZA4e1ntWbLVTnvrKG/calls?prim=true&order=asc&limit=10&since=331290 has a call at height 331292, but https://api.carthagenet.tzstats.com/explorer/contract/KT1CUTjTqf4UMf6c9A8ZA4e1ntWbLVTnvrKG/calls?prim=true&order=asc&limit=10&since=331291 does not, starting instead with a call at height 331293...
Docs typo and wrong type for errors property with actual response
From docs https://tzstats.com/docs/api/index.html#operations, endpoint https://api.tzstats.com/explorer/account/tz1WDujRWCYjLBDfZieXW6insg5EUbg1rCRK/op?order=desc&type=transaction,delegation&limit=250 response errors as an array, but in docs described as an object
Docker image and --stop 'height'
Hello, how can I use docker image and tell him to --stop height ?
Is there any possibility?
my .drone.yml
- name: tz-indexer-ocaml
image: blockwatch/tzindex:latest
user: root
detach: false
volumes:
- name: indexer-data
path: /data
environment:
TZ_RPC_HOST: ocaml-node-run
TZ_RPC_PORT: 48732
TZ_SERVER_ADDR: 0.0.0.0
TZ_SERVER_PORT: 8007
TZ_SERVER_CORS_ENABLE: true
TZ_DATABASE_PATH: /data/index
TZ_SNAPSHOT_PATH: /data/snapshots
Getting error while on build command
i am getting the following error while trying make build command
build
go clean
build .: cannot find module for path .
make: *** [Makefile:20: build] Error 1
is there anything i missed?
Different result count depending on order
Tzindex does not read rpc values from config file
I modified my config file with a custom rpc path like below
"rpc": {
"host": "http://192.168.64.2",
"port": 31840,
"threads": 2,
"disable_tls": true,
"dial_timeout": "10s",
"keepalive": "30m",
"idle_timeout": "30m",
"response_timeout": "60s",
"continue_timeout": "60s",
"idle_conns": 2
...
When I execute the following commands:
./tzindex run or ./tzindex run -c config.json
it still does not switch the rpc. I get
I know its accessing the right json file
Unrecoverable error when connecting to sandbox while sandbox is starting
Steps to reproduce:
- Create file
flextesa.ymlwith the following content:
version: '3.7'
services:
sandbox:
image: registry.gitlab.com/tezos/flextesa:image-babylonbox-run
command: babylonbox start
ports:
- 20000:20000
- Create file
tzindex.yml:
version: '3.7'
services:
indexer:
image: blockwatch/tzindex
environment:
TZ_RPC_HOST: sandbox
TZ_RPC_PORT: 20000
ports:
- 8000:8000
depends_on:
- sandbox
- Initialize docker swarm:
docker swarm init
- Deploy docker stack consisting of these two services - sandbox (flextesa) and indexer (tzindex) pointing to the sandbox:
docker stack deploy -c tzindex.yml -c flextesa.yml test
- Observe tzindex logs and notice
Unrecoverable errorlogged after some seconds:
» docker service logs test_indexer --follow itkach@Igors-MacBook-Pro
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.670541 INFO MAIN Using configuration file config.json
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.670597 INFO MAIN Blockwatch tzindex v5.3.2 -- 3109ffb
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.670610 INFO MAIN (c) Copyright 2018-2020 -- KIDTSUNAMI UG
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.670622 INFO MAIN Starting Blockwatch-tzindex/v5.3.2.3109ffb on 8 cores
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.670635 INFO MAIN Go version go1.13.6
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.770784 INFO MAIN Profiling mutex/blocking at 1.00% sample rate.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.770860 INFO MAIN Using bolt database ./db/xtz
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.774384 INFO BLOC Initializing blockchain crawler in sync mode.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.774412 INFO BLOC Creating blockchain storage.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.775338 INFO BLOC Creating account index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.796761 INFO BLOC Creating contract index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.818056 INFO BLOC Creating block index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.847749 INFO BLOC Creating op index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.874028 INFO BLOC Creating flow index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.885023 INFO BLOC Creating chain index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.903453 INFO BLOC Creating supply index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.922705 INFO BLOC Creating rights index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.936550 INFO BLOC Creating snapshot index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.951671 INFO BLOC Creating income index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.974373 INFO BLOC Creating gov index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:37.998875 INFO BLOC Creating bigmap index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:38.023952 INFO BLOC Initializing account index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:38.026509 INFO BLOC Initializing contract index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:38.029012 INFO BLOC Initializing block index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:38.030599 INFO BLOC Initializing op index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:38.032090 INFO BLOC Initializing flow index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:38.033498 INFO BLOC Initializing chain index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:38.034988 INFO BLOC Initializing supply index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:38.036352 INFO BLOC Initializing rights index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:38.037707 INFO BLOC Initializing snapshot index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:38.039024 INFO BLOC Initializing income index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:38.040470 INFO BLOC Initializing gov index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:38.042135 INFO BLOC Initializing bigmap index.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:38.043660 INFO BLOC Connecting to RPC server.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:38.046906 ERRO BLOC Connection failed: Get http://sandbox:20000/chains/main/blocks/head/header: dial tcp: lookup sandbox on 127.0.0.11:53: no such host
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:43.051672 INFO BLOC Fetching genesis block.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:43.055457 INFO BLOC Crawling Tezos Mainnet.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:43.058721 INFO BLOC Starting blockchain crawler.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:43.058802 WARN SRVR HTTP Server reachable on all interfaces (0.0.0.0)
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:43.058938 INFO BLOC Starting blockchain sync from height 1.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:43.058987 INFO BLOC Starting blockchain ingest.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:43.058998 INFO BLOC Starting blockchain monitor.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:43.059030 INFO BLOC Already synchronized. Starting in monitor mode.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:43.061253 INFO SRVR Starting HTTP server at 0.0.0.0:8000
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:45.878358 INFO BLOC Building genesis dataset.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:45.882883 INFO BLOC Happy New Blockchain Quarter 2 at block 1!
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:36:45.882962 INFO BLOC Happy New Blockchain Year 2 at block 1!
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:37:01.879147 INFO BLOC New protocol PsBabyM1eUXZseaJdmXFApDSBqj8YBfwELoxZHHW77EMcAbbwAS detected at 2
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:37:01.881311 INFO BLOC Upgrade to v005: registered 0 buggy delegates
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:37:01.881337 INFO BLOC Upgrade to v005: dropped 0 empty delegates
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:37:01.881758 INFO BLOC Upgrade to v005: executed 0 airdrops
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:37:01.881835 ERRO BLOC Unrecoverable error: runtime error: index out of range [0] with length 0
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:37:01.882026 ERRO BLOC {"stack":["goroutine 37 [running]:","runtime/debug.Stack(0xc000174c00, 0xc420fd, 0x17)","\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d","blockwatch.cc/tzindex/etl.(*Crawler).syncBlockchain.func1(0xc000114018, 0xc000150380, 0xd71c60, 0xc000ea8000)","\t/go/src/tzindex/etl/crawler.go:720 +0xe5","panic(0xbd49a0, 0xc000117ee0)","\t/usr/local/go/src/runtime/panic.go:679 +0x1b2","blockwatch.cc/tzindex/etl.(*Builder).Decorate(0xc0000e1ea0, 0xd71c60, 0xc000ea8000, 0x0, 0x0, 0x79028e)","\t/go/src/tzindex/etl/builder.go:885 +0x1bc9","blockwatch.cc/tzindex/etl.(*Builder).Build(0xc0000e1ea0, 0xd71c60, 0xc000ea8000, 0xc0002543c0, 0xc000ecc640, 0x33, 0x0)","\t/go/src/tzindex/etl/builder.go:226 +0x1cb","blockwatch.cc/tzindex/etl.(*Crawler).syncBlockchain(0xc000150380)","\t/go/src/tzindex/etl/crawler.go:854 +0x44b","created by blockwatch.cc/tzindex/etl.(*Crawler).Start","\t/go/src/tzindex/etl/crawler.go:344 +0x9c"]}
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:37:01.882059 INFO BLOC Stopping blockchain sync at height 1.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:37:01.931491 INFO BLOC Context cancelled. Stopping blockchain ingest.
test_indexer.1.rydsedftrqze@docker-desktop | 2020/02/12 14:37:01.931702 ERRO BLOC Exiting monitor loop on cancelled context
"block" parameter has no effect on `/explorer/contract/ID/calls`
Choosing an example contract that was recently active on Carthagenet, https://api.carthagenet.tzstats.com/explorer/contract/KT1CUTjTqf4UMf6c9A8ZA4e1ntWbLVTnvrKG/calls?prim=true&order=desc&limit=100 gives the same JSON result as https://api.carthagenet.tzstats.com/explorer/contract/KT1CUTjTqf4UMf6c9A8ZA4e1ntWbLVTnvrKG/calls?prim=true&order=desc&limit=100&block=BLNXc4JnUyjMh14KdTqcsJRqRkBs5LVSJiYPvZnfExMLrwHqRyL and https://api.carthagenet.tzstats.com/explorer/contract/KT1CUTjTqf4UMf6c9A8ZA4e1ntWbLVTnvrKG/calls?prim=true&order=desc&limit=100&block=331300 ...
block init: empty metadata in RPC response (maybe you are not using an archive node)
With the last version in master generates the following warning and doesn't seem to index anything:
$ ./tzindex run --rpcurl https://hangzhounet.smartpy.io
2022/04/07 00:10:49.500952 WARN MAIN Missing config file, using default values.
2022/04/07 00:10:49.501023 INFO MAIN Blockwatch tzindex v12.0 dev
2022/04/07 00:10:49.501031 INFO MAIN (c) Copyright 2018-2022 Blockwatch Data Inc.
2022/04/07 00:10:49.501042 INFO MAIN Starting tzindex.blockwatch.cc/v12.0.dev on 8 cores
2022/04/07 00:10:49.501049 INFO MAIN Go version go1.16.2
2022/04/07 00:10:49.506028 INFO MAIN Using bolt database ./db
2022/04/07 00:10:49.526745 INFO BLOC Initializing blockchain crawler in sync mode.
2022/04/07 00:10:49.526816 INFO BLOC Creating blockchain storage.
2022/04/07 00:10:49.532403 INFO BLOC Creating account index.
2022/04/07 00:10:49.670791 INFO BLOC Creating balance index.
2022/04/07 00:10:49.711302 INFO BLOC Creating contract index.
2022/04/07 00:10:49.790161 INFO BLOC Creating constant index.
2022/04/07 00:10:49.858943 INFO BLOC Creating block index.
2022/04/07 00:10:49.912165 INFO BLOC Creating op index.
2022/04/07 00:10:49.969932 INFO BLOC Creating chain index.
2022/04/07 00:10:50.032731 INFO BLOC Creating supply index.
2022/04/07 00:10:50.105097 INFO BLOC Creating bigmap index.
2022/04/07 00:10:50.215616 INFO BLOC Creating metadata index.
2022/04/07 00:10:50.284062 INFO BLOC Initializing account index.
2022/04/07 00:10:50.311522 INFO BLOC Initializing balance index.
2022/04/07 00:10:50.318882 INFO BLOC Initializing contract index.
2022/04/07 00:10:50.330374 INFO BLOC Initializing constant index.
2022/04/07 00:10:50.338913 INFO BLOC Initializing block index.
2022/04/07 00:10:50.354079 INFO BLOC Initializing op index.
2022/04/07 00:10:50.376303 INFO BLOC Initializing chain index.
2022/04/07 00:10:50.392760 INFO BLOC Initializing supply index.
2022/04/07 00:10:50.402942 INFO BLOC Initializing bigmap index.
2022/04/07 00:10:50.414582 INFO BLOC Initializing metadata index.
2022/04/07 00:10:50.421569 INFO BLOC Connecting to RPC server.
2022/04/07 00:10:50.737055 INFO BLOC Fetching genesis block.
2022/04/07 00:10:50.825675 INFO BLOC Crawling Tezos Hangzhounet2.
2022/04/07 00:10:50.893755 INFO BLOC Block cache with 1 entries built in 56.043664ms
2022/04/07 00:10:50.895104 INFO BLOC Starting blockchain crawler.
2022/04/07 00:10:50.895245 WARN SRVR HTTP Server reachable on localhost only
2022/04/07 00:10:50.895426 INFO BLOC Starting blockchain sync from height 1.
2022/04/07 00:10:50.895461 INFO BLOC Starting blockchain ingest.
2022/04/07 00:10:50.895587 INFO BLOC Starting blockchain monitor.
2022/04/07 00:10:50.899000 INFO SRVR Starting HTTP server at 127.0.0.1:8000
2022/04/07 00:10:52.652406 WARN BLOC crawler: RPC connection error: block init: empty metadata in RPC response (maybe you are not using an archive node)
2022/04/07 00:11:01.768824 INFO BLOC Processed 1 block in 12.24s (0 transactions, height 0, 2021-11-04 15:00:00 +0000 UTC, q=0, t=1.753921ms, s=syncing)
2022/04/07 00:11:02.080203 WARN BLOC crawler: RPC connection error: block init: empty metadata in RPC response (maybe you are not using an archive node)
2022/04/07 00:11:10.532895 WARN BLOC crawler: RPC connection error: block init: empty metadata in RPC response (maybe you are not using an archive node)
2022/04/07 00:11:20.364669 INFO BLOC Processed 2 blocks in 18.59s (0 transactions, height 0, 2021-11-04 15:00:00 +0000 UTC, q=0, t=14.830497ms, s=syncing)
2022/04/07 00:11:20.540038 WARN BLOC crawler: RPC connection error: block init: empty metadata in RPC response (maybe you are not using an archive node)
I've tried with several nodes with the same result.
Any idea what may be wrong?
[v6.0.3] Access-Control-Allow-Origin: "null", can't switch to "*"
Hi,
Starting using tzindex, looklike to be a great explorer api for tezos, however I found that the CORS policy is not set as intended:
Not working as excepted:
- Excepted to be set to
Access-Control-Allow-Origin: *when using arg--enable-cors - Excepted to be set to
Access-Control-Allow-Origin: *when using docker env variable'TZ_SERVER_CORS_ORIGIN=*'to set it toAccess-Control-Allow-Origin: *
I got it set toAccess-Control-Allow-Origin:for both case
Working as excepted:
However the docker env var with something else that * like TZ_SERVER_CORS_ORIGIN=http://example.com works !
i'm using the docker image on docker hub tag:v6.0.3
Example to reproduce, http://172.17.0.1:8732 is a valid tezos node,
docker run -it --rm \
-v /data/tzindex:/data:rw -p 0.0.0.0:8000:8000 \
-e 'TZ_SERVER_CORS_ORIGIN=*' \
blockwatch/tzindex:v6.0.3 \
tzindex run --enable-cors --dbpath /data --rpcurl http://172.17.0.1:8732Then have a look to the http header:
curl -v http://localhost:8000 serach Access-Control-Allow-Origin header
Is baker capacity accurate in V12?
Hi all,
I was looking at the baker capacity logic in v12 for my project, and I could not get my numbers to "match" those on tzstats.
After finding this code, I'm unsure what the logic is
If we look just at the v12 code
func (b Baker) StakingCapacity(p *tezos.Params, netRolls int64) int64 {
[...]
// 10% of staking balance must be locked
// deposit = stake * frozen_deposits_percentage / 100
// max_stake = total_balance / frozen_deposits_percentage * 100
// max_delegation = max_stake - total_balance
deposit_cap := util.Min64(b.DepositsLimit, b.TotalBalance())
if deposit_cap < 0 {
deposit_cap = b.TotalBalance()
}
return deposit_cap*100/int64(p.FrozenDepositsPercentage) - b.TotalBalance()
}I understand deposit_cap which is the max a baker is willing / can deposit.
Then deposit_cap*100/int64(p.FrozenDepositsPercentage) makes sense as the max stake the baker represents I guess.
But then b.TotalBalance() is subtracted from it. I can make sense of that. The baker's balance participates in the total stake it represents, but if you want to know its capacity to accept delegates, then you have to remove the baker's funds. I'm ok if that's how StakingCapacity is defined
But then the tzstats website, to show the available capacity, seems to do
StakingCapacity - StakingBalance with StakingBalance including the baker's balance.
(https://github.com/blockwatch-cc/tzstats/blob/ee6e347a074600ca1e60b53b364ec477ad906bc5/src/components/Accounts/AccountInfo/AccountInfo.js#L153)
It seems to me that the baker balance is subtracted twice resulting in an overestimation of the used capacity.
I must say I don't know very well everything, and obviously the v12 logic is very new but something feels not right to me. Could you please let me know if I'm missing something
Let me try to make it clearer with an example directly from tzstats, extracting the baker json (truncated)
{
"address": "tz1d6Fx42mYgVFnHUW8T8A7WBfJ6nD9pVok8",
"baker_since": "2018-09-25T12:49:55Z",
"grace_period": 475,
"baker_version": "61fed540",
"total_balance": 113277.444274,
"spendable_balance": 388.453791,
"frozen_balance": 112888.990483,
"delegated_balance": 1030292.164604,
"staking_balance": 1143569.608878,
"staking_capacity": 1019496.998466,
"deposits_limit": null,
}staking_capacity is 10 * total_balance - total_balance, consistent with the tzindex code above
But then, this baker shows a free capacity of -124,073 XTZ in the front end, which is staking_capacity - staking_balance, thus discounting twice the baker's total_balance, right?
Incorrect balances for the Foundation contracts
There are 16 smart contracts without origination operations that are delegated to TF bakers. Their balances are null in tzstats:
https://api.tzstats.com/tables/snapshot?address=KT1Msatnmdy24sQt6knzpALs4tvHfSPPduA2&columns=balance,delegate,cycle&order=desc&is_selected=1
'KT1TcAHw5gpejyemwRtdNyFKGBLc4qwA5gtw',
'KT1VsSxSXUkgw6zkBGgUuDXXuJs9ToPqkrCg',
'KT1Um7ieBEytZtumecLqGeL56iY6BuWoBgio',
'KT1THsDNgHtN56ew9VVCAUWnqPC81pqAxCEp',
'KT1RUT25eGgo9KKWXfLhj1xYjghAY1iZ2don',
'KT1Q1kfbvzteafLvnGz92DGvkdypXfTGfEA3',
'KT1FuFDZGdw86p6krdBUKoZfEMkcUmezqX5o',
'KT1CSKPf2jeLpMmrgKquN2bCjBTkAcAdRVDy',
'KT1WPEis2WhAc2FciM2tZVn8qe6pCBe9HkDp',
'KT1FN5fcNNcgieGjzxbVEPWUpJGwZEpzNGA8',
'KT1QuofAgnsWffHzLA7D78rxytJruGHDe7XG',
'KT1SLWhfqPtQq7f4zLomh8BNgDeprF9B6d2M',
'KT1PDAELuX7CypUHinUgFgGFskKs7ytwh5Vw',
'KT1Cz7TyVFvHxXpxLS57RFePrhTGisUpPhvD',
'KT1A56dh8ivKNvLiLVkjYPyudmnY2Ti5Sba3',
'KT1EWLAQGPMF2uhtVRPaCH2vtFVN36Njdr6z'
Cannot kill tzindex and restart it
Just installed tzindex (extracted binary from the public latest image and inserted in my own base image). Run in a statefulset in Kubernetes cluster. If I stop and restart the pod, tzindex refuses to restart.
Here are the logs
2019/11/09 15:48:04.836801 INFO MAIN Using configuration file /app/config.json
2019/11/09 15:48:04.836837 INFO MAIN Blockwatch tzindex v005-7-gf124983 -- f124983
2019/11/09 15:48:04.836840 INFO MAIN (c) Copyright 2018-2019 -- KIDTSUNAMI UG
2019/11/09 15:48:04.836843 INFO MAIN Starting Blockwatch-tzindex/v005-7-gf124983.f124983 on 8 cores
2019/11/09 15:48:04.936929 INFO MAIN Profiling mutex/blocking at 1.00% sample rate.
2019/11/09 15:48:04.936982 INFO MAIN Using bolt database /data/db/xtz
2019/11/09 15:48:04.938352 DEBG DATA Statedb database opened successfully.
2019/11/09 15:48:04.938504 INFO BLOC Initializing blockchain crawler in sync mode.
2019/11/09 15:48:04.938937 INFO BLOC Initializing account index.
2019/11/09 15:48:04.940115 DEBG DATA Account database opened successfully.
2019/11/09 15:48:04.940152 DEBG DATA Opening table account with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:16, CacheSize:4, FillLevel:0}
2019/11/09 15:48:04.940459 DEBG DATA pack: account table loaded journal with 0 entries
2019/11/09 15:48:04.940469 DEBG DATA pack: account table loaded tombstone with 0 entries
2019/11/09 15:48:04.940473 DEBG DATA pack: account table loading package headers from bucket
2019/11/09 15:48:04.940478 DEBG DATA pack: account table loaded 0 package headers
2019/11/09 15:48:04.940485 DEBG DATA Opening account_hash index with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:16, CacheSize:8, FillLevel:0}
2019/11/09 15:48:04.940503 DEBG DATA pack: loaded account_hash_index/ index journal with 0 entries
2019/11/09 15:48:04.940515 DEBG DATA pack: index account_hash_index/ loaded tombstone with 0 entries
2019/11/09 15:48:04.940520 DEBG DATA pack: account_hash_index/ index loading package headers from bucket
2019/11/09 15:48:04.940525 DEBG DATA pack: account_hash_index/ index loaded 0 package headers
2019/11/09 15:48:04.940535 INFO BLOC Initializing contract index.
2019/11/09 15:48:04.941792 DEBG DATA Contract database opened successfully.
2019/11/09 15:48:04.941814 DEBG DATA Opening table contract with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:10, CacheSize:2, FillLevel:0}
2019/11/09 15:48:04.941895 DEBG DATA pack: contract table loaded journal with 0 entries
2019/11/09 15:48:04.941902 DEBG DATA pack: contract table loaded tombstone with 0 entries
2019/11/09 15:48:04.941905 DEBG DATA pack: contract table loading package headers from bucket
2019/11/09 15:48:04.941909 DEBG DATA pack: contract table loaded 0 package headers
2019/11/09 15:48:04.941913 DEBG DATA Opening contract_hash index with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:15, CacheSize:2, FillLevel:0}
2019/11/09 15:48:04.941923 DEBG DATA pack: loaded contract_hash_index/ index journal with 0 entries
2019/11/09 15:48:04.941931 DEBG DATA pack: index contract_hash_index/ loaded tombstone with 0 entries
2019/11/09 15:48:04.941934 DEBG DATA pack: contract_hash_index/ index loading package headers from bucket
2019/11/09 15:48:04.941937 DEBG DATA pack: contract_hash_index/ index loaded 0 package headers
2019/11/09 15:48:04.941943 INFO BLOC Initializing block index.
2019/11/09 15:48:04.943051 DEBG DATA Block database opened successfully.
2019/11/09 15:48:04.943070 DEBG DATA Opening table block with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:15, CacheSize:8, FillLevel:0}
2019/11/09 15:48:04.943257 DEBG DATA pack: block table loaded journal with 0 entries
2019/11/09 15:48:04.943265 DEBG DATA pack: block table loaded tombstone with 0 entries
2019/11/09 15:48:04.943268 DEBG DATA pack: block table loading package headers from bucket
2019/11/09 15:48:04.943272 DEBG DATA pack: block table loaded 0 package headers
2019/11/09 15:48:04.943280 DEBG DATA Opening block_hash index with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:16, CacheSize:64, FillLevel:0}
2019/11/09 15:48:04.943290 DEBG DATA pack: loaded block_hash_index/ index journal with 0 entries
2019/11/09 15:48:04.943295 DEBG DATA pack: index block_hash_index/ loaded tombstone with 0 entries
2019/11/09 15:48:04.943298 DEBG DATA pack: block_hash_index/ index loading package headers from bucket
2019/11/09 15:48:04.943301 DEBG DATA pack: block_hash_index/ index loaded 0 package headers
2019/11/09 15:48:04.943305 INFO BLOC Initializing op index.
2019/11/09 15:48:04.944823 DEBG DATA Op database opened successfully.
2019/11/09 15:48:04.944848 DEBG DATA Opening table op with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:16, CacheSize:4, FillLevel:0}
2019/11/09 15:48:04.945026 DEBG DATA pack: op table loaded journal with 0 entries
2019/11/09 15:48:04.945037 DEBG DATA pack: op table loaded tombstone with 0 entries
2019/11/09 15:48:04.945040 DEBG DATA pack: op table loading package headers from bucket
2019/11/09 15:48:04.945108 DEBG DATA pack: op table loaded 5 package headers
2019/11/09 15:48:04.945118 DEBG DATA Opening op_hash index with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:16, CacheSize:128, FillLevel:0}
2019/11/09 15:48:04.945134 DEBG DATA pack: loaded op_hash_index/ index journal with 0 entries
2019/11/09 15:48:04.945140 DEBG DATA pack: index op_hash_index/ loaded tombstone with 0 entries
2019/11/09 15:48:04.945143 DEBG DATA pack: op_hash_index/ index loading package headers from bucket
2019/11/09 15:48:04.945155 DEBG DATA pack: op_hash_index/ index loaded 5 package headers
2019/11/09 15:48:04.945163 INFO BLOC Initializing flow index.
2019/11/09 15:48:04.946316 DEBG DATA Flow database opened successfully.
2019/11/09 15:48:04.946343 DEBG DATA Opening table flow with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:16, CacheSize:4, FillLevel:0}
2019/11/09 15:48:04.946428 DEBG DATA pack: flow table loaded journal with 0 entries
2019/11/09 15:48:04.946443 DEBG DATA pack: flow table loaded tombstone with 0 entries
2019/11/09 15:48:04.946446 DEBG DATA pack: flow table loading package headers from bucket
2019/11/09 15:48:04.946483 DEBG DATA pack: flow table loaded 9 package headers
2019/11/09 15:48:04.946491 INFO BLOC Initializing chain index.
2019/11/09 15:48:04.947794 DEBG DATA Chain database opened successfully.
2019/11/09 15:48:04.947814 DEBG DATA Opening table chain with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:16, CacheSize:2, FillLevel:0}
2019/11/09 15:48:04.947968 DEBG DATA pack: chain table loaded journal with 0 entries
2019/11/09 15:48:04.947978 DEBG DATA pack: chain table loaded tombstone with 0 entries
2019/11/09 15:48:04.947981 DEBG DATA pack: chain table loading package headers from bucket
2019/11/09 15:48:04.947984 DEBG DATA pack: chain table loaded 0 package headers
2019/11/09 15:48:04.947989 INFO BLOC Initializing supply index.
2019/11/09 15:48:04.949185 DEBG DATA Supply database opened successfully.
2019/11/09 15:48:04.949208 DEBG DATA Opening table supply with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:16, CacheSize:2, FillLevel:0}
2019/11/09 15:48:04.949331 DEBG DATA pack: supply table loaded journal with 0 entries
2019/11/09 15:48:04.949344 DEBG DATA pack: supply table loaded tombstone with 0 entries
2019/11/09 15:48:04.949349 DEBG DATA pack: supply table loading package headers from bucket
2019/11/09 15:48:04.949355 DEBG DATA pack: supply table loaded 0 package headers
2019/11/09 15:48:04.949360 INFO BLOC Initializing rights index.
2019/11/09 15:48:04.950686 DEBG DATA Rights database opened successfully.
2019/11/09 15:48:04.950713 DEBG DATA Opening table rights with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:16, CacheSize:2, FillLevel:0}
2019/11/09 15:48:04.950775 DEBG DATA pack: rights table loaded journal with 0 entries
2019/11/09 15:48:04.950783 DEBG DATA pack: rights table loaded tombstone with 0 entries
2019/11/09 15:48:04.950786 DEBG DATA pack: rights table loading package headers from bucket
2019/11/09 15:48:04.951032 DEBG DATA pack: rights table loaded 132 package headers
2019/11/09 15:48:04.951043 INFO BLOC Initializing snapshot index.
2019/11/09 15:48:04.952435 DEBG DATA Snapshot database opened successfully.
2019/11/09 15:48:04.952456 DEBG DATA Opening table snapshot with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:17, CacheSize:2, FillLevel:0}
2019/11/09 15:48:04.952532 DEBG DATA pack: snapshot table loaded journal with 0 entries
2019/11/09 15:48:04.952540 DEBG DATA pack: snapshot table loaded tombstone with 0 entries
2019/11/09 15:48:04.952543 DEBG DATA pack: snapshot table loading package headers from bucket
2019/11/09 15:48:04.952562 DEBG DATA pack: snapshot table loaded 5 package headers
2019/11/09 15:48:04.952567 INFO BLOC Initializing income index.
2019/11/09 15:48:04.953749 DEBG DATA Income database opened successfully.
2019/11/09 15:48:04.953768 DEBG DATA Opening table income with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:17, CacheSize:2, FillLevel:0}
2019/11/09 15:48:04.953898 DEBG DATA pack: income table loaded journal with 0 entries
2019/11/09 15:48:04.953909 DEBG DATA pack: income table loaded tombstone with 0 entries
2019/11/09 15:48:04.953912 DEBG DATA pack: income table loading package headers from bucket
2019/11/09 15:48:04.953919 DEBG DATA pack: income table loaded 0 package headers
2019/11/09 15:48:04.953923 INFO BLOC Initializing gov index.
2019/11/09 15:48:04.955021 DEBG DATA Gov database opened successfully.
2019/11/09 15:48:04.955039 DEBG DATA Opening table election with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:15, CacheSize:2, FillLevel:0}
2019/11/09 15:48:04.955599 DEBG DATA pack: election table loaded journal with 0 entries
2019/11/09 15:48:04.955612 DEBG DATA pack: election table loaded tombstone with 0 entries
2019/11/09 15:48:04.955769 DEBG DATA pack: election table loading package headers from bucket
2019/11/09 15:48:04.955858 DEBG DATA pack: election table loaded 0 package headers
2019/11/09 15:48:04.955895 DEBG DATA Opening table proposal with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:15, CacheSize:2, FillLevel:0}
2019/11/09 15:48:04.956176 DEBG DATA pack: proposal table loaded journal with 0 entries
2019/11/09 15:48:04.956185 DEBG DATA pack: proposal table loaded tombstone with 0 entries
2019/11/09 15:48:04.956189 DEBG DATA pack: proposal table loading package headers from bucket
2019/11/09 15:48:04.956194 DEBG DATA pack: proposal table loaded 0 package headers
2019/11/09 15:48:04.956200 DEBG DATA Opening table vote with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:15, CacheSize:2, FillLevel:0}
2019/11/09 15:48:04.956312 DEBG DATA pack: vote table loaded journal with 0 entries
2019/11/09 15:48:04.956324 DEBG DATA pack: vote table loaded tombstone with 0 entries
2019/11/09 15:48:04.956328 DEBG DATA pack: vote table loading package headers from bucket
2019/11/09 15:48:04.956333 DEBG DATA pack: vote table loaded 0 package headers
2019/11/09 15:48:04.956339 DEBG DATA Opening table ballot with opts pack.Options{PackSizeLog2:0, JournalSizeLog2:15, CacheSize:2, FillLevel:0}
2019/11/09 15:48:04.956389 DEBG DATA pack: ballot table loaded journal with 0 entries
2019/11/09 15:48:04.956397 DEBG DATA pack: ballot table loaded tombstone with 0 entries
2019/11/09 15:48:04.956400 DEBG DATA pack: ballot table loading package headers from bucket
2019/11/09 15:48:04.956404 DEBG DATA pack: ballot table loaded 0 package headers
2019/11/09 15:48:04.956422 INFO BLOC Initializing report manager.
2019/11/09 15:48:04.958555 DEBG DATA Reports database opened successfully.
2019/11/09 15:48:04.959699 DEBG DATA Opening table supply_stats with opts pack.Options{PackSizeLog2:12, JournalSizeLog2:10, CacheSize:2, FillLevel:100}
2019/11/09 15:48:04.959822 DEBG DATA pack: supply_stats table loaded journal with 0 entries
2019/11/09 15:48:04.959832 DEBG DATA pack: supply_stats table loaded tombstone with 0 entries
2019/11/09 15:48:04.959837 DEBG DATA pack: supply_stats table loading package headers from bucket
2019/11/09 15:48:04.959843 DEBG DATA pack: supply_stats table loaded 0 package headers
2019/11/09 15:48:04.960988 DEBG DATA Opening table activity with opts pack.Options{PackSizeLog2:12, JournalSizeLog2:10, CacheSize:2, FillLevel:100}
2019/11/09 15:48:04.961086 DEBG DATA pack: activity table loaded journal with 0 entries
2019/11/09 15:48:04.961094 DEBG DATA pack: activity table loaded tombstone with 0 entries
2019/11/09 15:48:04.961097 DEBG DATA pack: activity table loading package headers from bucket
2019/11/09 15:48:04.961100 DEBG DATA pack: activity table loaded 0 package headers
2019/11/09 15:48:04.962122 DEBG DATA Opening table age with opts pack.Options{PackSizeLog2:12, JournalSizeLog2:10, CacheSize:2, FillLevel:100}
2019/11/09 15:48:04.962161 DEBG DATA pack: age table loaded journal with 0 entries
2019/11/09 15:48:04.962169 DEBG DATA pack: age table loaded tombstone with 0 entries
2019/11/09 15:48:04.962172 DEBG DATA pack: age table loading package headers from bucket
2019/11/09 15:48:04.962175 DEBG DATA pack: age table loaded 0 package headers
2019/11/09 15:48:04.963251 DEBG DATA Opening table balance with opts pack.Options{PackSizeLog2:12, JournalSizeLog2:10, CacheSize:2, FillLevel:100}
2019/11/09 15:48:04.963504 DEBG DATA pack: balance table loaded journal with 0 entries
2019/11/09 15:48:04.963520 DEBG DATA pack: balance table loaded tombstone with 0 entries
2019/11/09 15:48:04.963523 DEBG DATA pack: balance table loading package headers from bucket
2019/11/09 15:48:04.963527 DEBG DATA pack: balance table loaded 0 package headers
2019/11/09 15:48:04.964546 DEBG DATA Opening table op_stats with opts pack.Options{PackSizeLog2:12, JournalSizeLog2:10, CacheSize:2, FillLevel:100}
2019/11/09 15:48:04.964599 DEBG DATA pack: op_stats table loaded journal with 0 entries
2019/11/09 15:48:04.964610 DEBG DATA pack: op_stats table loaded tombstone with 0 entries
2019/11/09 15:48:04.964614 DEBG DATA pack: op_stats table loading package headers from bucket
2019/11/09 15:48:04.964618 DEBG DATA pack: op_stats table loaded 0 package headers
2019/11/09 15:48:04.965645 INFO BLOC Connecting to RPC server.
2019/11/09 15:48:04.965709 DEBG JRPC GET /chains/main/blocks/head/header HTTP/1.1
Host: service-node-rpc:8732
Accept: application/json
Content-Type: application/json
User-Agent: Blockwatch-tzindex/v005-7-gf124983.f124983
2019/11/09 15:48:04.968029 DEBG JRPC HTTP/1.1 200 OK
Content-Length: 645
Content-Type: application/json
{"protocol":"PsBabyM1eUXZseaJdmXFApDSBqj8YBfwELoxZHHW77EMcAbbwAS","chain_id":"NetXdQprcVkpaWU","hash":"BKqDmsepj88WA5JYDe27bN4RG99vDNxvWhHSp4v4Ljr4yGJ8XcL","
level":686616,"proto":5,"predecessor":"BLgZRxvoMnLzTCNtkLcgg28bERPmPrpw2ofS51X5tke7vyZR8M5","timestamp":"2019-11-09T15:47:04Z","validation_pass":4,"operation
s_hash":"LLoZqvdXwfMToqVy3EwwgKtGTiTq9rsUecF24aMeK2QB3MryhuktY","fitness":["01","0000000000007a18"],"context":"CoWUKHsgdgJRf1HiaDkPKmfQDUuTEvsUWvA4PtDRfNXPsy
6e9LUu","priority":0,"proof_of_work_nonce":"12fe0a23baf80100","signature":"sigaWs7ry5UZxzBeqkBLXcRgCi64YKG1eXLrjfvfBEcoKvEuKYDAK8zjAEGWFbPRCFFkMWRPfop8c4hJDa
q5bmeotg1HvVDn"}
2019/11/09 15:48:04.968494 INFO BLOC Crawling Tezos Mainnet.
2019/11/09 15:48:04.968557 TRCE DATA query: block table non-indexed field 'h' for cond 0, fallback to table scan
2019/11/09 15:48:04.968568 TRCE DATA query: block table non-indexed field 'Z' for cond 1, fallback to table scan
2019/11/09 15:48:04.968587 INFO BLOC Stopping report manager.
2019/11/09 15:48:04.969786 DEBG DATA pack: closing supply_stats table with 0/0 records
2019/11/09 15:48:04.970834 DEBG DATA pack: closing activity table with 0/0 records
2019/11/09 15:48:04.971963 DEBG DATA pack: closing age table with 0/0 records
2019/11/09 15:48:04.973031 DEBG DATA pack: closing balance table with 0/0 records
2019/11/09 15:48:04.973996 DEBG DATA pack: closing op_stats table with 0/0 records
2019/11/09 15:48:04.976109 DEBG DATA Reports database closed successfully.
2019/11/09 15:48:04.976118 INFO BLOC Stopped report manager.
2019/11/09 15:48:04.976122 INFO BLOC Closing account index.
2019/11/09 15:48:04.976126 DEBG DATA pack: closing account table with 0/0 records
2019/11/09 15:48:04.976145 DEBG DATA pack: closing hash index account_hash_index/ with 0/0 records
2019/11/09 15:48:04.978926 DEBG DATA Account database closed successfully.
2019/11/09 15:48:04.978935 DEBG BLOC Storing account idx tip.
2019/11/09 15:48:04.980032 INFO BLOC Closing contract index.
2019/11/09 15:48:04.980040 DEBG DATA pack: closing contract table with 0/0 records
2019/11/09 15:48:04.980053 DEBG DATA pack: closing hash index contract_hash_index/ with 0/0 records
2019/11/09 15:48:04.982404 DEBG DATA Contract database closed successfully.
2019/11/09 15:48:04.982412 DEBG BLOC Storing contract idx tip.
2019/11/09 15:48:04.983395 INFO BLOC Closing block index.
2019/11/09 15:48:04.983403 DEBG DATA pack: closing block table with 0/0 records
2019/11/09 15:48:04.983417 DEBG DATA pack: closing hash index block_hash_index/ with 0/0 records
2019/11/09 15:48:04.986466 DEBG DATA Block database closed successfully.
2019/11/09 15:48:04.986475 DEBG BLOC Storing block idx tip.
2019/11/09 15:48:04.987491 INFO BLOC Closing op index.
2019/11/09 15:48:04.987500 DEBG DATA pack: closing op table with 0/0 records
2019/11/09 15:48:04.987513 DEBG DATA pack: closing hash index op_hash_index/ with 0/0 records
2019/11/09 15:48:04.990936 DEBG DATA Op database closed successfully.
2019/11/09 15:48:04.990944 DEBG BLOC Storing op idx tip.
2019/11/09 15:48:04.992021 INFO BLOC Closing flow index.
2019/11/09 15:48:04.992029 DEBG DATA pack: closing flow table with 0/0 records
2019/11/09 15:48:04.994190 DEBG DATA Flow database closed successfully.
2019/11/09 15:48:04.994198 DEBG BLOC Storing flow idx tip.
2019/11/09 15:48:04.995339 INFO BLOC Closing chain index.
2019/11/09 15:48:04.995348 DEBG DATA pack: closing chain table with 0/0 records
2019/11/09 15:48:04.997406 DEBG DATA Chain database closed successfully.
2019/11/09 15:48:04.997414 DEBG BLOC Storing chain idx tip.
2019/11/09 15:48:04.998493 INFO BLOC Closing supply index.
2019/11/09 15:48:04.998500 DEBG DATA pack: closing supply table with 0/0 records
2019/11/09 15:48:05.000729 DEBG DATA Supply database closed successfully.
2019/11/09 15:48:05.000738 DEBG BLOC Storing supply idx tip.
2019/11/09 15:48:05.001813 INFO BLOC Closing rights index.
2019/11/09 15:48:05.001821 DEBG DATA pack: closing rights table with 0/0 records
2019/11/09 15:48:05.004171 DEBG DATA Rights database closed successfully.
2019/11/09 15:48:05.004179 DEBG BLOC Storing rights idx tip.
2019/11/09 15:48:05.005318 INFO BLOC Closing snapshot index.
2019/11/09 15:48:05.005329 DEBG DATA pack: closing snapshot table with 0/0 records
2019/11/09 15:48:05.007532 DEBG DATA Snapshot database closed successfully.
2019/11/09 15:48:05.007541 DEBG BLOC Storing snapshot idx tip.
2019/11/09 15:48:05.008718 INFO BLOC Closing income index.
2019/11/09 15:48:05.008727 DEBG DATA pack: closing income table with 0/0 records
2019/11/09 15:48:05.010899 DEBG DATA Income database closed successfully.
2019/11/09 15:48:05.010907 DEBG BLOC Storing income idx tip.
2019/11/09 15:48:05.012018 INFO BLOC Closing gov index.
2019/11/09 15:48:05.012025 DEBG DATA pack: closing election table with 0/0 records
2019/11/09 15:48:05.013021 DEBG DATA pack: closing proposal table with 0/0 records
2019/11/09 15:48:05.014157 DEBG DATA pack: closing vote table with 0/0 records
2019/11/09 15:48:05.015145 DEBG DATA pack: closing ballot table with 0/0 records
2019/11/09 15:48:05.017406 DEBG DATA Gov database closed successfully.
2019/11/09 15:48:05.017415 DEBG BLOC Storing gov idx tip.
2019/11/09 15:48:05.019661 DEBG DATA Statedb database closed successfully.
2019/11/09 15:48:05.019671 Fatal: error initializing crawler: block not indexed
So, if the base is corrupted, how can it be shutdown without getting corrupted? And is there a mechanism to clean the database before starting tzindex ?
Thanks
latest docker: ReadContractCalls: cannot marshal response
curl http://<internal server with latest docker image>/explorer/contract/KT1TTtduU7q21hUpMPQQKERy1NYaCTr72Q4s/calls?since=BKokVkUFwZ3KTcPWFuRNJj9SBB7qnUMqNwNN1LWosWsBWW2zVZ2\&limit=1{
"errors": [
{
"code": 1003,
"status": 500,
"message": "internal server error",
"scope": "ReadContractCalls",
"detail": "cannot marshal response",
"reason": "json: error calling MarshalJSON for type *micheline.BigMapValue: micheline: type mismatch val_type=prim[Unit] type_code=list / type={\"annots\":[\"%disburse\"],\"args\":[{\"prim\":\"address\"}],\"prim\":\"list\"} -- value={\"prim\":\"Unit\"} / map=map[string]interface {}{}"
}
]
}To be more precise, 'latest' is image ID 5d6cb56838e6 ...
rpcurl: infer http when no mention
It does look like tzindex run --rpcurl "localhost:18731" is not equivalent to tzindex run --rpcurl "http://localhost:18731". IMO, it would be nice to add http automatically.
tzindex path change leads to broken data
hey,
situation:
we had data in /somedir-tzindex ... I've stopped tzindex (17.0.1), moved to /some-dir/tzindex, changed mount volumes in docker etc (it's /some-dir to /data) and restarted... it ended up with
2023/12/27 22:06:33.175428 INFO MAIN Using config file config.json
2023/12/27 22:06:33.175445 INFO MAIN Blockwatch Data Inc. TzIndex -- v17.0.1-alpine 2ca737b
2023/12/27 22:06:33.175446 INFO MAIN (c) Copyright 2018-2023 Blockwatch Data Inc.
2023/12/27 22:06:33.175447 INFO MAIN Go version go1.21.0
2023/12/27 22:06:33.175449 INFO MAIN Starting on 12 cores
2023/12/27 22:06:33.176233 INFO ETL Registered custom metadata extension.
2023/12/27 22:06:33.176239 INFO MAIN Using bolt database /data/tzindex
2023/12/27 22:06:33.176465 INFO ETL Initializing blockchain crawler in sync mode.
2023/12/27 22:06:33.176716 ERRO ETL op index with unexpected height 0/2379497
2023/12/27 22:06:33.176719 ERRO ETL event index with unexpected height 0/2379497
2023/12/27 22:06:33.176720 ERRO ETL flow index with unexpected height 0/2379497
2023/12/27 22:06:33.176721 ERRO ETL chain index with unexpected height 0/2379497
2023/12/27 22:06:33.176722 ERRO ETL storage index with unexpected height 0/2379497
2023/12/27 22:06:33.176723 ERRO ETL constant index with unexpected height 0/2379497
2023/12/27 22:06:33.176724 ERRO ETL ticket index with unexpected height 0/2379497
2023/12/27 22:06:33.176725 ERRO ETL block index with unexpected height 0/2379497
2023/12/27 22:06:33.176726 ERRO ETL bigmap index with unexpected height 0/2379497
2023/12/27 22:06:33.176726 ERRO ETL metadata index with unexpected height 0/2379497
2023/12/27 22:06:33.176728 ERRO ETL supply index with unexpected height 0/2379497
2023/12/27 22:06:33.176734 INFO ETL Closing account index.
2023/12/27 22:06:33.176819 INFO ETL Closing balance index.
2023/12/27 22:06:33.176885 INFO ETL Closing contract index.
2023/12/27 22:06:33.176943 INFO ETL Closing storage index.
2023/12/27 22:06:33.176992 INFO ETL Closing constant index.
2023/12/27 22:06:33.177044 INFO ETL Closing block index.
2023/12/27 22:06:33.177094 INFO ETL Closing op index.
2023/12/27 22:06:33.177139 INFO ETL Closing event index.
2023/12/27 22:06:33.177188 INFO ETL Closing flow index.
2023/12/27 22:06:33.177234 INFO ETL Closing chain index.
2023/12/27 22:06:33.177362 INFO ETL Closing supply index.
2023/12/27 22:06:33.177416 INFO ETL Closing bigmap index.
2023/12/27 22:06:33.177462 INFO ETL Closing metadata index.
2023/12/27 22:06:33.177509 INFO ETL Closing ticket index.
Error: error initializing crawler: indexer init: Missing database files! Looks like you deleted a database file.
running params:
"Env": [
"TZ_RPC_DIAL_TIMEOUT=60s",
"TZ_RPC_RESPONSE_TIMEOUT=240s",
"TZ_RPC_CONTINUE_TIMEOUT=240s",
"TZ_SERVER_READ_TIMEOUT=60s",
"TZ_SERVER_HEADER_TIMEOUT=120s",
"TZ_DB_ACCOUNT_CACHE_SIZE=256",
"TZ_DB_BALANCE_CACHE_SIZE=256",
"TZ_DB_BLOCK_CACHE_SIZE=16",
"TZ_DB_CONTRACT_CACHE_SIZE=8",
"TZ_DB_STORAGE_CACHE_SIZE=32",
"TZ_DB_OP_CACHE_SIZE=256",
"TZ_DB_FLOW_CACHE_SIZE=256",
"TZ_DB_BIGMAP_VALUES_CACHE_SIZE=2048",
"TZ_DB_BIGMAP_UPDATES_CACHE_SIZE=256",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"BUILD_TARGET=tzindex"
],
"Cmd": [
"-rpc.url=localhost:8732",
"-db.path=/data/tzindex",
"-light",
"-notls",
"-server.addr=0.0.0.0",
"-server.port=8080"
],
if I remove all the .db files in that folder and restarted tzindex, it works - and creates all the same files ("just" empty)..
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x76cc6f]
latest tzindex version
INFO MAIN Blockwatch Data Inc. TzIndex -- v15.0.0 49e3e0e
used params
tzindex -db.path=/data/tzindex -db.engine=bolt -rpc.url=http://somehostonlima -light -notls
1st issue - complaining about ticket.db
2023/02/20 08:31:31.929478 WARN MAIN Missing config file, using default values.
2023/02/20 08:31:31.929552 INFO MAIN Blockwatch Data Inc. TzIndex -- v15.0.0 49e3e0e
2023/02/20 08:31:31.929559 INFO MAIN (c) Copyright 2018-2023 Blockwatch Data Inc.
2023/02/20 08:31:31.929562 INFO MAIN Go version go1.20
2023/02/20 08:31:31.929568 INFO MAIN Starting on 64 cores
2023/02/20 08:31:31.929978 INFO MAIN Using bolt database /data/tzindex
2023/02/20 08:31:31.932042 INFO BLOC Initializing blockchain crawler in sync mode.
2023/02/20 08:31:31.932712 INFO BLOC Creating ticket index.
2023/02/20 08:31:31.933478 INFO BLOC Closing account index.
2023/02/20 08:31:31.934038 INFO BLOC Closing balance index.
2023/02/20 08:31:31.934485 INFO BLOC Closing contract index.
2023/02/20 08:31:31.935025 INFO BLOC Closing storage index.
2023/02/20 08:31:31.935531 INFO BLOC Closing constant index.
2023/02/20 08:31:31.935989 INFO BLOC Closing block index.
2023/02/20 08:31:31.936429 INFO BLOC Closing op index.
2023/02/20 08:31:31.936872 INFO BLOC Closing event index.
2023/02/20 08:31:31.937300 INFO BLOC Closing flow index.
2023/02/20 08:31:31.937733 INFO BLOC Closing chain index.
2023/02/20 08:31:31.938155 INFO BLOC Closing supply index.
2023/02/20 08:31:31.938580 INFO BLOC Closing bigmap index.
2023/02/20 08:31:31.939020 INFO BLOC Closing metadata index.
2023/02/20 08:31:31.939436 INFO BLOC Closing ticket index.
Error: error initializing crawler: creating ticket database: pack: creating database: database file "/data/tzindex/ticket.db" exists
so I had to rename ticket.db (it was from the previous version tzindex, with v14 support only), then it started and continued like that
2023/02/20 08:31:49.445677 WARN MAIN Missing config file, using default values.
2023/02/20 08:31:49.445744 INFO MAIN Blockwatch Data Inc. TzIndex -- v15.0.0 49e3e0e
2023/02/20 08:31:49.445749 INFO MAIN (c) Copyright 2018-2023 Blockwatch Data Inc.
2023/02/20 08:31:49.445756 INFO MAIN Go version go1.20
2023/02/20 08:31:49.445763 INFO MAIN Starting on 64 cores
2023/02/20 08:31:49.447148 INFO MAIN Using bolt database /data/tzindex
2023/02/20 08:31:49.448619 INFO BLOC Initializing blockchain crawler in sync mode.
2023/02/20 08:31:49.449231 INFO BLOC Creating ticket index.
2023/02/20 08:31:49.469937 INFO BLOC Initializing account index.
2023/02/20 08:31:49.631092 INFO BLOC Initializing balance index.
2023/02/20 08:31:50.548491 INFO BLOC Initializing contract index.
2023/02/20 08:31:50.837227 INFO BLOC Initializing storage index.
2023/02/20 08:31:51.784875 INFO BLOC Initializing constant index.
2023/02/20 08:31:51.786180 INFO BLOC Initializing block index.
2023/02/20 08:31:51.888427 ERRO DATA pack: header decode for table block pack 0000005a: pack: reading bloom filter: short buffer
2023/02/20 08:31:51.888449 WARN DATA pack: block table has corrupt or missing statistics! Re-scanning table. This may take some time...
2023/02/20 08:31:53.570051 INFO BLOC Initializing op index.
2023/02/20 08:31:55.374445 INFO BLOC Initializing event index.
2023/02/20 08:31:55.375667 INFO BLOC Initializing flow index.
2023/02/20 08:31:57.002874 INFO BLOC Initializing chain index.
2023/02/20 08:31:57.021798 INFO BLOC Initializing supply index.
2023/02/20 08:31:57.038513 INFO BLOC Initializing bigmap index.
2023/02/20 08:31:59.953758 INFO BLOC Initializing metadata index.
2023/02/20 08:31:59.955201 INFO BLOC Initializing ticket index.
2023/02/20 08:31:59.956317 INFO BLOC Connecting to RPC server.
2023/02/20 08:31:59.959242 INFO BLOC Closing account index.
2023/02/20 08:32:00.086516 INFO BLOC Closing balance index.
2023/02/20 08:32:00.105579 INFO BLOC Closing contract index.
2023/02/20 08:32:00.976140 INFO BLOC Closing storage index.
2023/02/20 08:32:01.005504 INFO BLOC Closing constant index.
2023/02/20 08:32:01.008819 INFO BLOC Closing block index.
2023/02/20 08:32:01.134209 INFO BLOC Closing op index.
2023/02/20 08:32:01.235399 INFO BLOC Closing event index.
2023/02/20 08:32:01.237799 INFO BLOC Closing flow index.
2023/02/20 08:32:01.270379 INFO BLOC Closing chain index.
2023/02/20 08:32:01.288975 INFO BLOC Closing supply index.
2023/02/20 08:32:01.315941 INFO BLOC Closing bigmap index.
2023/02/20 08:32:01.539708 INFO BLOC Closing metadata index.
2023/02/20 08:32:01.541902 INFO BLOC Closing ticket index.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x76cc6f]
goroutine 1 [running]:
blockwatch.cc/tzgo/tezos.(*Params).ForNetwork(0x0, {{0xf0?, {0xc00037d4d0?, 0xc000739600?, 0x0?}}})
/go/src/tzindex/vendor/blockwatch.cc/tzgo/tezos/protocols.go:100 +0x4f
blockwatch.cc/tzindex/etl.(*Crawler).Init(0xc000620360, {0xdd2df0?, 0xc00013a000?}, {0xb2c71d, 0x4})
/go/src/tzindex/etl/crawler.go:436 +0xd86
main.runServer()
/go/src/tzindex/cmd/tzindex/server.go:108 +0xdcd
main.run()
/go/src/tzindex/cmd/tzindex/main.go:32 +0x2b5
main.main()
/go/src/tzindex/cmd/tzindex/main.go:17 +0x1d
protocol support > 14
Hi,
is there any planned protocol upgrade, like Lima or Mumbai? We have one tezos (octez) node with Limanet and crawler ended up with store.metadata_not_found
log file ended up like this
2023/01/09 11:58:43.252880 INFO BLOC Processed 132 blocks in 10.09s (34599 transactions, height 2793352, 2022-10-13 23:46:44 +0000 UTC, q=0, t=37.833849ms, s=syncing
)
2023/01/09 11:58:52.197086 WARN BLOC crawler: RPC: tezos: kind = "permanent", id = "store.metadata_not_found"
2023/01/09 11:58:58.375710 INFO BLOC Reorg at height 2793471: parent BMDMopLtrwUTSG3PCBKryKRoQgQGKZJi27KRW1pjyqgvq8fJWwZ is not on our main chain, have BLqfmdk3he2DoiaoR8U8CPubefpFq4EkaP9VmhK8P9NjVGKJaQ1
2023/01/09 11:58:58.375734 INFO BLOC REORGANIZE: searching fork point side=BLqfmdk3he2DoiaoR8U8CPubefpFq4EkaP9VmhK8P9NjVGKJaQ1 main=BKjV8TaCuygYuh5jiSSG8bpX1nhsNHA1xwhdoMgG9pE6zakgKgc
2023/01/09 11:58:58.418870 ERRO BLOC Reorg failed: cannot find fork point in last 100 blocks
2023/01/09 11:58:58.420990 INFO BLOC Stopping blockchain sync due to too many errors at 2793472.
2023/01/09 11:58:58.441063 WARN BLOC crawler: RPC: tezos: kind = "permanent", id = "store.metadata_not_found"
2023/01/09 11:58:59.512891 INFO BLOC Stopping blockchain ingest on cancel.
Thanks
Multiple errors parsing operation on v6.0.2
On dockerhub v6.0.2:
curl http://OUR-INTERNAL-SERVER/explorer/op/ooHe73NX3PS9vu3PsYeGRaUd5RbSGy3GmdqGnPFiYaad9VJMqPm{
"errors": [
{
"code": 1003,
"status": 500,
"message": "internal server error",
"scope": "ReadOp",
"detail": "cannot marshal response",
"reason": "json: error calling MarshalJSON for type *micheline.BigMapValue: micheline: type mismatch val_type=prim[Unit] type_code=list / type={\"annots\":[\"%disburse\"],\"args\":[{\"prim\":\"address\"}],\"prim\":\"list\"} -- value={\"prim\":\"Unit\"} / map=map[string]interface {}{}"
}
]
}logs:
2020/04/30 21:37:16 ERRO MICH Rendering prim type bytes as address: invalid binary address length 34
2020/04/30 21:37:16 ERRO MICH RENDER ERROR
typ={"annots":["%disburse"],"args":[{"prim":"address"}],"prim":"list"}
val={"prim":"Unit"}
err=micheline: type mismatch val_type=prim[Unit] type_code=list / type={"annots":["%disburse"],"args":[{"prim":"address"}],"prim":"list"} -- value={"prim":"Unit"} / map=map[string]interface {}{}
2020/04/30 21:37:16 ERRO SRVR BW-7c1cdf59267ef57392f6ce856f879728f5f222a6 Error sending response: json: error calling MarshalJSON for type *micheline.BigMapValue: micheline: type mismatch val_type=prim[Unit] type_code=list / type={"annots":["%disburse"],"args":[{"prim":"address"}],"prim":"list"} -- value={"prim":"Unit"} / map=map[string]interface {}{} in struct server.ExplorerOpList{(*server.ExplorerOp)(0xc0001f0780), (*server.ExplorerOp)(0xc0001f0960)}
It's possible this is a partial duplicate of #25 but I'm pretty sure at least the first line of logs ("invalid binary address") is not. That part seems to be caused by the use of entrypoint-suffixed addresses which are a valid but rarely-used part of the protocol.
(It's also quite possible the rest of it is not a duplicate of #25 either ... it looks similar but when I tried to replicate #25 using an /op/ call it didn't seem to work.)
ERRO BLOC Unrecoverable error: runtime error: index out of range [0] with length 0
Hi,
I have been trying to sync your Blockchain indexer to my Private Tezos chain. However, I am facing a certain number of issues (tried a lot of solutions, but can't seem to make it work).
I am currently trying to tweak/test some new protocols. Using your explorer would be very helpful
to see the behavior of these new protocols on smart contracts etc...
Could you please tell me the steps/modifications required to sync your explorer with any personal protocol ?
My first deployment :
A private chain built on the sandbox-parameters.json specifications with these modified parameters :
time_between_blocks : ["10", "30"]"preserved_cycles": 2"blocks_per_cycle": 64
*"proof_of_work_threshold": "70368744177663"-> because the default-1brings a conversion error when the explorer tries to synchronize
To be more specific, here are my protocol parameters
[nodezero@node-zero tezos]$ cat sandbox-parameters.json
{ "bootstrap_accounts":
[ [ "edpkuBknW28nW72KG6RoHtYW7p12T6GKc7nAbwYX5m8Wd9sDVC9yav",
"4000000000000" ],
[ "edpktzNbDAUjUk697W7gYg2CRuBQjyPxbEg8dLccYYwKSKvkPvjtV9",
"4000000000000" ],
[ "edpkuTXkJDGcFd5nh6VvMz8phXxU3Bi7h6hqgywNFi1vZTfQNnS1RV",
"4000000000000" ],
[ "edpkuFrRoDSEbJYgxRtLx2ps82UdaYc1WwfS9sE11yhauZt5DgCHbU",
"4000000000000" ],
[ "edpkv8EUUH68jmo3f7Um5PezmfGrRF24gnfLpH3sVNwJnV5bVCxL2n",
"4000000000000" ] ], "preserved_cycles": 2, "blocks_per_cycle": 64,
"blocks_per_commitment": 4, "blocks_per_roll_snapshot": 4,
"blocks_per_voting_period": 64, "time_between_blocks": [ "10", "30" ],
"endorsers_per_block": 32, "hard_gas_limit_per_operation": "800000",
"hard_gas_limit_per_block": "8000000", "proof_of_work_threshold": "70368744177663",
"tokens_per_roll": "8000000000", "michelson_maximum_type_size": 1000,
"seed_nonce_revelation_tip": "125000", "origination_size": 257,
"block_security_deposit": "512000000",
"endorsement_security_deposit": "64000000", "block_reward": "16000000",
"endorsement_reward": "2000000", "cost_per_byte": "1000",
"hard_storage_limit_per_operation": "60000",
"test_chain_duration": "1966080", "quorum_min": 2000, "quorum_max": 7000,
"min_proposal_quorum": 500, "initial_endorsers": 1,
"delay_per_missing_endorsement": "1" }
I have changed the genesis_key in order to activate the new protocol, then started baking with one of the bootstrap_account
This first trial resulted to short sync between your explorer and my node (after the genesis block) -> up to level 3/4
In order to be on the same page, and avoid wasting your time, here are the steps made :
Steps used to compile and run Tzindex
go build tzindex.go./tzindex --profile-block tmp.txt run -c docker/config.json --host 192.168.193.124 --vvv
Command to run Tezos node
./tezos-node run --no-bootstrap-peers --rpc-addr *:8732 --expected-pow 1 -v -v --peer 192.168.193.123:9732 --connection=2
With these 2 commands, my node is synced with his peers + your indexer can communicate with it.
I have redirected the standard output to a file named sandbox_1_output.txt, to facilitate trace debug + enabled the --profile-block option, this one named tmpt.txt
The error I get is ERRO BLOC Unrecoverable error: runtime error: index out of range [0] with length 0, but I can't seem to fix it. Do you have any idea ?
The indexer only syncs until the third block, after, it stops because of the error (own opinion). The explorer API gives me :
http://127.0.0.1:8000/explorer/status
{
"status": "failed",
"blocks": 4,
"indexed": 3,
"progress": 0.75
}
My second deployment
Exactly the same, but this time, I try to put the parameters of the current babylonet protocol :
{ "bootstrap_accounts":
[ [ "edpkuBknW28nW72KG6RoHtYW7p12T6GKc7nAbwYX5m8Wd9sDVC9yav",
"4000000000000" ],
[ "edpktzNbDAUjUk697W7gYg2CRuBQjyPxbEg8dLccYYwKSKvkPvjtV9",
"4000000000000" ],
[ "edpkuTXkJDGcFd5nh6VvMz8phXxU3Bi7h6hqgywNFi1vZTfQNnS1RV",
"4000000000000" ],
[ "edpkuFrRoDSEbJYgxRtLx2ps82UdaYc1WwfS9sE11yhauZt5DgCHbU",
"4000000000000" ],
[ "edpkv8EUUH68jmo3f7Um5PezmfGrRF24gnfLpH3sVNwJnV5bVCxL2n",
"4000000000000" ] ], "preserved_cycles": 5, "blocks_per_cycle": 4096,
"blocks_per_commitment": 32, "blocks_per_roll_snapshot": 256,
"blocks_per_voting_period": 32768, "time_between_blocks": [ "60", "40" ],
"endorsers_per_block": 32, "hard_gas_limit_per_operation": "800000",
"hard_gas_limit_per_block": "8000000", "proof_of_work_threshold": "70368744177663",
"tokens_per_roll": "8000000000", "michelson_maximum_type_size": 1000,
"seed_nonce_revelation_tip": "125000", "origination_size": 257,
"block_security_deposit": "512000000",
"endorsement_security_deposit": "64000000", "block_reward": "16000000",
"endorsement_reward": "2000000", "cost_per_byte": "1000",
"hard_storage_limit_per_operation": "60000",
"test_chain_duration": "1966080", "quorum_min": 2000, "quorum_max": 7000,
"min_proposal_quorum": 500, "initial_endorsers": 1,
"delay_per_missing_endorsement": "8" }
This leads to the same error, at the same height
The only difference resides, if I recall properly, in less errors : the indexer doesn't have this kind of errors (but I'm not sure) :
2019/12/18 13:57:36.754305 TRCE DATA query: rights table non-indexed field 't' for cond 1, fallback to table scan
Do you have any idea on how to solve this issue ?
Thanks,
Guillaume
Incorrect number of missed blocks
https://api.tzstats.com/tables/income?address=tz1W5VkdB5s7ENMESVBtwyt9kyvLqPcUczRT&cycle=145&columns=n_blocks_lost
This method returns [[ 0 ]] that means that the baker didn't miss any block.
Now let's see his baking rights via RPC
https://mainnet.tezrpc.me/chains/main/blocks/589824/helpers/baking_rights?cycle=145&delegate=tz1W5VkdB5s7ENMESVBtwyt9kyvLqPcUczRT&max_priority=2
[
{
"level": 594710,
"delegate": "tz1W5VkdB5s7ENMESVBtwyt9kyvLqPcUczRT",
"priority": 0,
"estimated_time": "2019-09-05T03:19:51Z"
},
{
"level": 595767,
"delegate": "tz1W5VkdB5s7ENMESVBtwyt9kyvLqPcUczRT",
"priority": 0,
"estimated_time": "2019-09-05T20:56:51Z"
},
{
"level": 596792,
"delegate": "tz1W5VkdB5s7ENMESVBtwyt9kyvLqPcUczRT",
"priority": 1,
"estimated_time": "2019-09-06T14:03:06Z"
}
]Then let's get the block at the level 596792
https://mainnet.tezrpc.me/chains/main/blocks/596792/header
{
"hash": "BKpuDaitN2jc9HrhUQzQ43GRbm2ojvbXrcRkwuDj3nNfsSeFfW4",
"level": 596792,
"timestamp": "2019-09-06T20:41:36Z",
"priority": 2,
...
}Its priority is 2, so the baker actually missed the block at the level 596792.
Sync have stopped on block 984006
I have two tzindex nodes on different servers and they are both have stopped on 984006 block. tezos-node works fine. Could you help how to fix this?
tzindex release 6.0.3
tezos-node 7.1
tzindex logs
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.778369 DEBG JRPC GET /chains/main/blocks/984009 HTTP/1.1
Jun 06 09:38:36 xtz-01 tzindex[6300]: Host: 127.0.0.1:8732
Jun 06 09:38:36 xtz-01 tzindex[6300]: Accept: application/json
Jun 06 09:38:36 xtz-01 tzindex[6300]: Content-Type: application/json
Jun 06 09:38:36 xtz-01 tzindex[6300]: User-Agent: Blockwatch-tzindex/dev.none
Jun 06 09:38:36 xtz-01 tzindex[6300]: [1B blob data]
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.895265 DEBG BLOC Loaded 4154 total delegates
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.895709 ERRO SRVR BW-bab46c4e920990403958e83faf1840dffe39b8dc 404 (1305) GET /explorer/account/tz1esvRfFFB6jDbmApp5t9Mb6rP73XvipK1Z/op?offset=0&limit=0 HTTP/1.0 - ReadAccountOps failed (no such account): account not indexed
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.896386 INFO BLOC Stopping blockchain sync due to too many errors at 984006.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.896395 DEBG BLOC Flushing account.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.907551 DEBG BLOC Storing account idx tip.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.910161 DEBG BLOC Flushing contract.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.912565 DEBG BLOC Storing contract idx tip.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.915035 DEBG BLOC Flushing block.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.920783 DEBG BLOC Storing block idx tip.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.923311 DEBG BLOC Flushing op.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.937975 DEBG BLOC Storing op idx tip.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.940282 DEBG BLOC Flushing flow.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.943784 DEBG BLOC Storing flow idx tip.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.946027 DEBG BLOC Flushing chain.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.948318 DEBG BLOC Storing chain idx tip.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.950939 DEBG BLOC Flushing supply.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.953617 DEBG BLOC Storing supply idx tip.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.956050 DEBG BLOC Flushing rights.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.962377 DEBG BLOC Storing rights idx tip.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.964628 DEBG BLOC Flushing snapshot.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.974686 DEBG BLOC Storing snapshot idx tip.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.977060 DEBG BLOC Flushing income.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.979568 DEBG BLOC Storing income idx tip.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.981934 DEBG BLOC Flushing election.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.984376 DEBG BLOC Flushing proposal.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.986564 DEBG BLOC Flushing vote.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.989124 DEBG BLOC Flushing ballot.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.991410 DEBG BLOC Storing gov idx tip.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.993945 DEBG BLOC Flushing bigmap.
Jun 06 09:38:36 xtz-01 tzindex[6300]: 2020/06/06 09:38:36.996545 DEBG BLOC Storing bigmap idx tip.
Jun 06 09:38:37 xtz-01 tzindex[6300]: 2020/06/06 09:38:37.001810 ERRO BLOC Exiting monitor loop on cancelled context
Jun 06 09:38:37 xtz-01 tzindex[6300]: 2020/06/06 09:38:37.001847 DEBG BLOC Block 984009 download failed: Get http://127.0.0.1:8732/chains/main/blocks/984009: context canceled
Jun 06 09:38:37 xtz-01 tzindex[6300]: 2020/06/06 09:38:37.001854 INFO BLOC Context cancelled. Stopping blockchain ingest.
Jun 06 09:38:38 xtz-01 tzindex[6300]: 2020/06/06 09:38:38.164174 ERRO SRVR BW-dc1ebf5483d37d226f4590421ebaed548af2f5f5 404 (1305) GET /explorer/account/tz1RRTiLzuWgnjSgfKSUU28PDzAPaPfsmPTx/op?offset=0&limit=0 HTTP/1.0 - ReadAccountOps failed (no such account): account not indexed
Jun 06 09:38:39 xtz-01 tzindex[6300]: 2020/06/06 09:38:39.080296 ERRO SRVR BW-f87fa7a1745f5e1e8cdec46c13f70d135c50392b 404 (1305) GET /explorer/account/tz1ivcia9HyAfoLFMN9ynKoMLCDxckbn8Qz9/op?offset=0&limit=0 HTTP/1.0 - ReadAccountOps failed (no such account): account not indexed
Jun 06 09:38:43 xtz-01 tzindex[6300]: 2020/06/06 09:38:43.543129 ERRO SRVR BW-d62a0bdb37b401e63a710ae05b0534657aa37fb6 404 (1305) GET /explorer/account/tz1UQSKz5ffeqjWZVR7Dn6GpTCytFxHRvVmx/op?offset=0&limit=0 HTTP/1.0 - ReadAccountOps failed (no such account): account not indexed
Jun 06 09:38:43 xtz-01 tzindex[6300]: 2020/06/06 09:38:43.594102 ERRO SRVR BW-c1db059e934a57ef7079e2dc60dbfbc2c22dbc69 404 (1305) GET /explorer/account/tz1XaMygPHfcvo4AAibrC41W7NZZBkF1dpku/op?offset=0&limit=0 HTTP/1.0 - ReadAccountOps failed (no such account): account not indexed
Jun 06 09:38:43 xtz-01 tzindex[6300]: 2020/06/06 09:38:43.659582 ERRO SRVR BW-299a29333570f55f683f2bdb873e2d90e8767d74 404 (1305) GET /explorer/account/tz1UqJX3NoGDZ8NDRZNTzxT2tKCvnfcuDPib/op?offset=0&limit=0 HTTP/1.0 - ReadAccountOps failed (no such account): account not indexed
Jun 06 09:38:43 xtz-01 tzindex[6300]: 2020/06/06 09:38:43.947696 ERRO SRVR BW-7e9f7d923999dce01138e4a676f5f019d53102e2 404 (1305) GET /explorer/account/tz1UYbgo2XmJ3S3BvjFH9KMNVQvwQFVvtD34/op?offset=0&limit=0 HTTP/1.0 - ReadAccountOps failed (no such account): account not indexed
Jun 06 09:38:48 xtz-01 tzindex[6300]: 2020/06/06 09:38:48.288991 DEBG BLOC Ranks built in 693.878679ms
Jun 06 09:38:50 xtz-01 tzindex[6300]: 2020/06/06 09:38:50.709765 ERRO SRVR BW-e2c15aa9bfd8cd8270665c445466b5d0017f2001 404 (1305) GET /explorer/account/tz1Zxu75fPS5vE9QdcCSBxYPUabJSCHdfmRJ/op?offset=0&limit=0 HTTP/1.0 - ReadAccountOps failed (no such account): account not indexed
Tezos v9-release with florence/granada not working
2021/06/02 10:20:44.074918 INFO MAIN Using configuration file config.json
1s
22021/06/02 10:20:44.075013 INFO MAIN Blockwatch tzindex v8.0.1 -- e6e4128
1s
32021/06/02 10:20:44.075031 INFO MAIN (c) Copyright 2018-2021 -- Blockwatch Data Inc.
1s
42021/06/02 10:20:44.075045 INFO MAIN Starting Blockwatch-tzindex/v8.0.1.e6e4128 on 6 cores
1s
52021/06/02 10:20:44.075069 INFO MAIN Go version go1.16
1s
62021/06/02 10:20:44.080871 INFO MAIN Profiling mutex/blocking at 1.00% sample rate.
1s
72021/06/02 10:20:44.081369 INFO MAIN Using bolt database /data/index
1s
82021/06/02 10:20:44.089579 INFO BLOC Initializing blockchain crawler in sync mode.
1s
92021/06/02 10:20:44.089617 INFO BLOC Creating blockchain storage.
1s
102021/06/02 10:20:44.091547 INFO BLOC Creating account index.
1s
112021/06/02 10:20:44.115888 INFO BLOC Creating contract index.
1s
122021/06/02 10:20:44.149544 INFO BLOC Creating block index.
1s
132021/06/02 10:20:44.197256 INFO BLOC Creating op index.
1s
142021/06/02 10:20:44.224018 INFO BLOC Creating flow index.
1s
152021/06/02 10:20:44.241893 INFO BLOC Creating chain index.
1s
162021/06/02 10:20:44.261389 INFO BLOC Creating supply index.
1s
172021/06/02 10:20:44.278890 INFO BLOC Creating rights index.
1s
182021/06/02 10:20:44.284084 INFO BLOC Creating snapshot index.
1s
192021/06/02 10:20:44.307680 INFO BLOC Creating income index.
1s
202021/06/02 10:20:44.330585 INFO BLOC Creating gov index.
1s
212021/06/02 10:20:44.363308 INFO BLOC Creating bigmap index.
1s
222021/06/02 10:20:44.386183 INFO BLOC Initializing account index.
1s
232021/06/02 10:20:44.387133 INFO BLOC Initializing contract index.
1s
242021/06/02 10:20:44.387791 INFO BLOC Initializing block index.
1s
252021/06/02 10:20:44.388579 INFO BLOC Initializing op index.
1s
262021/06/02 10:20:44.389526 INFO BLOC Initializing flow index.
1s
272021/06/02 10:20:44.390503 INFO BLOC Initializing chain index.
1s
282021/06/02 10:20:44.391599 INFO BLOC Initializing supply index.
1s
292021/06/02 10:20:44.392262 INFO BLOC Initializing rights index.
1s
302021/06/02 10:20:44.392866 INFO BLOC Initializing snapshot index.
1s
312021/06/02 10:20:44.393512 INFO BLOC Initializing income index.
1s
322021/06/02 10:20:44.394197 INFO BLOC Initializing gov index.
1s
332021/06/02 10:20:44.394929 INFO BLOC Initializing bigmap index.
1s
342021/06/02 10:20:44.395515 INFO BLOC Connecting to RPC server.
1s
352021/06/02 10:20:44.802565 INFO BLOC Fetching genesis block.
1s
362021/06/02 10:20:44.804623 INFO BLOC Crawling Tezos Sandbox.
2s
372021/06/02 10:20:44.806644 INFO BLOC Starting blockchain crawler.
2s
382021/06/02 10:20:44.806857 WARN SRVR HTTP Server reachable on all interfaces (0.0.0.0)
2s
392021/06/02 10:20:44.806948 INFO BLOC Starting blockchain sync from height 1.
2s
402021/06/02 10:20:44.806997 INFO BLOC Starting blockchain ingest.
2s
412021/06/02 10:20:44.807014 INFO BLOC Starting blockchain monitor.
2s
422021/06/02 10:20:44.809122 INFO SRVR Starting HTTP server at 0.0.0.0:8002
2s
432021/06/02 10:20:45.604475 INFO BLOC Fetching bootstrap rights for 3(+1) preserved cycles
2s
442021/06/02 10:21:16.825678 INFO BLOC Building genesis dataset.
34s
452021/06/02 10:21:16.954633 WARN BLOC RPC connection error: rpc: generic operation: invalid operation type 'endorsement_with_slot'
34s
462021/06/02 10:21:18.978813 INFO BLOC Processed 1 block in 34.88s (16 transactions, height 1, 2021-03-04 20:00:03 +0000 UTC, q=1, t=2.153116199s, s=connecting)
36s
472021/06/02 10:21:18.980481 INFO BLOC New protocol PsFLorenaUUuikDWvMDr6fGBRG8kt3e3D3fHoXK1j1BFRxeSH4i detected at 2
36s
482021/06/02 10:21:22.935176 INFO BLOC Fetching bootstrap rights for 3(+1) preserved cycles
40s
492021/06/02 10:21:44.793124 INFO BLOC Reorg at height 1: parent BMFCHw1mv3A71KpTuGD3MoFnkHk9wvTYjUzuR9QqiUumKGFG6pM is not on our main chain, have BM2EMxsQsjCqud7g3VKE4qK8hXwpjnqjUWUHqNtDtwwVcD5cm1c
61s
502021/06/02 10:21:44.794208 INFO BLOC REORGANIZE: searching fork point side=BM2EMxsQsjCqud7g3VKE4qK8hXwpjnqjUWUHqNtDtwwVcD5cm1c main=BMAP2ZWJAhqCidHvTAxZ9d3vaL9winSaPQ7tnnbjEoAB59uzdAF
61s
512021/06/02 10:21:44.880239 ERRO BLOC Reorg failed: cannot find fork point in last 100 blocks
62s
522021/06/02 10:21:44.881897 WARN BLOC RPC connection error: rpc: generic operation: invalid operation type 'endorsement_with_slot'
62s
532021/06/02 10:21:44.934753 INFO BLOC Reorg at height 1: parent BMFCHw1mv3A71KpTuGD3MoFnkHk9wvTYjUzuR9QqiUumKGFG6pM is not on our main chain, have BM2EMxsQsjCqud7g3VKE4qK8hXwpjnqjUWUHqNtDtwwVcD5cm1c
62s
542021/06/02 10:21:44.934804 INFO BLOC REORGANIZE: searching fork point side=BM2EMxsQsjCqud7g3VKE4qK8hXwpjnqjUWUHqNtDtwwVcD5cm1c main=BMAP2ZWJAhqCidHvTAxZ9d3vaL9winSaPQ7tnnbjEoAB59uzdAF
62s
552021/06/02 10:21:44.998322 ERRO BLOC Reorg failed: cannot find fork point in last 100 blocks
62s
562021/06/02 10:21:45.057526 INFO BLOC Stopping blockchain sync due to too many errors at 2.
62s
572021/06/02 10:21:45.141543 ERRO BLOC Exiting monitor loop on cancelled context
62s
582021/06/02 10:21:49.882512 INFO BLOC Context cancelled. Stopping blockchain ingest.
Florencenet Update
Please update open source version for florence, ty ty.
Getting a crawl error
Fatal: error initializing crawler: unknown protocol deployment 2
CLI params not accepted when config.json is in path
Hi,
using latest tzindex
INFO MAIN Blockwatch Data Inc. TzIndex -- v15.0.0 49e3e0e
When I specify cmd params like
tzindex -db.path=/data/index -rpc.url=https://...
it ended up like
2023/02/20 08:05:25.320981 INFO BLOC Registered custom metadata extension.
2023/02/20 08:05:25.321011 INFO MAIN Using bolt database ./db/xtz
so db.path (and any other parameter) is not honored. When I remove config.json, it works. I expected that CLI params overwrite config loaded from config.json (or defaults).
Increase response time when getting account history
Hi! Can we increase response time when getting account history? In general time response from 6 to 50sec.
root@xtz-02:~# time curl -s 'http://127.0.0.1:8000/explorer/account/tz1ftXcwbkeef2GnwqecUw7M26EuGXWY1mHt/op?offset=0&limit=100'
{...}
real 0m17.786s
user 0m0.006s
sys 0m0.000s
root@xtz-02:~# time curl -s 'http://127.0.0.1:8000/explorer/account/tz1YhtbmHd7cWYohQRsdtfcriiYTaETbGKLS/op?offset=0&limit=100'
{...}
real 0m51.207s
user 0m0.000s
sys 0m0.007s
Some times response code 504.
Tzindex run on machine with:
- 8 CPU
- 64GB RAM
- NVME SSD 1TB
tzindex config.json. Which database parameters we can change for Increase performance?
{
"server": {
"addr": "0.0.0.0",
"port": 8000,
"scheme": "http",
"host": "127.0.0.1",
"workers": 50,
"read_timeout": "5s",
"header_timeout": "10s",
"write_timeout": "1000s",
"keepalive": "90s",
"shutdown_timeout": "15s",
"max_list_count": 50000,
"default_explore_count": 20,
"max_explore_count": 100,
"cors_enable": false,
"cors_origin": "*",
"cors_allow_headers": "Authorization, Accept, Content-Type, X-Api-Key, X-Requested-With",
"cors_expose_headers": "Date, X-Runtime, X-Request-Id, X-Api-Version",
"cors_methods": "GET, OPTIONS",
"cors_maxage": "86400",
"cors_credentials": "true",
"cache_enable": false,
"cache_control": "public"
},
"crawler": {
"queue": 1000,
"snapshot_path": "./db/xtz/snapshots",
"snapshot_blocks": [],
"snapshot_interval": 0
},
"database": {
"path": "./db/xtz",
"engine": "bolt",
"gc_interval": 50000000,
"gc_ratio": 1.0,
"log_slow_queries": "2s",
"account": {
"cache_size": 4
},
"account_index":{
"cache_size": 8
},
"contract": {
"cache_size": 2
},
"contract_index":{
"cache_size": 2
},
"op": {
"cache_size": 4
},
"op_index":{
"cache_size": 128
},
"block": {
"cache_size": 8
},
"block_index":{
"cache_size": 64
},
"flow": {
"cache_size": 4
},
"supply": {
"cache_size": 2
},
"chain": {
"cache_size": 2
},
"right": {
"cache_size": 2
},
"income": {
"cache_size": 2
}
},
"rpc": {
"host": "127.0.0.1",
"port": 8732,
"threads": 10,
"disable_tls": true,
"dial_timeout": "20s",
"keepalive": "30m",
"idle_timeout": "30m",
"response_timeout": "60s",
"continue_timeout": "60s",
"idle_conns": 50
},
"logging": {
"backend": "stdout",
"flags": "date,time,micro,utc",
"level": "info",
"blockchain": "info",
"database": "info",
"rpc": "info",
"server": "debug",
"report": "info",
"micheline": "info"
}
}
We use tzindex version 6.0.1
#./tzindex version
2020/04/14 11:29:03.799537 WARN MAIN Missing config file, using default values.
2020/04/14 11:29:03.799560 INFO MAIN Blockwatch tzindex dev -- none
2020/04/14 11:29:03.799564 INFO MAIN (c) Copyright 2018-2020 -- Blockwatch Data Inc.
2020/04/14 11:29:03.799571 INFO MAIN Starting Blockwatch-tzindex/dev.none on 8 cores
2020/04/14 11:29:03.799574 INFO MAIN Go version go1.13.9
2020/04/14 11:29:03.899669 INFO MAIN Profiling mutex/blocking at 1.00% sample rate.
Blockwatch tzindex dev -- none
(c) Copyright 2020 -- Blockwatch Data Inc.
Go version (client): go1.13.9
tzindex logs for last few minutes: https://pastebin.com/raw/FdTZtjsc
tzindex stuck at height 297251 on carthagenet : unexpected program tag 0x62696e617279
Hi,
I found tzindex stuck this morning with following errors, any idea how to fix this ?
I'm using docker image blockwatch/tzindex
tezos_indexer_1 | 2020/04/01 06:22:11.169391 INFO BLOC Processed 23 blocks in 10.03s (259 transactions, height 297242, 2020-03-26 12:24:29 +0000 UTC, q=1, t=602.48392ms, s=synced)
tezos_indexer_1 | 2020/04/01 06:22:13.798907 WARN BLOC RPC connection error: operation kind origination: micheline: unexpected program tag 0x62696e617279
tezos_indexer_1 | 2020/04/01 06:22:18.854727 INFO BLOC Reorg at height 297251: parent BMXadEGEUM3z4pTdcSt61zBcp3Q3DEM7R4bVAcqdbZjokPpxx8m is not on our main chain, have BMcVtsMY4cdsR9iLesUDSARkG4RvnoevqxHWjFQiUKucMTcxxbR
tezos_indexer_1 | 2020/04/01 06:22:18.863751 INFO BLOC REORGANIZE: searching fork point side=BMcVtsMY4cdsR9iLesUDSARkG4RvnoevqxHWjFQiUKucMTcxxbR main=BMNrzZzadHBYpXw8S9EF5258H69svPEiQHSBHvnMJyYmitVAAuw
tezos_indexer_1 | 2020/04/01 06:22:18.913948 INFO BLOC REORGANIZE: will assemble 0 blocks for detach and 1 for attach
tezos_indexer_1 | 2020/04/01 06:22:18.928949 ERRO BLOC REORGANIZE failed fetching main chain block BMXadEGEUM3z4pTdcSt61zBcp3Q3DEM7R4bVAcqdbZjokPpxx8m: operation kind origination: micheline: unexpected program tag 0x62696e617279
tezos_indexer_1 | 2020/04/01 06:22:18.928961 ERRO BLOC Reorg failed: operation kind origination: micheline: unexpected program tag 0x62696e617279
tezos_indexer_1 | 2020/04/01 06:22:18.975727 WARN BLOC RPC connection error: operation kind origination: micheline: unexpected program tag 0x62696e617279
tezos_indexer_1 | 2020/04/01 06:22:18.989576 INFO BLOC Reorg at height 297251: parent BMXadEGEUM3z4pTdcSt61zBcp3Q3DEM7R4bVAcqdbZjokPpxx8m is not on our main chain, have BMcVtsMY4cdsR9iLesUDSARkG4RvnoevqxHWjFQiUKucMTcxxbR
tezos_indexer_1 | 2020/04/01 06:22:18.989614 INFO BLOC REORGANIZE: searching fork point side=BMcVtsMY4cdsR9iLesUDSARkG4RvnoevqxHWjFQiUKucMTcxxbR main=BMNrzZzadHBYpXw8S9EF5258H69svPEiQHSBHvnMJyYmitVAAuw
tezos_indexer_1 | 2020/04/01 06:22:18.998814 INFO BLOC REORGANIZE: will assemble 0 blocks for detach and 1 for attach
tezos_indexer_1 | 2020/04/01 06:22:19.004707 ERRO BLOC REORGANIZE failed fetching main chain block BMXadEGEUM3z4pTdcSt61zBcp3Q3DEM7R4bVAcqdbZjokPpxx8m: operation kind origination: micheline: unexpected program tag 0x62696e617279
tezos_indexer_1 | 2020/04/01 06:22:19.004718 ERRO BLOC Reorg failed: operation kind origination: micheline: unexpected program tag 0x62696e617279
tezos_indexer_1 | 2020/04/01 06:22:19.013666 INFO BLOC Stopping blockchain sync due to too many errors at 297249.
tezos_indexer_1 | 2020/04/01 06:22:19.241620 ERRO BLOC Exiting monitor loop on cancelled context
tezos_indexer_1 | 2020/04/01 06:22:23.975832 INFO BLOC Context cancelled. Stopping blockchain ingest.
Sync have stopped on block 1212419
We have 4 xtz nodes (v7.4), with tzindex (v6.0.4) all standing at 1212419, the xtz node is working properly.
2020/11/13 08:49:07.879207 ERRO SRVR BW-43d60447f3e1a497c3ea685f1a697fd97b02edce 404 (1305) GET /explorer/account/tz1ikZ1E6Exw1vAb4NWzuUpmbyADTHM2TNad/op?offset=0&limit=0 HTTP/1.1 - ReadAccountOps failed (no such account): account not indexed
2020/11/13 08:49:08.353959 ERRO SRVR BW-ab11fe0b14f0aaf45547a469576d50afbe1fbf58 404 (1305) GET /explorer/account/tz1cQSQtfALQDgW3HZQqtbWvntR4JHiHEGBT/op?offset=0&limit=0 HTTP/1.1 - ReadAccountOps failed (no such account): account not indexed
2020/11/13 08:49:08.477131 ERRO SRVR BW-c433ee25f01f3582e0f66c66971a005b88fd2bad 404 (1305) GET /explorer/account/tz1Xis4LgvB8UuABjmaGEJJJqEWgt6a8kCJR/op?offset=0&limit=0 HTTP/1.1 - ReadAccountOps failed (no such account): account not indexed
2020/11/13 08:49:09.165164 ERRO SRVR BW-96419613bf9c36f357ce3dd0500a617b1f57a9f0 404 (1305) GET /explorer/account/tz1QdL1Nij979xPe3UMduAsH9b2Gz8ZgH5Cq/op?offset=0&limit=0 HTTP/1.1 - ReadAccountOps failed (no such account): account not indexed
2020/11/13 08:49:09.412012 ERRO SRVR BW-20de90f0a669221abc8ae1b42c2664bef1a2ce70 404 (1305) GET /explorer/account/tz1VtWX8ALGGSxmvNuJGSBYor7XvWC3XuUyT/op?offset=0&limit=0 HTTP/1.1 - ReadAccountOps failed (no such account): account not indexed
2020/11/13 08:49:10.041517 ERRO SRVR BW-9c75f96e337084652c19f51e4431d7b8dcba7efe 404 (1305) GET /explorer/account/tz1d3jAWYn4ym73TkGkjse5L5gTMo6RPkbJf/op?offset=0&limit=0 HTTP/1.1 - ReadAccountOps failed (no such account): account not indexed
2020/11/13 08:49:10.433871 ERRO SRVR BW-784f5a46e57b3fffa743afd656596b683ab679c4 404 (1305) GET /explorer/account/tz1WYBq8jr922opGEWx81W9xLSRXLaaxJJgY/op?offset=0&limit=0 HTTP/1.1 - ReadAccountOps failed (no such account): account not indexed
[Bug] Parsing of operation list JSON fails if "kind" is not the first field.
When testing the latest version of tzindex (12.0.3 right now) with TezEdge (current develop) we found that it was getting stuck and re-processing the same blocks again and again.
After some investigation @binier found that when parsing the operation list JSON string, some of the work is done manually, and it assumes that "kind" is the first field:
Lines 155 to 162 in 091c423
Octez does indeed put "kind" first (but that may not always be true), while TezEdge does not (at least not always). Ideally tzindex shouldn't assume the order of the fields.
Some more data:
Logs from error
2022/04/26 17:42:51.337941 DEBG BLOC crawler: queuing block 1 BLAuTiHQdSCXXhfDvv417JKsM8bYT5tNsrpmWyJiMrMdYJNuoLP q=0
2022/04/26 17:42:51.337980 DEBG BLOC crawler: fetching next block 2
2022/04/26 17:42:51.445860 WARN BLOC crawler: RPC connection error: rpc: unsupported op ""
2022/04/26 17:42:56.447496 DEBG BLOC crawler: fetching next block 1
2022/04/26 17:42:57.362569 DEBG BLOC crawler: queuing block 1 BLAuTiHQdSCXXhfDvv417JKsM8bYT5tNsrpmWyJiMrMdYJNuoLP q=0
2022/04/26 17:42:57.362590 DEBG BLOC crawler: reorg detected at 1. Falling back to crawl mode.
2022/04/26 17:42:58.363752 DEBG BLOC crawler: fetching next block 0
2022/04/26 17:42:58.474112 DEBG BLOC crawler: queuing block 0 BLockGenesisGenesisGenesisGenesisGenesis1db77eJNeJ9 q=0
2022/04/26 17:42:58.474150 DEBG BLOC crawler: fetching next block 1
2022/04/26 17:42:59.241439 DEBG BLOC crawler: queuing block 1 BLAuTiHQdSCXXhfDvv417JKsM8bYT5tNsrpmWyJiMrMdYJNuoLP q=0
2022/04/26 17:42:59.241488 DEBG BLOC crawler: fetching next block 2
2022/04/26 17:42:59.241761 TRCE BLOC Processing block 0 BLockGenesisGenesisGenesisGenesisGenesis1db77eJNeJ9
2022/04/26 17:42:59.264832 DEBG DATA Query: account.init_cache =>
2022/04/26 17:42:59.354937 WARN BLOC crawler: RPC connection error: rpc: unsupported op ""
Data being parsed
[
{
"counter": "1",
"fee": "357",
"gas_limit": "1000",
"kind": "reveal",
"metadata": {
"balance_updates": [
{
"change": "-357",
"contract": "tz1foxFdz2ieSj8P9YxKYBTEqYbgFXXEeiQY",
"kind": "contract",
"origin": "block"
},
{
"category": "fees",
"change": "357",
"cycle": 0,
"delegate": "tz3Q67aMz7gSMiQRcW729sXSfuMtkyAHYfqc",
"kind": "freezer",
"origin": "block"
}
],
"operation_result": {
"consumed_gas": "1000",
"consumed_milligas": "1000000",
"status": "applied"
}
},
"public_key": "edpkubSgCoMjB2a6vPWRNRHfGa6HUFfFyEUKoEQhWC8osnav8QzBdJ",
"source": "tz1foxFdz2ieSj8P9YxKYBTEqYbgFXXEeiQY",
"storage_limit": "0"
},
{
"amount": "1000000000000",
"counter": "2",
"destination": "tz1cg5EqC3WdZgRSvGJeW328S4KQNrT4jvyv",
"fee": "310",
"gas_limit": "1520",
"kind": "transaction",
"metadata": {
"balance_updates": [
{
"change": "-310",
"contract": "tz1foxFdz2ieSj8P9YxKYBTEqYbgFXXEeiQY",
"kind": "contract",
"origin": "block"
},
{
"category": "fees",
"change": "310",
"cycle": 0,
"delegate": "tz3Q67aMz7gSMiQRcW729sXSfuMtkyAHYfqc",
"kind": "freezer",
"origin": "block"
}
],
"operation_result": {
"allocated_destination_contract": true,
"balance_updates": [
{
"change": "-1000000000000",
"contract": "tz1foxFdz2ieSj8P9YxKYBTEqYbgFXXEeiQY",
"kind": "contract",
"origin": "block"
},
{
"change": "1000000000000",
"contract": "tz1cg5EqC3WdZgRSvGJeW328S4KQNrT4jvyv",
"kind": "contract",
"origin": "block"
},
{
"change": "-64250",
"contract": "tz1foxFdz2ieSj8P9YxKYBTEqYbgFXXEeiQY",
"kind": "contract",
"origin": "block"
}
],
"consumed_gas": "1420",
"consumed_milligas": "1420000",
"status": "applied"
}
},
"source": "tz1foxFdz2ieSj8P9YxKYBTEqYbgFXXEeiQY",
"storage_limit": "277"
}
]connection error: rpc: GET /chains/main/blocks/1 status 404
Hello! I have a issue when start tzindex. Build from current master (241def1). It seems that tzindex use wrong api request to tezos node.
tzindex run as
/opt/tzindex/bin/tzindex --config /opt/tzindex/config/config.json --dbpath /opt/tzindex/data run
logs
Oct 21 22:29:20 xtz-02 tzindex[19378]: 2019/10/21 22:29:20.016080 INFO BLOC Crawling Tezos Mainnet.
Oct 21 22:29:20 xtz-02 tzindex[19378]: 2019/10/21 22:29:20.019543 INFO BLOC Starting blockchain crawler.
Oct 21 22:29:20 xtz-02 tzindex[19378]: 2019/10/21 22:29:20.019582 WARN SRVR HTTP Server reachable on all interfaces (0.0.0.0)
Oct 21 22:29:20 xtz-02 tzindex[19378]: 2019/10/21 22:29:20.020376 DEBG SRVR Registering debug routes
Oct 21 22:29:20 xtz-02 tzindex[19378]: 2019/10/21 22:29:20.020728 INFO BLOC Starting blockchain sync from height 1.
Oct 21 22:29:20 xtz-02 tzindex[19378]: 2019/10/21 22:29:20.020744 INFO BLOC Starting blockchain ingest.
Oct 21 22:29:20 xtz-02 tzindex[19378]: 2019/10/21 22:29:20.020761 INFO SRVR Starting HTTP server at 0.0.0.0:8000
Oct 21 22:29:20 xtz-02 tzindex[19378]: 2019/10/21 22:29:20.020823 INFO BLOC Starting blockchain monitor.
Oct 21 22:29:20 xtz-02 tzindex[19378]: 2019/10/21 22:29:20.070340 WARN BLOC RPC connection error: rpc: GET /chains/main/blocks/1 status 404 ()
Oct 21 22:29:25 xtz-02 tzindex[19378]: 2019/10/21 22:29:25.073422 WARN BLOC RPC connection error: rpc: GET /chains/main/blocks/1 status 404 ()
Oct 21 22:29:30 xtz-02 tzindex[19378]: 2019/10/21 22:29:30.076258 WARN BLOC RPC connection error: rpc: GET /chains/main/blocks/1 status 404 ()
Oct 21 22:29:35 xtz-02 tzindex[19378]: 2019/10/21 22:29:35.122600 WARN BLOC RPC connection error: rpc: GET /chains/main/blocks/1 status 404 ()
Oct 21 22:38:51 xtz-02 tzindex[19378]: 2019/10/21 22:38:51.751244 WARN BLOC RPC connection error: rpc: GET /chains/main/blocks/1 status 404 ()
Oct 21 22:38:56 xtz-02 tzindex[19378]: 2019/10/21 22:38:56.753995 WARN BLOC RPC connection error: rpc: GET /chains/main/blocks/1 status 404 ()
Oct 21 22:39:01 xtz-02 tzindex[19378]: 2019/10/21 22:39:01.758360 WARN BLOC RPC connection error: rpc: GET /chains/main/blocks/1 status 404 ()
Oct 21 22:39:06 xtz-02 tzindex[19378]: 2019/10/21 22:39:06.761801 WARN BLOC RPC connection error: rpc: GET /chains/main/blocks/1 status 404 ()
Oct 21 22:39:11 xtz-02 tzindex[19378]: 2019/10/21 22:39:11.766047 WARN BLOC RPC connection error: rpc: GET /chains/main/blocks/1 status 404 ()
Oct 21 22:39:16 xtz-02 tzindex[19378]: 2019/10/21 22:39:16.768994 WARN BLOC RPC connection error: rpc: GET /chains/main/blocks/1 status 404 ()
Oct 21 22:39:21 xtz-02 tzindex[19378]: 2019/10/21 22:39:21.772234 WARN BLOC RPC connection error: rpc: GET /chains/main/blocks/1 status 404 ()
Oct 21 22:39:26 xtz-02 tzindex[19378]: 2019/10/21 22:39:26.775274 WARN BLOC RPC connection error: rpc: GET /chains/main/blocks/1 status 404 ()
Oct 21 22:39:31 xtz-02 tzindex[19378]: 2019/10/21 22:39:31.779969 WARN BLOC RPC connection error: rpc: GET /chains/main/blocks/1 status 404 ()
Oct 21 22:39:36 xtz-02 tzindex[19378]: 2019/10/21 22:39:36.783166 WARN BLOC RPC connection error: rpc: GET /chains/main/blocks/1 status 404 ()
...
check rest api of tzindex
# curl http://127.0.0.1:8000/
{
"errors": [
{
"code": 1000,
"status": 404,
"message": "resource not found",
"scope": "NotFound",
"detail": "Unrecognized request URL (GET: /).",
"request_id": "BW-e0fc7209586b65792cbe316a32305c04dec9a638"
}
]
}
# curl http://127.0.0.1:8000/explorer/status
{
"status": "connecting",
"blocks": -1,
"indexed": 0,
"progress": 0
}
tezos-node version
# /opt/tezos/bin/tezos-node --version
a67ca32c (2019-09-20 17:27:30 +0200)
Manual requests to tezos node
/chains/main/blocks/1
# curl -v http://127.0.0.1:8732/chains/main/blocks/1
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8732 (#0)
> GET /chains/main/blocks/1 HTTP/1.1
> Host: 127.0.0.1:8732
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 404 Not Found
< transfer-encoding: chunked
<
* Curl_http_done: called premature == 0
* Connection #0 to host 127.0.0.1 left intact
/chains/main/blocks
# curl -v http://127.0.0.1:8732/chains/main/blocks
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8732 (#0)
> GET /chains/main/blocks HTTP/1.1
> Host: 127.0.0.1:8732
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 200 OK
< content-length: 1010
< content-type: application/json
<
[["BLKdtUqn4aCm6ywf1seTk4AJ4hxYiMgth2sak8jcDiZYZKojmky"],["BKnc8X2HxGMKhxZvSCmpSJ6xC2wXKMYxUhsbGKzGxreabUTvYgW"],["BL6qW4CE19CW7WE4wkS2YTDe7baXmiNi1vFMZzDxQ2zx7Ak2zTR"],["BLdEEmZnmd7Y1g5FAApDMZPhPGmeFZDrmTNLVRraFGta8NUGBcm"],["BM3rTAyNMrHn7g6kcdZYu18PBRcNYbMVNtEXFkyQXDzg5dihbCc"],["BLCfBjyRcU96EARen2q6PNmypwDg2jhc3rRAEJDg3d634e3QUaW"],["BL18Jf5r3NenwCiQmjy15RVAtwPstPGisd39RehVgPde8N62bti"],["BMXgVKUHrxC846R8ELp7VSGXLdeGRcg3M8cJnPj2DeKNEebRpe4"],["BLzGmNCrp69JxRsMiFJCGLH2JknBhAdj8TT8hK7VzEhoYi8PpEX"],["BMD4HBxmmVyWodQA51wsFDcZE7j5bqhiNeeujTNfcoTJXtGJGk6"],["BMHJmEsWBMbPE7CTe8gVX5uT6UhtHWHeDcCTrSKhRc8mkSKJQsh"],["BLsLKa196r3jS7FUjc4pHnPjitkT5h3HAMVKZxmbQYJJr7gVbqy"],["BLipoMqDx2uwoDep2LfzzbfccMWN6Wgq2VZwT3bKQcJ7ALGbjWZ"],["BMDNqWLx6W5xWnsqZd4bS8i5UUr4SxVX54NRQ46QFWp2er3cM1L"],["BKpZebfeJN7sCDKS6ZhE8z4T7fi6Q87Rh7VoeirgELr4tysXgBp"],["BLJrJJ2NmiBCByq8mTghNsWgwKYMxGti15EAHhop278vY2QHW4Y"],["BLz34p1VZUrYAAN4bLYQ5wpbNhuEo5ZD4y622eA6y5UTdq9Sb72"],["BMBjFmSPdbR6HyCnCpcnLawWDQA1EqockN4cYN3qJiLZuWPisCZ"]]
* Curl_http_done: called premature == 0
* Connection #0 to host 127.0.0.1 left intact
/chains/main/blocks/head
# curl -v http://127.0.0.1:8732/chains/main/blocks/head
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8732 (#0)
> GET /chains/main/blocks/head HTTP/1.1
> Host: 127.0.0.1:8732
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 200 OK
< content-length: 25443
< content-type: application/json
<
{"protocol":"PsBabyM1eUXZseaJdmXFApDSBqj8YBfwELoxZHHW77EMcAbbwAS","chain_id":"NetXdQprcVkpaWU","hash":"BMGssnHsaAeVw9NmkvSaPAN88M6NpRSd4EegBWutpe13Hb1CvZc","header":{"level":660271,"proto":5,"predecessor":"BLKdtUqn4aCm6ywf1seTk4AJ4hxYiMgth2sak8jcDiZYZKojmky","timestamp":"2019-10-21T22:36:27Z","validation_pass":4,"operations_hash":"LLoaMnatgBFQPMKE71WGwZbEQ6hpLmGhVKWa8Cs56x1ttNKFsLj1A","fitness":["01","000000000000132f"]
...
config
{
"server": {
"addr": "0.0.0.0",
"port": 8000,
"scheme": "http",
"host": "127.0.0.1",
"workers": 50,
"read_timeout": "2s",
"header_timeout": "5s",
"write_timeout": "900s",
"keepalive": "90s",
"shutdown_timeout": "15s",
"max_list_count": 50000,
"default_explore_count": 20,
"max_explore_count": 100,
"cors_enable": false,
"cors_origin": "*",
"cors_allow_headers": "Authorization, Accept, Content-Type, X-Api-Key, X-Requested-With",
"cors_expose_headers": "Date, X-Runtime, X-Request-Id, X-Api-Version",
"cors_methods": "GET, OPTIONS",
"cors_maxage": "86400",
"cors_credentials": "true",
"cache_enable": false,
"cache_control": "public"
},
"crawler": {
"queue": 100,
"snapshot_path": "./db/xtz/snapshots",
"snapshot_blocks": [],
"snapshot_interval": 0
},
"database": {
"path": "./db/xtz",
"engine": "bolt",
"gc_interval": 50000000,
"gc_ratio": 1.0,
"log_slow_queries": "2s",
"account": {
"cache_size": 4
},
"account_index":{
"cache_size": 8
},
"contract": {
"cache_size": 2
},
"contract_index":{
"cache_size": 2
},
"op": {
"cache_size": 4
},
"op_index":{
"cache_size": 128
},
"block": {
"cache_size": 8
},
"block_index":{
"cache_size": 64
},
"flow": {
"cache_size": 4
},
"supply": {
"cache_size": 2
},
"chain": {
"cache_size": 2
},
"right": {
"cache_size": 2
},
"income": {
"cache_size": 2
}
},
"rpc": {
"host": "127.0.0.1",
"port": 8732,
"threads": 2,
"disable_tls": true,
"dial_timeout": "10s",
"keepalive": "30m",
"idle_timeout": "30m",
"response_timeout": "60s",
"continue_timeout": "60s",
"idle_conns": 2
},
"logging": {
"backend": "stdout",
"flags": "date,time,micro,utc",
"level": "info",
"blockchain": "info",
"database": "info",
"rpc": "info",
"server": "debug",
"report": "info",
"micheline": "info"
}
}
Missing transaction in account endpoint(?)
Dear TzIndex (and I assume TzStats) team,
The endpoint:
https://api.tzstats.com/explorer/account/tz1f4fSycx5gutm4Vn4QqwD7yVcThSY9uYaR/op
seems to be missing an operation:
https://api.tzstats.com/explorer/op/ooH7rtYU4axSv23EoSgYynh1rR85xTPTcwyQ2MhzE7NUqK2Fnmc
Am I using the API incorrectly or is it a bug?
Thanks!
Include chain id into db path
It would be convenient if data store were scoped to chain, e.g. full db path included chain id so that tzindex instance would gracefully handle switching/pointing to a different tezos network. In particular, this would make it easier to pair tzindex with a sandbox such as flextesa for testing/dev scenarios. Currently tzindex shuts down with an error if the tezos node it points to starts serving different chain data, such as the case when sandbox is started/restarted as a new chain.
Few suggestions to improve tzindex
- Provide non-docker build mechanism, not everyone has a docker installed and want to use, some prefer using LXC or KVM or any other type of virtualization, so maybe just add default
maketarget to build using go and another target to build docker, should be easy, e.g.:
BUILD_TARGET=tzindex
BUILD_VERSION=dev
BUILD_COMMIT=$(git rev-parse --short HEAD)
REPO=blockwatch.cc/tzindex
VERSION=$BUILD_VERSION-$BUILD_COMMIT
# clean everything
go clean
# update dependencies
go mod download
CGO_ENABLED=0 go build -a -v -o ./${BUILD_TARGET} -ldflags "-w -X ${REPO}/cmd.VERSION=${BUILD_VERSION} -X ${REPO}/cmd.GITCOMMIT=${BUILD_COMMIT}" ${BUILD_TARGET}.go
strip ./${BUILD_TARGET}
-
Tag every stable release with a version and provide basic changelog. I see there is a
v005on github as a latest release, maybe switch to different version notation, e.g.v005-1.0as I understand v005 means protocol number (protocol 005 stands for babylon) on tezos -
Add shortcuts for
--v --vv --vvv-v -vv -vvvas it's much easier to input and most linux/unix programs use one dash instead of two for verbose -
If you try to run
tzindexwithout specifyingconfig.jsonit will fail and try to accesshttps://127.0.0.1:8732instead of the default onehttp://127.0.0.1:8732and it's very hard to figure out for beginners that there is an issue in http scheme as there is no option in help oftzindexto switchtlson or off. Would be nice to add--tlsoption in the help and also switch it off by default.
Thanks!
larger-than-documented RAM usage
With the blockwatch/tzindex:v6.0.2-6-g69ae9fe docker image on carthagenet and an existing database that's fairly up to date (last Wednesday at least) the tzindex process will use 30+GB within a couple of minutes after startup (couldn't measure it precisely since the machine i'm running it on doesn't have enough RAM to get to a steady state). The README says 4-16GB required. It also claims to be tunable but so far I have been able to find a setting that has a significant effect either direction that I can detect.
Bind to external access
Hello, I'm working on some updates in my infrastructure. And facing trying to access from outside the instance. I'm sure that is not any kind of firewall blocking and the config seems to be set to allow access from 0.0.0.0.
{
"server": {
"addr": "0.0.0.0",
"port": 8000,
"scheme": "http",
"host": "0.0.0.0",
"workers": 50,
"read_timeout": "2s",
"header_timeout": "5s",
"write_timeout": "900s",
"keepalive": "90s",
"shutdown_timeout": "15s",
"max_list_count": 50000,
"default_explore_count": 20,
"max_explore_count": 100,
"cors_enable": false,
"cors_origin": "*",
"cors_allow_headers": "Authorization, Accept, Content-Type, X-Api-Key, X-Requested-With",
"cors_expose_headers": "Date, X-Runtime, X-Request-Id, X-Api-Version",
"cors_methods": "GET, OPTIONS",
"cors_maxage": "86400",
"cors_credentials": "true",
"cache_enable": false,
"cache_control": "public"
},
"crawler": {
"queue": 100,
"snapshot_path": "/tmp/db/xtz/snapshots",
"snapshot_blocks": [],
"snapshot_interval": 0
},
"database": {
"path": "/tmp/db/xtz",
"engine": "bolt",
"gc_interval": 50000000,
"gc_ratio": 1.0,
"log_slow_queries": "2s",
"account": {
"cache_size": 4
},
"account_index":{
"cache_size": 8
},
"contract": {
"cache_size": 2
},
"contract_index":{
"cache_size": 2
},
"op": {
"cache_size": 4
},
"op_index":{
"cache_size": 128
},
"block": {
"cache_size": 8
},
"block_index":{
"cache_size": 64
},
"flow": {
"cache_size": 4
},
"supply": {
"cache_size": 2
},
"chain": {
"cache_size": 2
},
"right": {
"cache_size": 2
},
"income": {
"cache_size": 2
},
"bigmap": {
"cache_size": 2
},
"bigmap_index": {
"cache_size": 16
}
},
"rpc": {
"host": "0.0.0.0",
"port": 8732,
"threads": 2,
"disable_tls": true,
"dial_timeout": "10s",
"keepalive": "30m",
"idle_timeout": "30m",
"response_timeout": "60s",
"continue_timeout": "60s",
"idle_conns": 2
},
"logging": {
"backend": "stdout",
"flags": "date,time,micro,utc",
"level": "info",
"blockchain": "info",
"database": "info",
"rpc": "info",
"server": "info",
"micheline": "info"
}
}
When running locally I'm able to do the request:
curl http://127.0.0.1:8000/explorer/status
{
"mode": "sync",
"status": "failed",
"blocks": 89127,
"indexed": 10243,
"progress": 0.1149258922660922
}
When running from outside the instance has no response.
When looking the logs i see the following message:
WARN SRVR HTTP Server reachable on localhost only
Am I missing something here?
Can someone help?
Many thanks, appreciate.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.