Warning/Errors in tzindex logs about tzindex

[echa]

Hey, happy to hear that! We will publish a new version soon.

The warnings are issued by the database layer when queries take longer than a pre-defined amount of time (default 1s, settable with TZ_DATABASE_LOG_SLOW_QUERIES=duration or in the config file under key database.log_slow_queries).

The origin of these warnings (and slow queries in general) are large table scans. To speed up queries you can configure the indexer to cache more data in memory. Optimal settings depend very much on your use case, default settings are conservative to have a low memory footprint. You can define how much data you like to cache on a per-table basis. The database works with packs as the smallest unit of processing/caching, each pack storing 32k-64k data rows. Here's how to determine cache requirements and set cache config:

1. Inspect database tables with GET /system/tables and look for packs_count and index_packs_count. They indicate the total number of packs in each table. Also look for high values of packs_bytes_read, they indicate which tables are hot spots.
2. Then set the number of cached packs close to the number of total packs, this ensures all data is kept in memory. Config file keys are called database.{table}.cache_size and database.{table}_index.cache_size, ENV vars are TZ_DATABASE_{TABLE}_CACHE_SIZE and TZ_DATABASE_{TABLE}_INDEX_CACHE_SIZE. Values are in number of packs, not bytes.
3. Monitor container memory usage to avoid swapping. You can also find per-table memory usage metrics in GET /system/mem. Go garbage-collects memory from time to time. You can tune how aggressive GC operates by setting the command line flag --gogc <num> (default 20, range 1..100, lower values trigger GC more often).

The 'account not indexed' errors indicate that the indexer has not seen the address on-chain so far. This is a typical error message for wallet-style use cases where the wallet creates a new address, sends a transaction and waits for inclusion on-chain. As long as the tx is not confirmed, the indexer does not know the address. In the future we may add mempool support, so that new addresses and pending operations are indexed as well.

FYI: We will have a new version of both OSS and PRO versions ready soon. There have been a few minor bugs and some new features that may be relevant for you.

[flaupretre]

Hi, thanks for this information. I will try to optimize memory using this procedure.