subquery / subql

SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.

Welcome to SubQuery!

Flexible, reliable, and decentralised APIs for your web3 project

SubQuery is an Open, Flexible, Fast and Universal data indexing framework for web3. Our mission is to help developers create the decentralised products of the future.

SubQuery allows every Substrate/Polkadot team to process and query their data. The project is inspired by the growth of data protocols serving the application layer and its aim is to help Polkadot/Substrate projects build better dApps by allowing anyone to reliably find and consume data faster. Today, anyone can query and extract Polkadot network data in only minutes and at no cost.

The future is multi-chain - SubQuery is no different. SubQuery is well on our way to support all leading blockchain networks with support for the following:

• Polkadot (and all Substrate networks)
• [Ethereum] (https://academy.subquery.network/quickstart/quickstart_chains/ethereum-gravatar.html)
• Cosmos (and all CosmWasm and Ethermint networks)
• Avalanche (Deprecated, please use Ethereum)
• Algorand
• NEAR
• Flare
• Polygon
• BSC

Get Started

Create a SubQuery project

You can follow our Quick Start Guide to learn how to create, initialize, build, and publish a new SubQuery Project using the @subql/cli tool.

Learn and improve with our comprehensive documentation

Dig into every term, usecases, and best-practices that help you build a dApp which your users love. Take a look at our detailed technical documentation.

Publish your SubQuery Project to our Managed Service

Don't want to worry about running your own indexing infrastructure? SubQuery provides a managed Service with industry leading hosting so you can sleep easy. Trusted by hundreds of projects with millions of daily requests, follow our publishing guide to see how you can upload your project to SubQuery Projects.

Run your own Indexer and Query Service

Follow our guide to run your own SubQuery local node that you can use to debug, test, and run you own GraphQL server

You're going to need to a Postgres database, a node to extract chain data, and a moderately powerful computer to run the indexer in the background.

You'll also use our custom-built GraphQL query service @subql/query to interact with your SubQuery project.

Components

• @subql/cli
• @subql/node
• @subql/query
• @subql/common
• @subql/common-substrate
• @subql/types

Contribute

We love contributions and feedback from the community. To contribute the code, we suggest starting by creating an issue in our main repository so we can give you support.

Copyright

SubQuery is a project built with love from the team at SubQuery all the way from New Zealand

Copyright © 2022 SubQuery Pte Ltd authors & contributors

subql's Issues

[cli] support launch a @subql/node to index current project

It will be better if user don't have to clone the project so they can run @subql/node from their local, instead we let user run subql node start.

[node] Node dbModule input should be pass from environment

At moment dbModule in node is using constant values for username, password and database, it should be gather from the environment variable in subquery project docker-compose file, so user can defined them freely correspond with their db specs.

Create and publish docker image for subquery node

It will be much simpler for subquery user to use the docker image initialize the node.

[Cli] should load schema from project.yml

current schema.graphql path is fixed, should be load from project.yml

[node] fix BigInt transformation when column value is null

In objectTypeToModelAttributes we defined a custom getter and setter to transform column value to / from BigInt.

But it didn't handle null well.

[node] wrap execution of mapping function in a postgres transaction

This way when error occurs, all changes will be rolled back.

[cli] need step-by-setp interaction with user

As we have proposed, we need to add step-by-step interaction with the user for cli, guides the user to complete customisation of the project.yaml and create handlers of the selected type

[cli] model template need to update

1. no longer need BN types
2. save() should await store to set
3. if get record from store is null, should return null instead of create entity
4. change "BigInt" to "bigint"

review eslint rules and apply the rule

• review and config eslint rules
• apply eslint to the project
• add linting to PR workflow

[@subql/node] support load subquery project from its tarball pack

It is promised in the grant, and we have already supported packing subquery project into tarball.

For @subql/node, it will need to unpack the tarball file into a temp directory, then it will be no different with current flow.
Also the checksum of the tarball file need to be recorded in db.

[node] expose syncing status via api

a sample payload for the api return

{
"syncing": true,
"finalizedBlock": 10000,
"syncedBlock": 500
}

crash with event filter used

Errors indexer [IndexerManager] failed to handler block 2392, error: Cannot read property 'block' of undefined │

This is thrown in utils/substrate.ts: #filterEvents. Because some events might not be owned by an extrinsic. (Phase is not ApplyExtrinsic)

But we still want user have access to the block that produce the event.

We shall make the following changes:

1. change SubstrateEvent, add block: SubstrateBlock
2. wrapEvents will require block as parameter

[node] add bin/ for `@subql/node` and publish it to npmjs

Add startup script for @subql/node, and publish it to npmjs. Then user can install it to global via npm i -g @subql/node and run it directly.

We shall also update subql-node's docker image, use npm i -g @subql/node instead of build from source code.

[node] support specVersion filter for all runtime handlers

[node] use main field in package.json for subquery's entry file

Currently subql node will try to load dist/index from the given local subquery path, but rather we should first find package.json from that path and use the path defined in main field as the entry file path.

[node] support filters for CallHandler and EventHandler

only stable release will update latest tag on npmjs

Currently the package will by default be attached to the latest tag on the registry, we want only stable release to update on latest tag, prerelease will publish beta tag.

[node] support success filter for callHandler

Success or failure extrinsics will all trigger callHandler, and the status can be visited via extrinsic.success.
If we can provide a success filter, user can use it instead of check in the mapping function.

[node] increase ws maximum frame size

When processing polkadot blockheight: 29231, there's a large sudo.setCode call and cause api fail.

API-WS: disconnected from wss://polkadot.api.onfinality.io/public-ws: 1009:: Frame size of 3691879 bytes exceeds maximum accepted frame size

When create the WsProvider, there's a way to customize max frame size, we will need to increase it to avoid this error.

[cli] init command allow user pass project name

[@subql/node] improve schema name generation

Currently I use the subquery project's directory name as schema name, but it could lead to troublesomeness when the name is not a valid schema name.

I think a good way is create a sequence in public schema and name all subquery project schema as subql[x]

[node] auto resume indexing after ws reconnect

Currently, if ws connection was disconnected and then recovered, indexing process can not automatically resume.

[node] tar dependency should be under node package

move tar dependency to package.json under node package

add a script to initialise hasura

The first time hasura starts, there're no queries available, because hasura doesn't know any about our schema and tables.
We will need a script to setup hasura, tells it which tables it need to track and customize the names of our tables (remove schema prefix).

References:
https://hasura.io/docs/1.0/graphql/core/schema/custom-field-names.html
https://hasura.io/docs/1.0/graphql/core/api-reference/schema-metadata-api/table-view.html#set-table-custom-fields

Is it possible to support configuration of typesbundle ?

https://github.com/OnFinality-io/subql/blob/8d1dd79c7607e84fe335f9354de3e78a56ddabac/packages/node/src/indexer/api.service.ts#L13-L19

Only configuration types are supported now. So I can't configure types based on the specVersion of the chain . It is recommended to add configuration for typesBundle typesAlias typesChain and typesSpec, just like @polkadot/api.

https://github.com/polkadot-js/api/blob/master/packages/types/src/types/registry.ts#L59-L81

[node] add elasticsearch service support for expose service logs

elasticsearch is the most popular service for store log data. we'd better add ES support directly in the indexer node service.

e.g.:

yarn start -f ./projectdir --es-url=es.service:9200

BigInt support, use bigint from esnext instead of BN

Currently we use BN to represent big number, it is a 3rd lib so we will have to inject it into the sandbox which is not very safe.

I suggest we use the primitive type BigInt which doesn't have these problems.

support custom types injection

Subql is a tool for all substrate-based chains, therefore we should allow user to pass their own type definitions so indexer can connect to their chain and decode data.

There can be two stages for this support

1. We allow user to provide a json like type definition.
2. We allow user to provide a npm lib which holds the type definition.

The second stage is more complicated because to support it we need to make changes to our sandbox impl, add whitelist to allow the user provided lib be loaded. Also we need to add an addition step which will install that particular package after we download the subquery project.

[node] load configurations from a config file

User can define config items in a config file and pass it to @subql/node.

And they are still allowed to pass config in command argments to override.

[node] support load subquery project from a git repo

It is good to support a scenario that user gives a git repo url to the subql node and the node download the code to a temp dir and use it from there.

For a url uses protocol http(s) or git, we can assume that is the case, and the url refers to a git repo.

[common] parse graphql schema to a intermediate object

We can extract entity definitons from the given graphql.schema, but the return type we keep using GraphQLObjectType from graphql package, which is not good design. It makes the downstream codes hard to make use of it and adds unnecessary dependency.

deploy k8s yaml

For people who wants to use it in k8s

[cli] init starter project should create a clean project for user

remove git from the cloned starter folder

[doc] fix baseUrl when using custom domain

the original site: https://onfinality-io.github.io/subql/ works well, but when switching to our custom domain doc.subquery.network, it will be blank because it tried to load js from doc.subquery.network/subql

[query] prevent user to query too many records which effect db performance issue.

set a default max records user can query

[node] Improve logging in subql-node

console.log definitely is not a good idea to use in a production ready software.
A few goals to achieve for the logging of subql-node

• configurable(level) for each topic
• format supports: plain, json.
• --debug will enable printing error stacks
• ansi color

Some solutions are worth looking into.

• https://github.com/pinojs/pino
• https://github.com/winstonjs/winston
• https://github.com/baryon/tracer
• https://github.com/xpl/ololog
• https://github.com/gajus/roarr

[cli] command subql init should add hint in error message or have a default option

use command "subql init" without add "--starter" will promote error, but also provide insufficient information for user.
It should have --starter as a default option , or list options in the promote message.

Improve testing

• Print unit test coverage and raise the coverage for the whole repo
• Add integration test: load and index scaffold project with @subql/node

cache metadata before batch query

It happens when we query 100 blocks as a batch, and we don't have their metadata/registry cached. All 100 queries will trigger rpc.chain.getMetadata which leads to OOM kill.

We already manually call api.getBlockRegistry at the beginning of the block fetching loop, but that's not enough.

We should also check specVersion of the endblockheight+1 of each batch, if specVersion is bumped, we need to call api.getBlockRegistry for that block as well to prevent the worst scenario from happening.

[query] turn off error stack return in the response in the production

Add bps benchmark (block processed per second)

Similar output to substrate node

2021-01-17 00:00:00 [Subquery Project Name] 10.0 bps, target=#xxx, current: #xxx, estimate time: x d, y h, z m 

[node] Prefetch blocks to accelerate indexing process

Preparing blocks is a bottleneck currently when indexing blocks. Eventually we can go to the solution that cache all block data in postgres and feed indexer from there, but at the moment I feel it adds too much complexity.

If we implement prefetching, let's say, 100 blocks a time, that should be already a big improvement and satisfaction comparing to the effort we devote.

[cli] update scaffold to include call/event handler

Since we have supported call handler and event handler, we should put them in the scaffold template. And improve the comments to make it clear how user should adjust the scaffold to suit user's own need.

patch api that exposed in sandbox context

We will allow access to ApiPromise of @polkadot/api, but with a patch that

1. all state query will locked to a given blockhash which can be set by IndexerManager
2. api.query...at() will be removed
3. subscribe like queries will be removed
4. api.tx., api.rx.tx. will all be disabled
5. api.rpc., api.rx.rpc. will all be disabled
6. api.consts, events, errors will be replaced to the version at the given blockhash
7. api.errors and api.events will be replaces according to the metadata at that block
8. patch api._rx
9. make sure api.derive works disable api.derive.* and api.rx.derive.*
10. invalidate caches in api.derive

[node] respect startBlock defined in project.yaml

Currently indexer always starts from block 0, but since we let user specify it, we should use that instead (if provided)

Support type: Bytes and Float

improve docs

Contents to be added/updated:
[x] contents about call handler and event handler, including usage of filter
[] explanation of our sandbox mechanism
[] custom type definition
[x] usage of hasura

[node] api to expose health check status

add a metadata table to log running status of current subquery project

we can log several runtime information to this _metadata table

• current indexed block height
• target block height
• @polkadot/api version used
• network name
• genesis hash
• has errors
• uptime

[node] Node initialization should await Postgres get ready

Current node image often failed during initialization, due to it start running before the Postgres ready to accept connection.