covenantsql / covenantsql

A decentralized, trusted, high performance, SQL database with blockchain features

中文简介

CovenantSQL(CQL) is a Byzantine Fault Tolerant relational database built on SQLite:

• ServerLess: Free, High Availabile, Auto Sync Database Service for Serverless App
• SQL: Most SQL-92 support.
• Decentralize: Running on Open Internet without Central Coordination.
• Privacy: Access with Granted Permission and Encryption Pass.
• Immutable: Query History in CQL is Immutable and Trackable.
• Permission: Column Level ACL and SQL Pattern Whitelist.

What is CQL?

• Open source alternative of Amazon QLDB
• Low cost DBaaS
• Just like filecoin + IPFS is the decentralized file system, CQL is the decentralized database

Quick Start

CQL client supports:

• macOS X 10.9+
• Linux 2.6.23+ (x86, x86_64, armeabi-v7a, arm64-v8a)

Developer Guide

MacOS

• 🍺 Homebrew users can just run:

  brew install cql

• non-Homebrew users can run:

  sudo bash -c 'curl -L "https://mac.gridb.io/cql" | tar xzv -C /usr/local/bin/ --strip-components=1'

Linux

• Just run：

  sudo bash -c 'curl -L "https://linux.gridb.io/cql" | tar xzv -C /usr/local/bin/ --strip-components=1'

To continue: TestNet Quickstart

SDKs

CovenantSQL Testnet is already released, have a try.

• Golang
• Java
• NodeJS
• Python
• Microsoft Excel (by community)

How CQL works

3 Layers Arch

• Layer 1: Global Consensus Layer (the main chain, the middle ring in the architecture diagram):
  • There will only be one main chain throughout the network.
  • Mainly responsible for database Miner and the user’s contract matching, transaction settlement, anti-cheating, shard chain lock hash and other global consensus matters.
• Layer 2: SQL Consensus Layer (shard chain, rings on both sides):
  • Each database will have its own separate shard chain.
  • Mainly responsible for: the signature, delivery and consistency of the various Transactions of the database. The data history of the permanent traceability is mainly implemented here, and the hash lock is performed in the main chain.
• Layer 3: Datastore Layer (database engine with SQL-92 support):
  • Each Database has its own independent distributed engine.
  • Mainly responsible for: database storage & encryption, query processing & signature, efficient indexing.

For more details

Consensus Algorithm

CQL supports 2 kinds of consensus algorithm:

1. DPoS (Delegated Proof-of-Stake) is applied in Eventually consistency mode database and also Layer 1 (Global Consensus Layer) in BlockProducer. CQL miners pack all SQL queries and its signatures by the client into blocks thus form a blockchain. We named the algorithm Xenomint.
2. BFT-Raft (Byzantine Fault-Toleranted Raft)^bft-raft is applied in Strong consistency mode database. We named our implementation Kayak. The CQL miner leader does a Two-Phase Commit with Kayak to support Transaction.^transaction

CQL database consistency mode and node count can be selected in database creation with command cql create '{"UseEventualConsistency": true, "Node": 3}'

Comparison

	Ethereum	Hyperledger Fabric	Amazon QLDB	CovenantSQL
Dev language	Solidity, ewasm	Chaincode (Go, NodeJS)	?	Python, Golang, Java, PHP, NodeJS, MatLab
Dev Pattern	Smart Contract	Chaincode	SQL	SQL
Open Source	Y	Y	N	Y
Nodes for HA	3	15	?	3
Column Level ACL	N	Y	?	Y
Data Format	File	Key-value	Document	Filefuse, Key-value, Structured
Storage Encryption	N	API	Y	Y
Data Desensitization	N	N	N	Y
Multi-tenant	DIY	DIY	N	Y
Throughput (1s delay)	15~10 tx/s	3500 tx/s	?	11065 tx/s (Eventually Consistency) 1866 tx/s (Strong Consistency)
Consistency Delay	2~6 min	< 1 s	?	< 10 ms
Secure for Open Internet	Y	N	Only in AWS	Y
Consensus	PoW + PoS(Casper)	CFT	?	DPoS (Eventually Consistency) BFT-Raft (Strong Consistency)

FootNotes

• BFT-Raft: A CQL leader offline needs CQL Block Producer to decide whether to wait for leader online for data integrity or promote a follower node for availability. This part is still under construction and any advice is welcome.

• Transaction: Talking about ACID, CQL has full "Consistency, Isolation, Durability" and a limited Atomicity support. That is even under strong consistency mode, CQL transaction is only supported on the leader node. If you want to do "read v, v++, write v back" parallelly and atomically, then the only way is "read v from the leader, v++, write v back to leader"

• FUSE: CQL has a simple FUSE support adopted from CockroachDB. The performance is not very ideal and still has some issues. But it can pass fio test like:

  fio --debug=io --loops=1 --size=8m --filename=../mnt/fiotest.tmp --stonewall --direct=1 --name=Seqread --bs=128k --rw=read --name=Seqwrite --bs=128k --rw=write --name=4krandread --bs=4k --rw=randread --name=4krandwrite --bs=4k --rw=randwrite

Network Stack

DH-RPC := TLS - Cert + DHT

Layer	Implementation
RPC	net/rpc
Naming	Consistent Secure DHT
Pooling	Session Pool
Multiplex	smux
Transport Security	Enhanced TLS
Network	TCP or KCP for optional later

Test Tools we use

• Global Network Topology Emulator is used for network emulating.
• Liner Consistency Test

Papers

Our team members published:

• Thunder crystal: a novel crowdsourcing-based content distribution platform
• Analyzing streaming performance in crowdsourcing-based video service systems
• Performance Analysis of Thunder Crystal: A Crowdsourcing-Based Video Distribution Platform

that inspired us:

• Bitcoin: A Peer-to-Peer Electronic Cash System
• S/Kademlia
  • S/Kademlia: A practicable approach towards secure key-based routing
• vSQL: Verifying arbitrary SQL queries over dynamic outsourced databases

Performance

Strong consistency bench result (2 miners, 8 core aws c5.2xlarge):

As seen above, the concurrency pressure on the database increased gradually in the first 5 hours, and the write TPS also increased. When the TPS no longer grows, the concurrent pressure is maintained for 100 hours.

Demos

• CovenantForum
• Weibo Bot @BlockPin
• Markdown Editor with CovenantSQL sync
• Web Admin for CovenantSQL
• How CovenantSQL works(video)

Use cases

Traditional App & ĐApp use cases

Traditional App

Privacy data

If you are a developper of password management tools just like 1Password or LastPass. You can use CQL as the database to take benefits:

1. Serverless: no need to deploy a server to store your user's password for sync which is the hot potato.
2. Security: CQL handles all the encryption work. Decentralized data storage gives more confidence to your users.
3. Regulation: CQL naturally comply with GDPR.

IoT storage

CQL miners are deployed globally, IoT node can write to nearest CQL miner directly.

1. Cheaper: Without passing all the traffic through a gateway, you can save a large bandwidth fee. And, CQL is a shared economic database which makes storage cheaper.
2. Faster: CQL consensus protocol is designed for Internet where network latency is unavoidable.

Open data service

For example, you are the most detailed Bitcoin OHLC data maintainer. You can directly expose an online SQL interface to your customers to meet a wide range of query needs.

1. CQL can limit specific SQL query statements to meet the needs while also balancing data security;
2. CQL can record SQL query records on the blockchain, which is very convenient for customers to check their bills for long-tail customers and billing, like this
3. For customers with high performance requirements, Slave nodes can be deployed at the customer to meet the needs of customers with low latency queries while enabling almost real-time data updates.

Secure storage

Thanks to the CQL data history is immutable, CQL can be used as a storage for sensitive operational logs to prevent hacking and erasure access logs.

ĐApp

Storing data on Bitcoin or Ethereum is quite expensive ($4305 / MB on Ethereum 2018-05-15). Programming is very complicated due to the lack of support for structured data storage. CQL gives you a low-cost structured SQL database and also provides more room for ĐApp to exchange data with real-world.

TestNet

• Quick Start
• MainChain Explorer
• SQLChain Explorer
• Demo & Forum

Contact

• Blog
• YouTube
• Mail
• Forum
• 
• 
• Add CovenantSQL to join WeChat group

  

covenantsql's Issues

cql-adapter cannot process “-” in quote correctly.

desc command cannot process "-" in quote while select can.

Syntax error on drop table and create same table multiple times

table_name = "test_python_driver"

for i in range(2):
    cur.execute("drop table if exists `%s`" % table_name)
    cur.execute('''create table `%s` (
        name text not null,
        primary key(name)
        )''' % table_name)

While I use pycovenantsql to drop table and create table for multiple times, the covenantsql return me an err like table `test_python_driver` already exists. It seems second drop sql does not affected but returned true.

Blocks are not written to chain

In bp logs there are messages that blocks are created, from miner to messages displayed cached_block_count (increases with each block). However, no blocks are added to the chain itself
Only the first empty block is displayed in the Explorer
Used the latest version from github for testing. Starting with 'make start'

signal SIGBUS: bus error code

covenantsql_observer         | time="2018-12-23T07:35:53Z" level=debug msg="PingBP resp: &proto.PingResp{Msg:\"Pong\", Envelope:proto.Envelope{Version:\"\", TTL:0, Expire:0, NodeID:(*proto.RawNodeID)(nil), _ctx:context.Context(nil)}}"
covenantsql_observer         | unexpected fault address 0x7f47b829a040
covenantsql_observer         | fatal error: fault
covenantsql_observer         | [signal SIGBUS: bus error code=0x2 addr=0x7f47b829a040 pc=0x93e9db]
covenantsql_observer         |
covenantsql_observer         | goroutine 5 [running]:
covenantsql_observer         | runtime.throw(0xd74b5c, 0x5)
covenantsql_observer         | 	/usr/local/go/src/runtime/panic.go:608 +0x72 fp=0xc0000508d8 sp=0xc0000508a8 pc=0x42e162
covenantsql_observer         | runtime.sigpanic()
covenantsql_observer         | 	/usr/local/go/src/runtime/signal_unix.go:387 +0x2d7 fp=0xc000050928 sp=0xc0000508d8 pc=0x443477
covenantsql_observer         | github.com/CovenantSQL/CovenantSQL/vendor/github.com/coreos/bbolt.(*DB).meta(0xc0001183c0, 0xc00023a0c0)
covenantsql_observer         | 	/go/src/github.com/CovenantSQL/CovenantSQL/vendor/github.com/coreos/bbolt/db.go:808 +0x3b fp=0xc000050978 sp=0xc000050928 pc=0x93e9db
covenantsql_observer         | github.com/CovenantSQL/CovenantSQL/vendor/github.com/coreos/bbolt.(*Tx).init(0xc0002ae0e0, 0xc0001183c0)
covenantsql_observer         | 	/go/src/github.com/CovenantSQL/CovenantSQL/vendor/github.com/coreos/bbolt/tx.go:50 +0xa2 fp=0xc000050a18 sp=0xc000050978 pc=0x944b52
covenantsql_observer         | github.com/CovenantSQL/CovenantSQL/vendor/github.com/coreos/bbolt.(*DB).beginTx(0xc0001183c0, 0xc3e2e0, 0x7f47bc7d7000, 0x0)
covenantsql_observer         | 	/go/src/github.com/CovenantSQL/CovenantSQL/vendor/github.com/coreos/bbolt/db.go:486 +0x99 fp=0xc000050a88 sp=0xc000050a18 pc=0x93d5e9
covenantsql_observer         | github.com/CovenantSQL/CovenantSQL/vendor/github.com/coreos/bbolt.(*DB).Begin(0xc0001183c0, 0x4d5f00, 0xc000050af8, 0x40f608, 0x8)
covenantsql_observer         | 	/go/src/github.com/CovenantSQL/CovenantSQL/vendor/github.com/coreos/bbolt/db.go:463 +0x6e fp=0xc000050ab8 sp=0xc000050a88 pc=0x93d51e
covenantsql_observer         | github.com/CovenantSQL/CovenantSQL/vendor/github.com/coreos/bbolt.(*DB).View(0xc0001183c0, 0xc000050b30, 0x0, 0x0)
covenantsql_observer         | 	/go/src/github.com/CovenantSQL/CovenantSQL/vendor/github.com/coreos/bbolt/db.go:610 +0x3c fp=0xc000050b08 sp=0xc000050ab8 pc=0x93debc
covenantsql_observer         | main.(*Service).getHighestBlockV2(0xc0000a1590, 0xc0002a400d, 0x40, 0x0, 0xc0002b2748, 0xd28ba0, 0xc00022a300)
covenantsql_observer         | 	/go/src/github.com/CovenantSQL/CovenantSQL/cmd/cql-observer/service.go:694 +0xbe fp=0xc000050b70 sp=0xc000050b08 pc=0xa619fe
covenantsql_observer         | main.(*explorerAPI).GetHighestBlockV2(0xc000142258, 0xf6b8c0, 0xc0002ae000, 0xc000228700)
covenantsql_observer         | 	/go/src/github.com/CovenantSQL/CovenantSQL/cmd/cql-observer/api.go:349 +0xf0 fp=0xc000050bf0 sp=0xc000050b70 pc=0xa57df0
covenantsql_observer         | main.(*explorerAPI).GetHighestBlockV2-fm(0xf6b8c0, 0xc0002ae000, 0xc000228700)
covenantsql_observer         | 	/go/src/github.com/CovenantSQL/CovenantSQL/cmd/cql-observer/api.go:573 +0x48 fp=0xc000050c20 sp=0xc000050bf0 pc=0xa656a8
covenantsql_observer         | net/http.HandlerFunc.ServeHTTP(0xc00013f8c0, 0xf6b8c0, 0xc0002ae000, 0xc000228700)
covenantsql_observer         | 	/usr/local/go/src/net/http/server.go:1964 +0x44 fp=0xc000050c48 sp=0xc000050c20 pc=0x6925c4
covenantsql_observer         | github.com/CovenantSQL/CovenantSQL/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc0001aa460, 0xf6b8c0, 0xc0002ae000, 0xc000228700)
covenantsql_observer         | 	/go/src/github.com/CovenantSQL/CovenantSQL/vendor/github.com/gorilla/mux/mux.go:162 +0xf1 fp=0xc000050d48 sp=0xc000050c48 pc=0x960d41
covenantsql_observer         | net/http.serverHandler.ServeHTTP(0xc000124a90, 0xf6b8c0, 0xc0002ae000, 0xc000228500)
covenantsql_observer         | 	/usr/local/go/src/net/http/server.go:2741 +0xab fp=0xc000050d78 sp=0xc000050d48 pc=0x69522b
covenantsql_observer         | net/http.(*conn).serve(0xc00022c0a0, 0xf6c280, 0xc000064540)
covenantsql_observer         | 	/usr/local/go/src/net/http/server.go:1847 +0x646 fp=0xc000050fc8 sp=0xc000050d78 pc=0x691626
covenantsql_observer         | runtime.goexit()
covenantsql_observer         | 	/usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000050fd0 sp=0xc000050fc8 pc=0x45ba41
covenantsql_observer         | created by net/http.(*Server).Serve
covenantsql_observer         | 	/usr/local/go/src/net/http/server.go:2851 +0x2f5

Info about mainnet?

Hi,

Where can I find information about CovenantSQL mainnet? And how to get PTC for mainnet?

用 cql 来查看你的钱包余额时出错

[root@localhost bin]# cql -config conf/config.yaml -get-balance
ERRO[0001] connect to 120.79.254.36:11105 failed: dial tcp 120.79.254.36:11105: connect: connection refused caller="client.go:60 rpc.dial"
ERRO[0001] connect to 120.79.254.36:11105: dial tcp 120.79.254.36:11105: connect: connection refused caller="client.go:156 rpc.dialToNodeEx"
ERRO[0001] dialToNode failed: dial tcp 120.79.254.36:11105: connect: connection refused caller="client.go:99 rpc.DialToNode"
ERRO[0001] dialing to node: 0000011839f464418166658ef6dec09ea68da1619a7a9e0f247f16e0d6c6504d failed: dial tcp 120.79.254.36:11105: connect: connection refused caller="rpcutil.go:140 rpc.(*Caller).CallNodeWithContext"
ERRO[0001] call DHT.Ping failed: dial tcp 120.79.254.36:11105: connect: connection refused caller="rpcutil.go:253 rpc.PingBP"
ERRO[0001] init covenantsql client failed: dial tcp 120.79.254.36:11105: connect: connection refused caller="main.go:196 main.main"

Improve commit messages for better project tracking and changelogs

An issue inspired by ipfs/aegir#30

给CovenantSQL提点建议

我是大约一年前碰巧知道了CovenantSQL这个项目，做去中心化数据库。我曾经关注过的Bluzelle也属于这个领域，但Bluzelle开发进度很慢，又基于ETH环境，所以总觉得不是那么纯粹。

从区块链生态上看，filecoin做了文件的去中心化存储，那么数据的去中心化存储怎么解决？放眼望去，大量项目集中于公链、跨链、defi、dapp，但如果不解决数据的去中心存储，区块链应用就缺少一块最重要的基石。

现在CovenantSQL测试网已经上线，但主网上线日期迟迟未定，而牛市已来，如果CovenantSQL错过这一轮机会，还能会出头之日吗？

成就一个NB产品/企业，无外乎以下三点：

• 先发优势
• 自己没有犯致命错误
• 有足够多的钱

先发优势：先发优势的关键是做品牌对应并养成用户习惯，例如：搜索百度一下、聊天微信、购物去淘宝，加密货币想到比特币，智能合约想到以太坊，去中心化存储是IPFS，那么CovenantSQL要争的位置就是去中心化的数据库。如果有另外一家提前在市值、知名度，特别是开发人员认知上做到了先入为主，CovenantSQL就失去了先机，先发优势则荡然无存。

自己不要犯致命错误：发展过程中不可能不犯错误，但别致命，以太坊the DAO攻击事件让以太坊差点胎死腹用。有时候看运气，有时候也是被竞争对手成就，最后活下来的往往是没犯致命错误的。

有足够的钱：市场有起就有落，有足够多的钱才能保证熊市时不必裸泳，才能保证能比竞争对手活得久。就如郭德纲相声如言：谁活得久，谁就是艺术家。

现在牛市已来，如果CovenantSQL纠结于合规问题，如果不能推动上主网，如果不储存足够过的弹药，机会稍纵即逝。

还是那句老话：选择比努力更重要。

选择趋势就是选择风险与机遇并存，否则何必做这个项目。

[Document] some Quickstart pages are outdated

Describe the bug
https://covenantsql.io/blog/quickstart-zh?lang=zh is outdated
https://testnet.covenantsql.io/quickstart is outdated

no cql-utils anymore after https://github.com/CovenantSQL/CovenantSQL/releases/tag/v0.5.0

Expected behavior
english version of https://developers.covenantsql.io/docs/quickstart, which is updated

testnet.covenantsql.io上facebook发完有检测不到的情况

facebook发完后，提示为 “你的帖子没有包含我们需要的 URL，请复制步骤1中的文字重新发送”。

我试着连发了几次，都不行。

Potential deadlock in testing

I will leave this issue here for further investigation. Check CI log here for details.

Error with fio test on cql-fuse

With FUSE support is awesome, but some bugs here

1. mount CovenantFS with
   ./cql-fuse -config conf/config.yaml -dsn covenantsql://0c07e3f6d8a57511d70955a3cfd7311895ab05ad5d7303279bf78dc2df40c17c -mount ./mnt
2. then cd to the ./mnt dir, run fio
   fio --loops=2 --size=100m --filename=./fiotest.tmp --stonewall --name=Seqread --bs=1m --rw=read --name=Seqwrite --bs=1m --rw=write --name=4krandread --bs=4k --rw=randread --name=4krandwrite --bs=4k --rw=randwrite

fio test Seqread failed and produced these logs:

ERRO[0019] call RPC failed                               caller="rpcutil.go:103 rpc.(*PersistentCaller).Call" error="too many SQL variables" rpc=DBS.Query stack="[#0 rpc.(*PersistentCaller)[email protected]:103      #1 client.(*conn)[email protected]:352      #2 client.(*conn)[email protected]:251      #3 database/sql.(*Tx)[email protected]:2007      #4 database/[email protected]:3074      #5 database/sql.(*Tx)[email protected]:2006      #6 [email protected]:43      #7 [email protected]:35      #8 main.(*Node)[email protected]:184      #9 vendor/bazil.org/fuse/fs.(*Server)[email protected]:945     ]"
2018/11/19 16:25:02 too many SQL variables

Pub Sub so we can see the CUD events occuring on the SQLite DB

Is your feature request related to a problem? Please describe.

Copy of Original of Email:
"
i am trying to work out if the system can handle reads and writes for a Postresql db.
For handling writes, you normally need to be constantly parsing the WAL. I can only see a SQLite implemenation of that.
I guess you use the SQLite only as a proxy for all the other databases.
My main aim with your system is to enable us to have many databases and to validate what is coming out of them.
on the read side for queries i am curious if you allow pass through, so clients can make a SQL query and its all just passed through magically to the database.
Its a bit hard with the currnt code to work that out.
the Test Net in docker and linearizability is pretty smart. Reminds me of "Call me Maybe" DB testing.
"

---

To build larger systems on top, you need to react to changes in the system.
For example maybe you want to build some microservices, and they need to know when a CUD ( Create, Update, Delete) occurs on the underlying data so that they can be triggered to do whatever they need to do.

Describe the solution you'd like
A golang based solution.
A simple JSON stream.
We spoke on Gitter about Redis, but it would also be good if its not tied to that because these days there are so many other ways of doing pub sub, like NATS, Kafka, etc.

Describe alternatives you've considered
Well there are no other ways other than polling form the top down into the DB and that is just not going to scale.

Additional context
We spoke on gitter about this use case and feature request, so the context is there.
Cant see a way to get an anchor link into the part of gitter where we discussed this, so just have to put the link to the room here: https://gitter.im/CovenantSQL/CovenantSQL

2 Server Documentation

I am trying to figure out how to bring online 2 servers in to the "cluster" so I have redundancy of the miners and other components

I already have 1 Host running all the Private Deploy and pushed the container to a private docker repository

Now I am trying to figure out what "minimum" I need to bring online a second server.

I suspect the steps are close but I am pretty sure I copied too much information from Server1

Server 1: Follow private deploy process
Server 1: Change the CovenantSql/test/.../config.yml to change 127.0.0.1 to IP of host
Server 1: Push created image to private repo
Server 2: Pull image from private repo
?Server 2: rsync ConvenantSql/ directory from Server 1 (I suspect this is my problem)
?Server 2: go to ConventSql directory and run the make start

Server 2 starts and when I connect to the console "csq console......" the existing DB is there but the blockchain between the 2 servers seems to be different as I add new data

I know I am screwing it up by copying too much data from Server1 to server2.

So what do I need to do to "Initialize" a second host. With the minimum amount of "files"?

Wanted: who is using CovenantSQL

Who is Using CQL

First of all, thanks sincerely for constantly using and supporting CQL. We will try our best to keep CQL better, and keep growing community.

The purpose of this issue

• We’d like to listen to the community to make CQL better.
• We want to attract more people to contribute to CQL.
• We're willing to learn more CQL use scenarios in the field for better planning.

What we expect from you

Pls. submit a comment in this issue to include the following information:

• your company, school or organization.
• your city and country.
• your contact info: blog, email, twitter (at least one，If is not convenient to open your contact information, please send me separately).
• for what business scenario do you use CQL.

You can refer to the following sample answer for the format:

* Orgnizatioin: Amazon
* Location: Beijing, China
* Contact: abc@company
* Purpose: Data Sharing、Data onChain etc.

Action now，build CQL

If you are still concerning about using CQL in production, you can join us on Gitter We will track CQL status from your side by the information you provided above, and assist you join the community to build the CQL together.

Thanks again for your participation !
CQL Team

谁在使用 CQL

首先诚挚地感谢每一位持续关注并使用 CQL 的朋友。我们会持续投入，力图把 CQL 变得更好，把 CQL 社区和生态变得更加繁荣。

此 Issue 的目的

聆听社区的声音，让 CQL 变得更好
吸引更多的人参与贡献
更多的了解 CQL 的实际使用场景，以方便下一步的规划

我们期待您能提供

在此提交一条评论, 评论内容包括:

• 您所在公司、学校或组织
• 您所在的城市、国家
• 您的联系方式: 微博、邮箱、微信 (至少一个，如果不方便公开您的联系信息，请单独发送给我)
• 您将 CQL 用于哪些业务场景

您可以参考下面的样例来提供您的信息：

* 组织：xx 公司
* 地点：**北京
* 联系方式：abc@company
* 业务场景：数据共享联盟、数据上链、数据溯源 等

开始行动，使用 CQL

如果您还对 CQL 的开发和生产上线有问题，您可以通过扫描以下微信加入我们，我们会根据您上面提供的信息来跟踪您的使用情况，辅助您加入社区共建。

再次感谢你的参与!!!
CQL 团队

I have developed an Excel add-in for this project

First of all, Thank you guys for offering me such a good SQL Database to use.

I stored some macro economy & finance data in this database, and developed a excel add-in to browser massive data & export it easier.

Here is my work's source code: https://github.com/melancholiaforever/CQL_Excel

if u guys want other add-in such as MATLAB or R, I may give a hand.

Command cqld -version failed without -config parameter

Why should I add config file for showing version?

https://covenantsql.io/blog/quickstart-zh 文档不是最新的

https://covenantsql.io/blog/quickstart-zh

基于微信wcdb的中文全文搜索

基于微信wcdb的中文全文搜索
https://github.com/Tencent/wcdb

[BUG] Compiling with go 1.13.4 on macOS Catalina leads to segfault

To Reproduce

1. Build with go 1.13.4 on macOS
2. Run cql generate:

==> /usr/local/Cellar/cql/0.8.0/bin/cql generate /private/tmp/cql-test-20191102-86418-1intxwf/confgen
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb01dfacedebac1e pc=0x7fff68322476]

runtime stack:
runtime.throw(0x4d70a61, 0x2a)
	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/panic.go:774 +0x72
runtime.sigpanic()
	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/signal_unix.go:378 +0x47c

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x4acd2c0, 0xc0000efdf8, 0x4d7ce82)
	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/cgocall.go:128 +0x5b fp=0xc0000efdc8 sp=0xc0000efd90 pc=0x4004beb
github.com/CovenantSQL/CovenantSQL/crypto/secp256k1._Cfunc_secp256k1_context_create_sign_verify(0x0)
	_cgo_gotypes.go:170 +0x4a fp=0xc0000efdf8 sp=0xc0000efdc8 pc=0x43ba70a
github.com/CovenantSQL/CovenantSQL/crypto/secp256k1.init.1()
	/private/tmp/cql-20191102-84097-c1dwy1/CovenantSQL-0.8.0/src/github.com/CovenantSQL/CovenantSQL/crypto/secp256k1/secp256.go:38 +0x22 fp=0xc0000efe10 sp=0xc0000efdf8 pc=0x43bd202
runtime.doInit(0x59ccec0)
	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/proc.go:5222 +0x8a fp=0xc0000efe40 sp=0xc0000efe10 pc=0x403e9fa
runtime.doInit(0x59d0420)
	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/proc.go:5217 +0x57 fp=0xc0000efe70 sp=0xc0000efe40 pc=0x403e9c7
runtime.doInit(0x59d2860)
	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/proc.go:5217 +0x57 fp=0xc0000efea0 sp=0xc0000efe70 pc=0x403e9c7
runtime.doInit(0x59d0ee0)
	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/proc.go:5217 +0x57 fp=0xc0000efed0 sp=0xc0000efea0 pc=0x403e9c7
runtime.doInit(0x59ccf80)
	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/proc.go:5217 +0x57 fp=0xc0000eff00 sp=0xc0000efed0 pc=0x403e9c7
runtime.doInit(0x59d7500)
	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/proc.go:5217 +0x57 fp=0xc0000eff30 sp=0xc0000eff00 pc=0x403e9c7
runtime.doInit(0x59cd280)
	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/proc.go:5217 +0x57 fp=0xc0000eff60 sp=0xc0000eff30 pc=0x403e9c7
runtime.main()
	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/proc.go:190 +0x1da fp=0xc0000effe0 sp=0xc0000eff60 pc=0x403268a
runtime.goexit()
	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000effe8 sp=0xc0000effe0 pc=0x405ef21

Desktop (please complete the following information):

• OS: macOS Catalina (10.15)
• Go Version: 1.13.4
• Cql Version: 0.8.0

[BUG] connection failed:

$ cql create -config=~/.cql/testnet-conf/config.yaml     -db-node 1 -wait-tx-confirm
INFO[0000] Geting bp address from dns: bp02.testnet.gridb.io
INFO[0001] Self register to blockproducer: 00000000000589366268c274fdc11ec8bdb17e668d2f619555a2e9c1a29c91d8
ERROR init covenantsql client failed                error="call DHT.Ping failed: dial to node 00000000000589366268c274fdc11ec8bdb17e668d2f619555a2e9c1a29c91d8 failed: dialing new session connection failed: connect to node bp00.cn.gridb.io:7777 failed: dial tcp 52.83.254.84:7777: i/o timeout"

Screenshots

Desktop (please complete the following information):

• OS: Ubuntu
• Docker
• Go Version
• Cql Version : cql HEAD-1ee09f3-20190807094154 linux amd64 go1.12.6

cql 增加一个 update 更新自己的命令（参考nvm）

cql 增加一个 update 更新自己的命令（参考nvm）https://github.com/nvm-sh/nvm

Private Key Format should be human readable

Background

CovenantSQL Private Key is encrypted with user inputted master key with a key hash suffix but stored in binary. It's not suitable for copy & paste or backup.

Goal

Private Key serialization and format can be identical with Bitcoin：

1. https://en.bitcoin.it/wiki/Private_key
2. https://en.bitcoin.it/wiki/Mini_private_key_format

Panic while docker stop miner.

time="2018-11-20T07:04:28Z" level=info msg="kayak leader apply" dc=205724 dq=826 eq=1578 fc=2358469 fp=9535526 fr=0 lc=251642 lp=7597386 r=534948 t=19745427
time="2018-11-20T07:04:28Z" level=info msg="kayak leader apply" dc=7554366 dq=719 eq=1008 fc=8217140 fp=3894101 fr=0 lc=7592117 lp=494967 r=534950 t=20200052
time="2018-11-20T07:04:28Z" level=info msg="miner stopped"
time="2018-11-20T07:04:28Z" level=info msg="Stopping Server Loop"
time="2018-11-20T07:04:28Z" level=panic msg="WRITE LOG FAILED: wal is closed" caller="runtime.go:763 kayak.(*Runtime).newLog" stack="[#0 kayak.(*Runtime)[email protected]:763      #1 kayak.(*Runtime)[email protected]:457      #2 kayak.(*Runtime)[email protected]:319      #3 worker.(*Database)[email protected]:323      #4 worker.(*Database)[email protected]:222      #5 worker.(*DBMS)[email protected]:272      #6 worker.(*DBMSRPCService).Query@dbms_rpc.go:75      #7 runtime.call64@asm_amd64.s:523      #8 [email protected]:447     ]"
time="2018-11-20T07:04:28Z" level=info msg="kayak leader apply" dq=0 eq=0 fc=0 fp=0 fr=0 lc=0 lr=0 r=0
panic: (*logrus.Entry) (0xf036c0,0xc01c5fd140)

goroutine 1469237 [running]:
github.com/CovenantSQL/CovenantSQL/vendor/github.com/sirupsen/logrus.Entry.log(0xc00009a120, 0xc00485f500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/CovenantSQL/CovenantSQL/vendor/github.com/sirupsen/logrus/entry.go:216 +0x2cf
github.com/CovenantSQL/CovenantSQL/vendor/github.com/sirupsen/logrus.(*Entry).Panic(0xc0194cecc0, 0xc02449d5d0, 0x1, 0x1)
	/go/src/github.com/CovenantSQL/CovenantSQL/vendor/github.com/sirupsen/logrus/entry.go:290 +0xab
github.com/CovenantSQL/CovenantSQL/vendor/github.com/sirupsen/logrus.(*Entry).Panicf(0xc0194cecc0, 0xf26313, 0x14, 0xc02449d708, 0x1, 0x1)
	/go/src/github.com/CovenantSQL/CovenantSQL/vendor/github.com/sirupsen/logrus/entry.go:344 +0xed
github.com/CovenantSQL/CovenantSQL/vendor/github.com/sirupsen/logrus.(*Logger).Panicf(0xc00009a120, 0xf26313, 0x14, 0xc02449d708, 0x1, 0x1)
	/go/src/github.com/CovenantSQL/CovenantSQL/vendor/github.com/sirupsen/logrus/logger.go:200 +0x85
github.com/CovenantSQL/CovenantSQL/vendor/github.com/sirupsen/logrus.Panicf(0xf26313, 0x14, 0xc02449d708, 0x1, 0x1)
	/go/src/github.com/CovenantSQL/CovenantSQL/vendor/github.com/sirupsen/logrus/exported.go:168 +0x5f
github.com/CovenantSQL/CovenantSQL/utils/log.Fatalf(0xf26313, 0x14, 0xc02449d708, 0x1, 0x1)
	/go/src/github.com/CovenantSQL/CovenantSQL/utils/log/logwrapper.go:317 +0x53
github.com/CovenantSQL/CovenantSQL/kayak.(*Runtime).newLog(0xc0000662c0, 0xb20000, 0xc031cc9c00, 0x5ea, 0xa88, 0x0, 0x0, 0x11ac660f4df)
	/go/src/github.com/CovenantSQL/CovenantSQL/kayak/runtime.go:763 +0x148
github.com/CovenantSQL/CovenantSQL/kayak.(*Runtime).leaderLogPrepare(0xc0000662c0, 0xc031cc9c00, 0x5ea, 0xa88, 0x5ea, 0xa88, 0x0)
	/go/src/github.com/CovenantSQL/CovenantSQL/kayak/runtime.go:457 +0x50
github.com/CovenantSQL/CovenantSQL/kayak.(*Runtime).Apply(0xc0000662c0, 0x11404e0, 0xc000096060, 0xe98140, 0xc031e63e00, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/CovenantSQL/CovenantSQL/kayak/runtime.go:319 +0x321
github.com/CovenantSQL/CovenantSQL/worker.(*Database).writeQuery(0xc0001f62d0, 0xc031e63e00, 0x0, 0x0, 0x0)
	/go/src/github.com/CovenantSQL/CovenantSQL/worker/db.go:323 +0x17f
github.com/CovenantSQL/CovenantSQL/worker.(*Database).Query(0xc0001f62d0, 0xc031e63e00, 0x40, 0xc0001f62d0, 0x1)
	/go/src/github.com/CovenantSQL/CovenantSQL/worker/db.go:222 +0x49
github.com/CovenantSQL/CovenantSQL/worker.(*DBMS).Query(0xc00001e410, 0xc031e63e00, 0x40, 0xd32aa17dd4e2d01, 0xc0372ee240)
	/go/src/github.com/CovenantSQL/CovenantSQL/worker/dbms.go:272 +0x68
github.com/CovenantSQL/CovenantSQL/worker.(*DBMSRPCService).Query(0xc00000e228, 0xc031e63e00, 0xc008786b60, 0x0, 0x0)
	/go/src/github.com/CovenantSQL/CovenantSQL/worker/dbms_rpc.go:75 +0x1db
reflect.Value.call(0xc0000692c0, 0xc00000e2a8, 0x13, 0xf18f02, 0x4, 0xc00032df18, 0x3, 0x3, 0xc037243740, 0x8, ...)
	/usr/local/go/src/reflect/value.go:447 +0x449
reflect.Value.Call(0xc0000692c0, 0xc00000e2a8, 0x13, 0xc000040f18, 0x3, 0x3, 0xe71200, 0xc00025b640, 0x1139580)
	/usr/local/go/src/reflect/value.go:308 +0xa4
net/rpc.(*service).call(0xc00009d840, 0xc00001e370, 0xc01f593be0, 0xc01f593bf0, 0xc000020c80, 0xc004762200, 0xe98140, 0xc031e63e00, 0x16, 0xe2d8e0, ...)
	/usr/local/go/src/net/rpc/server.go:384 +0x14e
created by net/rpc.(*Server).ServeCodec
	/usr/local/go/src/net/rpc/server.go:481 +0x47e

build error

Sending build context to Docker daemon 55.5 MB
Step 1/16 : FROM golang:1.10.3-stretch as builder
Error parsing reference: "golang:1.10.3-stretch as builder" is not a valid repository/tag: invalid reference format
make: *** [build] Error 1

Cannot receive tokens from testnet

Followed the quickstart generated wallet and verified my
twitter post but cannot get the tokens, the website is stuck on the last step. Can anyone help get the test tokens?

Support for MS Windows

Is your feature request related to a problem? Please describe.
I cannot compile CovenantSQL to run on MS Windows.

Describe the solution you'd like
Please provide a description what it requires building binaries.

Describe alternatives you've considered
None.

File can’t be copied because there isn’t enough free space

Cannot copy or move with finder files.

如何在安卓上整合CovenantSQL的教程（做一个进程服务引入？）

如何在安卓上整合CovenantSQL的教程（做一个进程服务引入？）

How do we use ConvenanSQL as an alternative to QLDB?

The documentation on using ConvenantSQL as an alternative to QLDB is scarce. Can anybody please elaborate on this?

truncate database cmd not found

co:0f1a6b81b80da80001b41d891a38ff50e6bd07cd9504377670848c5638b604ca=> TRUNCATE  test;
ERRO[0118] call RPC failed                               caller="rpcutil.go:107 rpc.(*PersistentCaller).Call" error="apply failed: execute at #0 failed: syntax error at position 9 near 'TRUNCATE'" rpc=DBS.Query stack="[#0 rpc.(*PersistentCaller)[email protected]:107      #1 client.(*conn)[email protected]:399      #2 client.(*conn)[email protected]:347      #3 client.(*conn)[email protected]:249      #4 database/[email protected]:31      #5 database/sql.(*DB)[email protected]:1468      #6 database/[email protected]:3076      #7 database/sql.(*DB)[email protected]:1463      #8 database/sql.(*DB)[email protected]:1448      #9 database/sql.(*DB)[email protected]:1426     ]"
error: covenantsql: apply failed: execute at #0 failed: syntax error at position 9 near 'TRUNCATE'

Master key should not be input with env or args

查看自己的钱包余额时，如果之前有输入master key，则提示失败。

错误提示如下：

./cql -config conf/config.yaml -get-balance

ERRO[0000] decrypt private key error caller="privatekeystore.go:50 crypto/kms.LoadPrivateKey"

INFO[0000] load private key failed: invalid PKCS#7 padding caller="privatekeystore.go:90 crypto/kms.InitLocalKeyPair"

ERRO[0000] unexpected error while loading private key: invalid PKCS#7 padding caller="privatekeystore.go:109 crypto/kms.InitLocalKeyPair"

ERRO[0000] init covenantsql client failed: invalid PKCS#7 padding caller="main.go:196 main.main"

Can not use double quote in table name

As I know, double quote in table name is SQL92 standard, why I got an error from CovenantSQL?

Could not run in docker based alpine image

Maybe missing libc, could you build static instead?

Quickstart cli document 404

CLI README document url not found in the quickstart.
In addition, cql command needs master key (with -password master-key) if the key is not empty by default.This should be mentioned in the document.

Get TestNet token HERE

Please paste your wallet address here. We will transfer TestNet token for you.

• TestNet Howto in English: https://developers.covenantsql.io/docs/en/quickstart
• TestNet中文指引: https://developers.covenantsql.io/docs/zh-CN/quickstart

cql cannot parse covenantsql url with params

Create database returns url with params:

$ ./bin/cql -config conf/config.yaml -create 1
INFO[0000] the newly created database is: "covenantsql://0f1a6b81b80da80001b41d891a38ff50e6bd07cd9504377670848c5638b604ca?use_follower=false&use_leader=true"

But cql cmd cannot parse database url with params

$ ./bin/cql -config conf/config.yaml -dsn "covenantsql://0f1a6b81b80da80001b41d891a38ff50e6bd07cd9504377670848c5638b604ca?use_follower=false&use_leader=true"
INFO[0000] connecting to "covenantsql://0f1a6b81b80da80001b41d891a38ff50e6bd07cd9504377670848c5638b604ca%3Fuse_follower=false&use_leader=true?use_follower=false&use_leader=true"
ERRO[0000] run cli error                                 caller="main.go:290 main.main" error="covenantsql: parse covenantsql://0f1a6b81b80da80001b41d891a38ff50e6bd07cd9504377670848c5638b604ca%3Fuse_follower=false&use_leader=true?use_follower=false&use_leader=true: invalid URL escape \"%3F\"" stack="[#0 [email protected]:290      #1 [email protected]:

类似flutter sqflite 的插件（方便开发，安卓版本）

类似flutter sqflite 的插件（方便开发，安卓版本）
sqflite | Flutter Package - Dart Pub
https://pub.dartlang.org/packages/sqflite -

建议缩短数据库地址 base62编码下 ，协议名称用 cql:// ， 不然命令行中删除修改很麻烦，要按很久的删除键

建议缩短数据库地址 base62编码下 ，协议名称用 cql:// ， 不然命令行中删除修改很麻烦，要按很久的删除键

get stable coin balance failed

Hey, im simply following the instructions found on https://testnet.covenantsql.io/quickstart, but i am experiencing "get stable coin balance failed"

INFO[0006] read full failed error="read tcp 192.168.100.207:51094->39.98.191.190:7777:
read: connection reset by peer"
ERRO[0006] get stable coin balance failed caller="main.go:215 main.main" error="broken pipe" stack="[#0 [email protected]:215 #1 [email protected]:198 #2 runtime.goexit@asm_amd64.s:2361 ]"

Is this simply a firewall issue or what?

Any plan to NodeJS support?

[BUG] Problem getting the client package

Describe the bug
When I type go get "github.com/CovenantSQL/CovenantSQL/client" and after that go get "github.com/CovenantSQL/CovenantSQL/cmd/cql" I get the same terminal prompt:
go/src/github.com/CovenantSQL/CovenantSQL/blockproducer/interfaces/transaction_wrapper.go:169:20: decodeDriver.TryDecodeAsNil undefined (type codec.genHelperDecDriver has no field or method TryDecodeAsNil)
Afterwards, when importing either package I get the error:
../../go/src/github.com/CovenantSQL/CovenantSQL/blockproducer/interfaces/transaction_wrapper.go:169:20: decodeDriver.TryDecodeAsNil undefined (type codec.genHelperDecDriver has no field or method TryDecodeAsNil)
This happens for WriteArrayElem, ReadArrayElem and TryDecodeAsNil
How can I fix this? Thanks!

To Reproduce
Commands to reproduce the behavior:

1. In the terminal type go get "github.com/CovenantSQL/CovenantSQL/client"
2. You will get the first behaviour described
3. In any Go IDE execute a basic script importing the package, for example:
   package main

import (
"fmt"
"github.com/CovenantSQL/CovenantSQL/client"
)
func main() {
fmt.Println("Hello World")
}
4. You will get the error described

生成的钱包地址无效

ERRO[0004] get Particle balance failed error="no such token balance"

另外，申请水龙头是报错钱包地址无效

Static build get some warnings about glibc

dlopen getaddrinfo getgrouplist getgrgid_r getgrnam_r getpwnam_r getpwuid_r
Temporally change running docker image to glibc version alpine in this commit ace6fa5

ThunderDB has been renamed to CovenantSQL

改个名，改的更low了，Covenant太长，sql为什么不是DB

New release with go module support

Please make a new release with go module support so that brew can use it for releases.

Homebrew/homebrew-core#47627

Stuck in 2pc inconsistent state error

I've used the http api to save some data, and i've created a database success.
But when I tried to insert a bulk sql request for more than 3000 line, like 'insert or replace into xxx values(xxx)', I got {'data': None, 'status': 'process log: invalid request supplied', 'success': False} error.
I tried again, got {'data': None, 'status': 'process log: twopc: inconsistent state, currently in tx: conn = 1491013221004001315, seq = 1, time = 1536228051350372242', 'success': False} error instead, and this database could never accecpt any command, but only this error returned.

Please help me check what's wrong and how to get it right. My database id is 053d0bb19637ffc7b4a94e3c79cc71b67a768813b09e4b67f1d6159902754a8b

[BUG]

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Commands to reproduce the behavior:

1. cd to '...'
2. run 'cql ...'
3. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. Linux/MacOS]
• Docker [e.g. chrome, safari]
• Go Version [e.g. 22]
• Cql Version

Config

• Your cql config, Typically "~/.cql/config.yaml"

Additional context
Add any other context about the problem here.

Where can I find covenantsql.io/covenantsql_adapter

Hi, where can I find this docker image covenantsql.io/covenantsql_adapter?

It's not on docker hub, and there is no instruction to build it from source code.

[Usecase Question]能把CovenantSQL用作简单的DAO(分布自治组织)数据库吗?

能把CovenantSQL用作简单的DAO(分布自治组织)数据库吗?

比如 一群 音乐发烧友 想维护一份共同口味的 音乐列表
(前提是每个人 已经有了长期在线VPS服务器, 比如我们是程序员音乐发烧友群),

首先第1个成员安装了CovenantSQL, 在成员表tb_members里把自己的公钥insert进去,
然后第2个成员把自己的application/proposal发送到某个临时的DHT表里,
第1个成员可以投赞成或反对票, 超过50%已有成员赞成, 新成员就从DHT表移动到正式的tb_members表里,
投票insert/delete 音乐表 tb_musics 也用类似的简单多数表决法
然后非成员用户可以 只读这个数据库的 tb_members 和 tb_musics,
核心需求就是一旦这个 音乐爱好者数据库 建立起来了, 就没有任何个体成员能破坏掉整个数据库

看了CovenantSQL的testnet文档 没太看明白怎么实现我这个用例,
另外担心mainnet收费, 因为我们这个程序员音乐发烧友群 每人都已有VPS服务器了.

能基于这个简单的用例 写份 CovenantSQL的教程么?
或者实现我这种简单的DAO(分布自治组织)用例 有更简单的办法?