arana-db / arana Goto Github PK

What would you like to be added:
select * from student where uid between 1 and 10 order by created_at

Why is this needed:

What would you like to be added:

Why is this needed:

add unit test coverage Github action for Pull Request.

What happened:

no readme

What you expected to happen:

readme

Anything else we need to know?:

tell us what it is and how to use it.

What would you like to be added:

Why is this needed:

What would you like to be added:

Why is this needed:

Integrate JavaScript for sharding rule.

What would you like to be added:

Why is this needed:

What happened:
default tag not work when parse config
What you expected to happen:
default tag should work when var is empty
How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Welcome to use arana. To know who is using it, pls append your org info as follow,

• Organization：Alibaba (Required)
• Location: Hangzhou, Zhejiang, China（Required）
• Contact: [email protected] (Optional)
• Purpose：used as our fundamental network transport lib (Required)
• Registry: zookeeper（Required）
• Version: 1.5.5（Required）
• Com/Org Logo：(Optional)
  Thanks again for your participation!

欢迎使用 arana，首先感谢你的使用，其次您可以参考下面的样例来提供您的信息以收集下使用场景：

* 组织：阿里巴巴（Required）
* 地点：**杭州（Required）
* 联系方式：[email protected]（Optional）
* 场景：阿里内部服务化改造（Required）
* 注册中心: zookeeper（Optional）
* 版本: 1.5.5（Optional）
* 公司图标：（Optional）

再次感谢你的参与!!!

What would you like to be added:

Why is this needed:

support count(*), count(field_name), max, min and so on MySQL aggregate function.

What would you like to be added:

Integrated go-mysql-server as embed MySQL test server.

https://github.com/dolthub/go-mysql-server

Why is this needed:

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Start server refer this PR comment: #90 .

execute:select * from student.

2022-03-25T11:43:14.767+0800    DEBUG   executor/redirect.go:135        ComQuery: select connection_id()
2022-03-25T11:43:14.773+0800    DEBUG   executor/redirect.go:135        ComQuery: select * from student
2022-03-25T11:43:14.773+0800    DEBUG   optimize/optimizer.go:149       compute shards: result=["*"], isFullScan=false
2022-03-25T11:43:14.773+0800    ERROR   mysql/server.go:161     mysql_server caught panic:
implement me
github.com/arana-db/arana/pkg/mysql.(*Listener).handle.func1
        /Users/dongzonglei/source_code/Github/arana/pkg/mysql/server.go:161
runtime.gopanic
        /usr/local/go/src/runtime/panic.go:965
github.com/arana-db/arana/pkg/runtime/plan.UnionPlan.ExecIn
        /Users/dongzonglei/source_code/Github/arana/pkg/runtime/plan/union.go:39
github.com/arana-db/arana/pkg/runtime.(*defaultRuntime).Execute
        /Users/dongzonglei/source_code/Github/arana/pkg/runtime/runtime.go:545
github.com/arana-db/arana/pkg/executor.(*RedirectExecutor).ExecutorComQuery
        /Users/dongzonglei/source_code/Github/arana/pkg/executor/redirect.go:180
github.com/arana-db/arana/pkg/mysql.(*Listener).ExecuteCommand.func1
        /Users/dongzonglei/source_code/Github/arana/pkg/mysql/server.go:558
github.com/arana-db/arana/pkg/mysql.(*Listener).ExecuteCommand
        /Users/dongzonglei/source_code/Github/arana/pkg/mysql/server.go:589
github.com/arana-db/arana/pkg/mysql.(*Listener).handle
        /Users/dongzonglei/source_code/Github/arana/pkg/mysql/server.go:206
2022-03-25T11:43:14.775+0800    DEBUG   executor/redirect.go:135        ComQuery: select connection_id()

What would you like to be added:

Why is this needed:

Support js for parse config topology.

Cannot connect the arana server over mycli

The mycli is a very popular command-line tool, we should support it.

What happened:

1. start server
2. connect with mycli -h127.0.0.1 -udksl employees -p123456 -P13306

What you expected to happen: Should connect successfully

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

2021-12-30T16:22:00.085+0800    DEBUG   executor/redirect.go:137        ComQuery: SET NAMES utf8mb4
2021-12-30T16:30:24.726+0800    DEBUG   executor/redirect.go:137        ComQuery: SET NAMES utf8mb4
^[[20~2021-12-30T16:30:37.444+0800      DEBUG   executor/redirect.go:137        ComQuery: SET NAMES utf8mb4
2021-12-30T16:30:46.986+0800    DEBUG   executor/redirect.go:137        ComQuery: SET NAMES utf8mb4
2021-12-30T16:31:01.774+0800    DEBUG   executor/redirect.go:137        ComQuery: SET NAMES utf8mb4
2021-12-30T16:31:04.551+0800    DEBUG   executor/redirect.go:137        ComQuery: SET NAMES utf8mb4
2021-12-30T16:31:04.553+0800    INFO    executor/redirect.go:193        SELECT emp_no, birth_date, first_name, last_name, gender, hire_date FROM employees 
                WHERE emp_no = 100001
2021-12-30T16:31:04.554+0800    ERROR   mysql/server.go:156     mysql_server caught panic:
trying to call recycleReadPacket while currentEphemeralPolicy is 0
github.com/dubbogo/arana/pkg/mysql.(*Listener).handle.func1
        /Users/dongzonglei/source_code/Github/arana/pkg/mysql/server.go:156
runtime.gopanic
        /usr/local/go/src/runtime/panic.go:971
github.com/dubbogo/arana/pkg/mysql.(*Conn).recycleReadPacket
        /Users/dongzonglei/source_code/Github/arana/pkg/mysql/conn.go:345
github.com/dubbogo/arana/pkg/mysql.(*Listener).handle
        /Users/dongzonglei/source_code/Github/arana/pkg/mysql/server.go:186

What would you like to be added:
support binary protocol from arana to physics db

What would you like to be added:
The statement of SHOW TABLES should only list those logical tables and non-sharding tables.
Please see the official manual, https://dev.mysql.com/doc/refman/5.7/en/show-tables.html

Why is this needed:
The tables in ARANA means 'logical tables' instead of 'physical tables', we need to handle it. For example, we have student_0000..student0031, the actual result should be student.

What happened:

当执行集成测试查询时，aranan会报错：
mysql_server caught panic:
trying to call recycleReadPacket while currentEphemeralPolicy is 0

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):
启动main函数，然后执行 test 包下面的 integration_test.go 文件的 TestSelect方法，即可复现问题

Anything else we need to know?:

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

1. Start server refer this PR comment: #90 .
2. Connect the server with mycli: mycli -h127.0.0.1 -P13306 -udksl employees -p123456
3. show databases; is OK.
4. use employees; is OK.
5. show tables; is not OK. error message: (1105, 'unknown error: no such logical database \x02employees')

What would you like to be added:

Why is this needed:

add newest TiDB SQL Parser to arana-db

suggest define a configuration item, configure the maxrows.

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

What happened: the SHOW DATABASES returns a column type of DECIMAL,

What you expected to happen: it should return VARCHAR.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

What would you like to be added:

Why is this needed:

Support MySQL Limit x, y

What would you like to be added:

Why is this needed:

What would you like to be added:

Why is this needed:

What would you like to be added:
Support SHOW VARIABLES.

Why is this needed:

What would you like to be added:
Need to return empty rows when query from information_schema.columns, because it will cause incorrect mycli behavior.

Why is this needed:

What would you like to be added:

Why is this needed:

Support MySQL "truncate table xxx"

将etcd 作为配置中心:
配置获取策略：
默认拉取etcd中的数据，当etcd 中不存在数据时，将配置文件的数据同步到etcd当中。
已实现功能：
etcd作为配置存储，更新，初始化配置文件到etcd 中。
还需要实现功能点：
存储到etcd中的key value 键值对的粒度根据情况再划分小一些；
接入etcd 的watch 机制 和加入对应的 key value 的version 进行相关版本匹配；
将此方案的配置替换现有实现中的配置加载策略

https://github.com/dubbogo/arana/blob/15aa18f61ae8cadf874d2ead860bd88bb5608d66/test/integration_test.go#L113

action mysql root account password is root, test is 123456, did not fail

ref: https://dev.mysql.com/worklog/task/?id=7766

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

What would you like to be added:
implement simple DELETE statement, for example: delete from student where uid = 1 limit 1.

Why is this needed:
It is a basic feature.

What would you like to be added:
Support table metadata for star expand, in case:
select * from person
->
select id, name from person

Why is this needed:

What would you like to be added:

Why is this needed:

var v = &struct {
	EmpNo     int       `gorm:"emp_no"`
	BirthDate time.Time `gorm:"birth_date"`
	FirstName string    `gorm:"first_name"`
	LastName  string    `gorm:"last_name"`
	Gender    string    `gorm:"gender"`
	HireDate  time.Time `gorm:"hire_date"`
}{}

db, _ := sql.Open("mysql", "dksl:123456@tcp(127.0.0.1:13306)/employees?timeout=1s&readTimeout=1s&writeTimeout=1s&parseTime=true&loc=Local&charset=utf8mb4,utf8") //连接数据库
defer db.Close()

row := db.QueryRow(`SELECT emp_no FROM employees`)
if err := row.Scan(&v.EmpNo); err != nil {
	panic(err)
}
fmt.Println(v.EmpNo)

运行结果：
panic: sql: no rows in result set

What happened:
表中有值，也给客户端返回结果了，但是客户端没解析出来

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Now, the project file License header format is:

//
// Licensed to Apache Software Foundation (ASF) under one or more contributor
// license agreements. See the NOTICE file distributed with
// this work for additional information regarding copyright
// ownership. Apache Software Foundation (ASF) licenses this file to you under
// the Apache License, Version 2.0 (the "License"); you may
// not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.
//

we want to fix it:

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

What would you like to be added:
The show databases statement should only list those databases in the same tenant.
Please see the official manual, https://dev.mysql.com/doc/refman/5.7/en/show-databases.html

Why is this needed:
The database in ARANA means 'logical database' instead of 'physical database', we need to handle it.

What would you like to be added:

Why is this needed:

https://github.com/testcontainers/testcontainers-go

https://www.testcontainers.org/

What would you like to be added:

• Use Prometheus go client to add more metrics.
• Use OpenTelemetry to add more trace.
• Add config.

Use Prometheus client-go and OpenTelemetry is because they are the most used now. :) Correct me if I am wrong, or if you have better choice, just comment.

Why is this needed:
Observability is very important for basic software architecture.There are three sections in this field.

What would you like to be added:

Why is this needed:

Now, unit test coverage for arana code is only 24%, we need add more unit test and increase unit test coverage.

func (t *DataSourceType) UnmarshalText(text []byte) error {
	if t == nil {
		return errors.New("can't unmarshal a nil *DataSourceRole")
	}
	if t.unmarshalText(bytes.ToLower(text)) {
		return fmt.Errorf("unrecognized protocal type: %q", text)
	}
	return nil
}

What happened:
支持的角色类型、协议类型，返回了error的错误
What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

if t.unmarshalText(bytes.ToLower(text)) {
		return fmt.Errorf("unrecognized protocal type: %q", text)
	}

改为

if !t.unmarshalText(bytes.ToLower(text)) {
		return fmt.Errorf("unrecognized protocal type: %q", text)
	}

Anything else we need to know?:

What would you like to be added:

Why is this needed:

Support uuid for distributed primary key.

What would you like to be added:
Handling login without database, for example:

$ mycli -h127.0.0.1 -P13306 -uarana -p123456

Why is this needed:

What would you like to be added:
Generate pk when executing INSERT:

1. register entrance
2. impls: snowflake/sequence table based/...

Why is this needed:

success

mysql -h192.168.200.161 -P13306 -udksl employees -p123456

error

mysql -h192.168.200.161 -P13306 -udksl -p123456

What would you like to be added:

Why is this needed:

Support Drop table SQL statement.

What happened:

dubbogo 积累的一些历史旧账

What you expected to happen:

多注意细节，多看看历史问题，多翻翻旧账，照亮来时路，少犯错。

What would you like to be added:

Why is this needed:

when user execute show databases, we should show the tenant's logic databases to user, and if the the SQL is the show databases where name=xxxxx with condition, we should filter the logic databases.

https://dev.mysql.com/doc/refman/8.0/en/show-databases.html

常规的请求应答模型，filter 机制只需要在代理层实现一个 filter chain，类似于高速公路上的收费站，每条通路上设置一个收费站，实现对请求的拦截。由于 DB 路由有 sharding 的需求，它的 filter 机制应该类似于水站，水源经过水站拦截后，分散成无数的水管，每个水管上安装有水阀。

需要两种类型的 filter，一种是 executor filter，一种是 connection filter。