datafuselabs / databend-go Goto Github PK
View Code? Open in Web Editor NEWGolang driver for databend cloud
License: Apache License 2.0
Golang driver for databend cloud
License: Apache License 2.0
i use ng for databend-query
go driver SDK is 0.4.8
i use db.Exec("INSERT INTO testd VALUES('111.111.111.111',?)",x) ,the ng return 2 logs , post and get was success, post url and get url go to same node.
but
my code like this:
scope, err := db.Begin()
batch, err := scope.Prepare(fmt.Sprintf("INSERT INTO %s VALUES", "testd"))
for i := 0; i < 10; i++ {
, err = batch.Exec(i,x)
}
scope.Commit()
the ng return logs post 200 , get 404 insert failed :
1703745676.977 "10.49.2.134" 422 "POST" "10.49.2.187" "/v1/query" "" "10.49.2.187" 8000 "/v1/query" "-" "HTTP/1.1" 1557 1239 200 0.008 "-" "databend-go/0.4.2" "-" "-" "-" "-" "10.49.2.57:8000" "-" [28/Dec/2023:06:41:16 +0000]
1703745676.983 "10.49.2.134" 323 "GET" "10.49.2.187" "/v1/query/685870bc-8e5a-4098-8715-793f70d08a10/final" "_" "10.49.2.187" 8000 "/v1/query/685870bc-8e5a-4098-8715-793f70d08a10/final" "-" "HTTP/1.1" 241 118 404 0.005 "-" "databend-go/0.4.2" "-" "-" "-" "-" "10.49.2.207:8000" "-" [28/Dec/2023:06:41:16 +0000]
databend-query log timeout 60s
the post url and get get url not go to same node. so is a bug?
if compress is true then compress the file to upload.
)panic: failed to scan result &sql.Row{err:(*fmt.wrapError)(0x1400009e4e0), rows:(*sql.Rows)(nil)}: newTextRows: failed to create a data parser for the type 'VariantObject': type VariantObject is not supported
on executing PRESIGN UPLOAD @~/test.csv
the result of PRESIGN contains one JSON field like:
Now the go driver use databend cloud API, we should support clickhouse http protocol.
The document showed a not working batch insert example
It should be like the code in test
databend-go/tests/main_test.go
Line 104 in 0aa66ff
When executing queries on a large dataset, we're encountering errors.
failed to query page: All attempts fail:
#1: 404 {"error":{"code":"404","message":"route error: query 8e7ba07b-31f9-44c6-8acb-afebc6a94419 SHOULD be on server L0VVVZjokcB7K2ot8xQtW, but current server is RtMjqHG7wzwNB5P1KWIpo6, which started at 2024-07-15T23:34:17.851+00:00(100326.88 secs ago)"}}. please check your arguments
What could be causing these errors with large datasets?
Are there any recommended configurations or best practices for handling large data queries in this setup?
Should we consider scaling our resources, and if so, how?
We appreciate any insights or suggestions you can provide to help us resolve this issue and optimize our Databend queries for large datasets in our Kubernetes environment. Thank you in advance for your assistance!
Should make a test as the following steps:
insert
, update
, delete
concurrentlyThen there should not has exceed max session error.
currently we support Open with a DSN, however sometimes we wanna pass some hooks into the driver.
a Config struct might better on doing this job than a string DSN
db.QueryRow
has GET" "10.49.2.4" "/v1/query/f29bec11-7128-4fde-abc7-013e965d5f54/final
error when query through nginx.
when I upgrade github.com/datafuselabs/databend-go v0.5.3 to github.com/datafuselabs/databend-go v0.5.9
scope, err := db.Begin()
batch, err := scope.Prepare(fmt.Sprintf("INSERT INTO %s VALUES", "testd"))
m := make([]interface{},2)
m[0]="zzz"
m[1]=x
for i := 0; i < 2; i++ {
_, err = batch.Exec(m...)
fmt.Println(err,i)
}
err = scope.Commit()
this code exec error
The PR: #78
add some test around session level role setting in golang driver, which behave as expected, while it did not change unset related issues but there are some unexpected thing occurs.
I think it is related to the PR
datafuselabs/databend#13062
Issue error
https://github.com/datafuselabs/databend-go/actions/runs/6765657226/job/18385601947?pr=78
stage := &godatabend.StageLocation{
Name: "fluentbit",
Path: fmt.Sprintf("fluentbit/%d-%s", time.Now().Unix(), filepath.Base(filename)),
}
fmt.Println(stage.Name)
ctxd, cannel := context.WithCancel(context.Background())
defer cannel()
err = apiClient.UploadToStage(ctxd,stage, input,size)
package main
import (
"database/sql"
"fmt"
_ "github.com/datafuselabs/databend-go"
)
func main() {
db, err := sql.Open("databend", "http://databend:databend@localhost:8000")
if err != nil {
fmt.Println(err)
return
}
scope, err := db.Begin()
if err != nil {
return
}
batch, err := scope.Prepare(fmt.Sprintf("INSERT INTO %s VALUES", "gottt"))
for i := 0; i < 10; i++ {
name := fmt.Sprintf("name%d", i)
if i == 9 {
name = ""
}
_, err = batch.Exec(i, name, i*2)
}
err = scope.Commit()
}
when id = 9, the name is NULL
:
But the name should be "" (empty string).
When execute batch insert, the csv file generated is:
0,name0,0
1,name1,2
2,name2,4
3,name3,6
4,name4,8
5,name5,10
6,name6,12
7,name7,14
8,name8,16
9,,18
databend kernet will handle the empty string as NULL
when copy into.
you may get a ProvisionWarehouseTimeout on cloud resource exhausted. This error is safe to retry, because the query is not actually sent to the warehouse.
with this error get handled, the sdk could be more robust on access cloud warehouse from databend cloud.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.