Coder Social home page Coder Social logo

exasol-driver-go's Introduction

Exasol Go SQL Driver

Build Status Go Reference

Quality Gate Status

Maintainability Rating Bugs Code Smells Coverage

This repository contains a Go library for connection to the Exasol database.

This library uses the standard Golang SQL driver interface for easy use.

Prerequisites

To use the Exasol Go Driver you need an Exasol database in the latest 7.1 or 8 version. Older versions might work but are not supported.

Usage

Create Connection

With Exasol Config

We recommend using the provided builder to build a connection string. The builder ensures all values are escaped properly.

package main

import (
    "database/sql"
    "github.com/exasol/exasol-driver-go"
)

func main() {
    database, err := sql.Open("exasol", exasol.NewConfig("<username>", "<password>").
                                              Host("<host>").
                                              Port(8563).
                                              String())
    // ...
}

If you want to login via OpenID tokens use exasol.NewConfigWithRefreshToken("token") or exasol.NewConfigWithAccessToken("token"). See the documentation about how to configure OpenID authentication in Exasol. OpenID authentication is only supported with Exasol 7.1.x and later.

With Exasol DSN

You can also create a connection replacing the builder with a simple string:

package main

import (
    "database/sql"
    _ "github.com/exasol/exasol-driver-go"
)

func main() {
    database, err := sql.Open("exasol",
            "exa:<host>:<port>;user=<username>;password=<password>")
    // ...
}

If a value in the connection string contains a ; you need to escape it with \;. This ensures that the driver can parse the connection string as expected.

Execute Statement

result, err := exasol.Exec(`
    INSERT INTO CUSTOMERS
    (NAME, CITY)
    VALUES('Bob', 'Berlin');`)

Query Statement

rows, err := exasol.Query("SELECT * FROM CUSTOMERS")

Use Prepared Statements

preparedStatement, err := exasol.Prepare(`
    INSERT INTO CUSTOMERS
    (NAME, CITY)
    VALUES(?, ?)`)
result, err = preparedStatement.Exec("Bob", "Berlin")
preparedStatement, err := exasol.Prepare("SELECT * FROM CUSTOMERS WHERE NAME = ?")
rows, err := preparedStatement.Query("Bob")

Transaction Commit and Rollback

To control the transaction state manually, you need to disable autocommit (enabled by default):

database, err := sql.Open("exasol",
                "exa:<host>:<port>;user=<username>;password=<password>;autocommit=0")
// or
database, err := sql.Open("exasol", exasol.NewConfig("<username>", "<password>")
                                          .Port(<port>)
                                          .Host("<host>")
                                          .Autocommit(false)
                                          .String())

After that you can begin a transaction:

transaction, err := exasol.Begin()
result, err := transaction.Exec( ... )
result2, err := transaction.Exec( ... )

To commit a transaction use Commit():

err = transaction.Commit()

To rollback a transaction use Rollback():

err = transaction.Rollback()

Import Local CSV Files

Use the sql driver to load data from one or more CSV files into your Exasol Database. These files must be local to the machine where you execute the IMPORT statement.

Limitations:

  • Only import of CSV files is supported at the moment, FBV is not supported.
  • The SECURE option is not supported at the moment.
result, err := exasol.Exec(`
IMPORT INTO CUSTOMERS FROM LOCAL CSV FILE './testData/data.csv' FILE './testData/data_part2.csv'
  COLUMN SEPARATOR = ';' 
  ENCODING = 'UTF-8' 
  ROW SEPARATOR = 'LF'
`)

See also the usage notes about the file_src element for local files of the IMPORT statement.

Connection String

The golang Driver uses the following URL structure for Exasol:

exa:<host>[,<host_1>]...[,<host_n>]:<port>[;<prop_1>=<value_1>]...[;<prop_n>=<value_n>]

Host-Range-Syntax is supported (e.g. exasol1..3). A range like exasol1..exasol3 is not valid.

Supported Driver Properties

Property Value Default Description
autocommit 0=off, 1=on 1 Switch autocommit on or off.
clientname string Go client Tell the server the application name.
clientversion string Tell the server the version of the application.
compression 0=off, 1=on 0 Switch data compression on or off.
encryption 0=off, 1=on 1 Switch automatic encryption on or off.
validateservercertificate 0=off, 1=on 1 TLS certificate verification. Disable it if you want to use a self-signed or invalid certificate (server side).
certificatefingerprint string Expected fingerprint of the server's TLS certificate. See below for details.
fetchsize numeric, >0 128*1024 Amount of data in kB which should be obtained by Exasol during a fetch. The application can run out of memory if the value is too high.
password string Exasol password.
resultsetmaxrows numeric Set the max amount of rows in the result set.
schema string Exasol schema name.
user string Exasol username.

Configuring TLS

We recommend to always enable TLS encryption. This is on by default, but you can enable it explicitly via driver property encryption=1 or config.Encryption(true). Please note that starting with version 8, Exasol does not support unencrypted connections anymore, so you can't use encryption=0 or config.Encryption(false).

There are two driver properties that control how TLS certificates are verified: validateservercertificate and certificatefingerprint. You have these three options depending on your setup:

  • With validateservercertificate=1 (or config.ValidateServerCertificate(true)) the driver will return an error for any TLS errors (e.g. unknown certificate or invalid hostname).

    Use this when the database has a CA-signed certificate. This is the default behavior.

  • With validateservercertificate=1;certificatefingerprint=<fingerprint> (or config.ValidateServerCertificate(true).CertificateFingerprint("<fingerprint>")) you can specify the fingerprint (i.e. the SHA256 checksum) of the server's certificate.

    This is useful when the database has a self-signed certificate with invalid hostname but you still want to verify connecting to the correct host.

    Note: You can find the fingerprint by first specifying an invalid fingerprint and connecting to the database. The error will contain the actual fingerprint.

  • With validateservercertificate=0 (or config.ValidateServerCertificate(false)) the driver will ignore any TLS certificate errors.

    Use this if the server uses a self-signed certificate and you don't know the fingerprint. This is not recommended.

Configure Logging

Error Logger

By default the driver will log warnings and error messages to stderr. You can configure a custom error logger with

logger.SetLogger(log.New(os.Stderr, "[exasol] ", log.LstdFlags|log.Lshortfile))

Trace Logger

By default the driver does not log any trace or debug messages. To investigate problems you can configure a custom trace logger with

logger.SetTraceLogger(log.New(os.Stderr, "[exasol-trace] ", log.LstdFlags|log.Lshortfile))

You can deactivate trace logging with

logger.SetTraceLogger(nil)

Information for Users

Information for Developers

exasol-driver-go's People

Contributors

anastasiiasergienko avatar cyrixsimon avatar kaklakariada avatar morazow avatar nightapes avatar pj-spoelders avatar sabitor avatar tomuben avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

exasol-driver-go's Issues

Rename UseTLS to align with JDBC driver naming conventions

Configuration option UseTLS(false) / usetls=0 actually means to ignore TLS certificate errors (InsecureSkipVerify=true). That's misleading and should be renamed to validateservercertificate=true/false like in the JDBC driver.

Default value for validateservercertificate must be true.

Implement exists logic

Especially for testing it is very beneficial to just ensure that a database object is present.
Currently the error I get is e.g.

E-GOD-11: execution failed with SQL error code '42500' and message 'schema RESOURCES_VIEW_TEST_TESTMAIN already exists (Session: 1719069525976612864)'

In these cases it would be good if I could distinct the error cause and say something like:

if errors.Is(err, exasol.ErrObjectExist) {
  // EOK
}

Driver reports wrong version

We found out in the EXA_STATISTICS.EXA_DBA_AUDIT_SESSION table driver logs itself as exasol-driver-go v1.0.0 even though our user is currently using v0.1.2 so I expect to see this version in the logs.

Exasol version: 7.0.18

Invalid SQL Parsing

When executing an insert statement that contains 'import into ' as a value the following error is raised.

E-EGOD-27: could not parse import query

Example statement.
insert into table1 values ('import into {{dest.schema}}.{{dest.table}} ) from local csv file ''{{file.path}}'' ');

Improve code quality

Problem

After the code is covered with tests, we want to refactor it and improve code quality

Integration tests fail with Exasol 7.1.1

When using docker image exasol/docker-db:7.1.1 the integration tests fail.

Examples for test failures when running go test -v -coverprofile=coverage.out ./...:

=== RUN   TestIntegrationSuite/TestConnectWithWrongPassword
    integration_test.go:60: 
                Error Trace:    integration_test.go:60
                Error:          Error message not equal:
                                expected: "[08004] Connection exception - authentication failed."
                                actual  : "local error: tls: unexpected message"
                Test:           TestIntegrationSuite/TestConnectWithWrongPassword
...
=== RUN   TestIntegrationSuite/TestConnect
    suite.go:63: test panicked: runtime error: invalid memory address or nil pointer dereference
        goroutine 94 [running]:
        runtime/debug.Stack(0xc000365778, 0xa43e40, 0xf0b7a0)
                /usr/lib/golang/src/runtime/debug/stack.go:24 +0x9f
        github.com/stretchr/testify/suite.failOnPanic(0xc00008a780)
                /home/user/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:63 +0x5b
        panic(0xa43e40, 0xf0b7a0)
                /usr/lib/golang/src/runtime/panic.go:965 +0x1b9
        database/sql.(*Rows).Columns(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
                /usr/lib/golang/src/database/sql/sql.go:2942 +0x5c
        github.com/exasol/exasol-driver-go_test.(*IntegrationTestSuite).TestConnect(0xc00058e400)
                /home/user/git/exasol-driver-go/integration_test.go:42 +0x113
        reflect.Value.call(0xc000079080, 0xc0006240d0, 0x13, 0xaefa53, 0x4, 0xc00005ee30, 0x1, 0x1, 0xc0000f9cf8, 0x40da0a, ...)
                /usr/lib/golang/src/reflect/value.go:476 +0x8e7
        reflect.Value.Call(0xc000079080, 0xc0006240d0, 0x13, 0xc0000f9e30, 0x1, 0x1, 0xc6c1d3, 0x26, 0x438)
                /usr/lib/golang/src/reflect/value.go:337 +0xb9
        github.com/stretchr/testify/suite.Run.func1(0xc00008a780)
                /home/user/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:158 +0x379
        testing.tRunner(0xc00008a780, 0xc0005127e0)
                /usr/lib/golang/src/testing/testing.go:1193 +0xef
        created by testing.(*T).Run
                /usr/lib/golang/src/testing/testing.go:1238 +0x2b3

Add dsn builder

Problem

At the moment you need to know the dsn format and build a string to use our sql driver. Add a simple dsn builder for easier use of the driver.

Support for URL as connection parameter

Background

In some cases, the user might want to connect to an URL, instead of host:port. A use case could be to have a websocket proxy implemented in a RestApi. Then the go driver should accept the URL path and ignore the port.

Acceptance Criteria

Add a new parameter to the configuration UrlPath, which contains the URL Path and the Query parameters.
For example, having the RestApi my-api.com/v1/databases/websocket?token=abc, the driver configuration should be:

  • Host: my-api.com
  • UrlPath: /v1/databases/websocket?token=abc

Implement secure IMPORT

At the moment IMPORT ignores the secure flag and uploads the file via http.

Create certificate on the fly and upload via https

Add tests

Problem

We want to cover existing code with tests.

Test with Go 1.22

The latest Go release adds support for nullable types:

A new Null[T] type in database/sql provides a way to scan nullable columns.

See https://go.dev/blog/go1.22

We need to add tests that this works with the go driver.

Insecured connection triggers "x509: certificate signed by unknown authority" Error

First of all, thanks for this cool package !

When using the insecure option, i still do get an "x509: certificate signed by unknown authority" when querying the DB :

package main

import (
	"database/sql"
	"fmt"
	"log"

	_ "github.com/exasol/exasol-driver-go"
)

func main() {
	log.Printf("Drivers=%#v\n", sql.Drivers())
	db, err := sql.Open("exasol", "exa:<host>:<port>;user=<user>;password=<password>;insecure=1")
	if err != nil {
		fmt.Println(err)
	}
	defer db.Close()

	err = db.Ping()
	if err != nil {
		fmt.Println(err)
	}

}

returns :

$ go run exasol_sql.go
2021/07/10 20:51:13 Drivers=[]string{"exasol"}
x509: certificate signed by unknown authority

When using the go-exasol-client i can set the same config with :

conf := exasol.ConnConf{
		Host:      <host>",
		Port:      <port>,
		Username:  <user>,
		Password:  <password>,
		TLSConfig: &tls.Config{InsecureSkipVerify: true},
	}

which works seamlessly. But i would like to take advantage of the binding with the "database/sql" package.
If anyone could give any hint, i would be very thankful !

Cheers,
Clement.

Import failes with exasol ip range syntax

The import command fails if a ip-range is specified when a connection is established.

sql.Open("exasol", exasol.NewConfig("user", "password").
			ClientName("client").
			ClientVersion("0.0.0").
			Host("10.1.0.10..10.1.0.12").
			Port(port).
			Autocommit(false).
			String())

The ip range can't be handeled correctly from net.Dial. This causes the error - E-EGOD-26: could not create proxy connection to import file.

func (c *connection) handleImportQuery(query string) (string, error) {
	paths, err := getFilePaths(query)
	if err != nil {
		return "", err
	}

	p, err := newProxy(c.config.host, c.config.port)
func newProxy(host string, port int) (*proxy, error) {
	uri := fmt.Sprintf("%s:%d", host, port)
	con, err := net.Dial("tcp", uri)
	if err != nil {
		wrappedErr := fmt.Errorf("%w: could not create TCP connection to %s, %s", ErrInvalidProxyConn, uri, err.Error())
		errorLogger.Print(wrappedErr)
		return nil, wrappedErr
	}

	p := &proxy{
		connection: con,
		isClosed:   false,
	}

	return p, nil
}

FetchSize is bytes not kilobytes

Follow DSNConfigBuilder.FetchSize through and it sets FetchCommand.NumBytes in result_set.go without ever being multiplied by 1024.

Add license

Problem

A license file is missing in the repository.
For open-source projects, we usually use an MIT license.

Fix workflow

Problem

We should specify the main branch on push instead of the master in the workflow file

IMPORT FROM LOCAL FBV

As a kind a follow up for #98, IMPORT command also supports local "FBV" files, see syntactic diagram for file_src in IMPORT.

Therefore, one might want to extend the regular expression even further ...

Implement host with multiple values

When opening a connection, the driver will randomly choose an address from the specified address range. If the connection fails, the driver will continue to try all other possible addresses. You can also specify an IP range with a comma-separated list.

For example:
 
myhost:8563: Single server with name myhost and port 8563. Or single host myhost with multiple IP addresses mapped to it and port 8563.
myhost1,myhost2:8563: Two servers with port 8563.
myhost1..4:8563: Four servers (myhost1, myhost2, myhost3, myhost4) and port 8563.
192.168.6.11..14:8563: Four servers from 192.168.6.11 to 192.168.6.14 and port 8563.

URL Path should not be encoded

When using an URLPath for connecting to a websocket (implemented in #102 #103), the driver wrongly URL-encodes the complete path.

Example: when using path /api/v1/internal/accounts/org/databases/db/clusters/cluster/ws?ticket=abc, the driver sends /api/v1/internal/accounts/org/databases/db/clusters/cluster/ws%3Fticket=abc in the request.

Inappropriate process fetch command

I am trying to get big query response and encounter problem: if results doesn't fit in one fetch I get panic: runtime error: index out of range [18725] with length 18725 (number depends on fetchsize property)

goroutine 1 [running]:
github.com/exasol/exasol-driver-go.(*queryResults).Next(0xc000030200, 0xc0002d0320, 0xa, 0xa, 0x1866da0, 0x197eb01)
       ./mod/github.com/exasol/[email protected]/result_set.go:106 +0x4a5

After some research I discovered the root of issue, I guess. In ./result_set.go in func (results *queryResults) Next(dest []driver.Value) there is code

result := &SQLQueryResponseResultSetData{}
err := results.con.send(context.Background(), &FetchCommand{
	Command:         Command{"fetch"},
	ResultSetHandle: results.data.ResultSetHandle,
	StartPosition:   results.rowPointer,
	NumBytes:        results.con.config.FetchSize,
}, result)
if err != nil {
	return err
}

results.fetchedRows = results.fetchedRows + result.NumRowsInMessage

But, according fetch command structure, response doesn't have NumRowsInMessage field, and thereby application panic, if result set doesn't fit in on fetch.
I suppose, we need change result.NumRowsInMessage -> result.NumRows and that's all, but I am not so good in low-level driver implementation, so please, check it by yourself and correct me, if I am wrong.

Logging

Hi,

what would you think about to add an optional logging configuration? That will allow to pass in an own preferred logger which satisfy the standard go logging interface.

E-EGOD-11: execution failed with SQL error code '00000' and message 'getDouble: JSON value is not a double

Hallo,

if you try to insert a decimal value with a prepared statement you will get an JSON error.

Error:
panic: E-EGOD-11: execution failed with SQL error code '00000' and message 'getDouble: JSON value is not a double

Table structure:
create or replace table work.dummy (a integer, b float);

Go Code:
`
exadb, err := sql.Open("exasol", config.String())
if err != nil {
panic(err)
}
defer exadb.Close()

stmt, err := exadb.Prepare("insert into work.dummy values(?,?)")
if err != nil {
	panic(err)
}

_, err = stmt.Exec(1, 1)
if err != nil {
	panic(err)
}

`

Querying large data set causes out of memory error

Hi,

when executing a query the data is stored in the SQLQueryResponseResultSetData struct.

type SQLQueryResponseResultSetData struct {
	ResultSetHandle  int              `json:"resultSetHandle"`
	NumColumns       int              `json:"numColumns,omitempty"`
	NumRows          int              `json:"numRows"`
	NumRowsInMessage int              `json:"numRowsInMessage"`
	Columns          []SQLQueryColumn `json:"columns,omitempty"`
	Data             [][]interface{}  `json:"data"`
}

func (results *queryResults) Next(dest []driver.Value) error {
...
		if results.data.Data == nil {
			results.data.Data = result.Data
		} else {
			for i, d := range result.Data {
				results.data.Data[i] = append(results.data.Data[i], d...)
			}
		}

	}

	for i := range dest {
		dest[i] = results.data.Data[i][results.rowPointer]
	}
	results.rowPointer = results.rowPointer + 1
	return nil
}

If the result set is large, it consumes a lot of memory and may cause a crash due to lack of memory.

Rows.Err is not set correctly in the event of an error

Hi,

the following error during fetching the data set from the database is not set correctly.

Error
W-EGOD-17: could not receive data: 'websocket: close 1006 (abnormal closure): unexpected EOF'

Code
for rows.Next() { err = scanFunc(rows) if err != nil { return err } } if err = rows.Err(); err != nil { return err }

Rows.Err returns the error for advancing to the next row or error. But the error W-EGOD-17 ... is not set in rows.Err.

You only see the output of the internal logger of the driver.
websocket.go:124: W-EGOD-17: could not receive data: 'websocket: close 1006 (abnormal closure): unexpected EOF'

DSN parsing not working when parameters end with special character

Following the issue #23 , i finally found out what the root cause was.

If the password given ends with an exclamation mark, e.g. user=ARCHIMEDES;password=123Eureka!;encryption=1;usetls=0, when building the connection configuration in dsn.go , the parsed password returned is 123Eureka!;encryption=1 instead of 123Eureka!

I could fix the problem by adjusting the following function as follow:

func extractParameters(parametersString string) []string {
        // Original code
	//reg := regexp.MustCompile(`[\w];`)
	//return splitAfter(parametersString, reg)

        // dirty fix
	return strings.Split(parametersString, ";")
}

is there a better way to address this issue ?

Thanks a lot in advance,

Best Regards,

Clement.

Invalid import statement leads to goroutine leak

Hi,

the execution of an invalid import statement, that causes a database error, leads to a goroutine leak.

I have analyzed the problem and could clarify why this leak occurs. In the exec method the errs.Wait() will never return if the import statement causes an error.

func (c *connection) exec(ctx context.Context, query string, args []driver.Value) (driver.Result, error) {
...
	if isImportQuery(query) {
...
		errs.Go(func() error { return c.uploadFiles(p, originalQuery) })
	}
...
	err := errs.Wait()
	close(result)

	if err != nil {
		return nil, err
	}

	return <-result, nil
}

This is because the import statement is handled through two goroutines, one for reading the file and another for handling the sql statement.

func (c *connection) exec(ctx context.Context, query string, args []driver.Value) (driver.Result, error) {
...
errs.Go(func() error { return c.uploadFiles(p, originalQuery) })
...
errs.Go(c.executeSimpleWrapper(errctx, query, result))
...

Both goroutines increase the counter of the waitgroup (errs.wg). Whenever an error is returned from the goroutine which handels the sql statement the first goroutine never recognizes this and therefore keeps waiting. So overall the additional goroutine with import statements which handels the file processing does not notice when an error occurs and therefore never returns.

Exasol behind a proxy

Thanks a lot for your work on this driver. It seems to be what I'm looking for. My question is if it is possible to use the proxy object to put exasol behind it?

We want to do this to add extra security to the connections and to sniff the queries that are sent to exasol before they reach the dB. My idea is to use a TCP listener and then decode the information to send it to the DB.

Do you think it is possible to do it with this repo?

Make parseDsn public

Currently there seems no public way of converting a string DSN to a DSNConfig. Would be really helpful if parseDsn would be public.

Reduce log verbosity

The Go driver writes to many unhelpful log messages. See this PR by @sabitor (#96):

While using the Exasol Go driver I realized that there is a log message which gets printed out to stdout every time when I close the Exasol connection, which is not desirable.
I removed that line in the close function to prevent this extra log output.

Add support for TLS certificate fingerprints

Validating the server certificate fails when the certificate contains a wrong hostname (e.g. when using the exasol docker container or when the original certificate was not replaced with a real one).

To improve security and avoid ignoring certificate errors completely with InsecureSkipVerify=true / validateservercertificate=false (see #36 for renaming of this option) we should support specifying the fingerprint (=sha256 checksum) of the certificate when connecting to the database.

When the user specifies a fingerprint, the driver must

  • verify if the server's certificate matches the given fingerprint and
  • ignore the wrong hostname in the certificate.

The combinationvalidateservercertificate=false && fingerprint != nil should be considered invalid.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.