tiledb-inc / tiledb-go Goto Github PK

Go Interface to the TileDB storage manager

License: MIT License

Go 99.78% C 0.22%

tiledb golang golang-library go storage-manager hdfs s3 array

tiledb-go's Introduction

TileDB Go Bindings

This package provides TileDB golang bindings via cgo. The bindings have been designed to be idomatic Go. runtime.SetFinalizer is used to ensure proper free'ing of C heap allocated structures.

Quick Links

GoDoc API documentation: https://pkg.go.dev/github.com/TileDB-Inc/TileDB-Go
Full Installation Docs: https://docs.tiledb.com/main/how-to/installation
Quick Install: https://docs.tiledb.com/main/how-to/installation/quick-install
Full developer documentation for all APIs and integrations: https://docs.tiledb.com

Installation

Supported Platforms

Currently the following platforms are supported:

Linux
macOS (OSX)

Prerequisites

This package requires the TileDB shared library be installed and on the system path. See the official TileDB installation instructions for installation methods. TileDB must be compiled with serialization support enabled.

Environment setup

Make sure you have Go installed on your system. This guide assumes you are using Go 1.17 or later, which fully supports modules. You can check your Go version by running go version.

Initialization steps

go mod init github.com/<github_username>/repository_name

Go Installation

To install these bindings you can use go get:

 go get -v github.com/TileDB-Inc/TileDB-Go

To install package with test dependencies:

go get -v -t github.com/TileDB-Inc/TileDB-Go

Go Testing

Package tests can be run with:

go test github.com/TileDB-Inc/TileDB-Go

Compatibility

TileDB-Go follows semantic versioning. Currently TileDB core library does not, as such the below table reference which versions are compatible.

TileDB-Go Version	TileDB Version
0.7.X	1.6.X
0.8.0	1.7.0
0.8.1	1.7.0
0.8.2	1.7.2
0.8.3	>=1.7.3
0.8.4	>=1.7.3
0.8.5	>=1.7.3
0.9.0	2.0.X
0.10.0	2.1.X
0.11.0	2.2.X
0.12.0	2.3.X
0.13.0	>=2.4.X
0.14.0	>=2.5.X
0.15.0	>=2.8.X
0.16.0	2.10.X
0.17.0	2.11.X
0.18.0	2.12.X
0.19.0	2.13.X
0.20.0	2.14.X
0.21.0	2.15.X
0.22.0	2.16.X
0.23.0	2.17.X
0.24.0	2.18.X
0.25.0	2.19.X
0.26.0	2.20.X
0.27.0	2.21.X
0.28.0	2.22.X
0.29.0	2.23.X
0.30.0	2.24.X
0.31.0	2.25.X

Deprecated Functionality

0.21.0

The query methods (Set)?Buffer(Var|Nullable|Var|Unsafe)* are deprecated because the corresponding TileDB core methods are removed. The methods will be supported for 2 releases and are expected to be removed in release 0.23. It is recommended to use the proper combination of (Set|Get)DataBuffer, (Set|Get)ValidityBuffer and (Set|Get)OffsetBuffer.

0.23.1

The query methods (Add|Get)?Range are deprecated because they are deprecated in TileDB core. It is recommend to use the Subarray type for building queries. The methods will be removed in the release following their removal from TileDB core.

0.24.0

Array.DeleteFragments is deprecated in favor of tiledb.DeleteFragments which binds to C.tiledb_array_delete_fragments_v2 the preferred method to delete fragments in TileDB 2.18.0.

0.30.3

All deprecated APIs in TileDB-Go are removed as the corresponding C-APIs will be removed in the following TileDB release.

tiledb-go's People

Contributors

Stargazers

Watchers

Forkers

emi80 doytsujin laszlo-kiss royassis jdblischak kokizzu nullhypothesis

tiledb-go's Issues

Better support for TILEDB_STRING types

Encoding / Decoding support for UTF / UCS types.

Add travis ci check for formating

We should have a check in travis ci for making sure all files are formatted. If any file is not formatted via go fmt the check should fail. This should be the first check that is run in CI.

Adjust SetBuffer/SetBufferVar to return pointer for effective buffer size

An oversight of the go api is we currently provide no way to get the effective buffer size for reads. The c api provides this functionality by updating the pointer to the buffer_size with the effective size after reads.

Currently we set the cBufferSize, but do not provide a way to return to the user. We should make bufferSize a uint64, then pass get a pointer to it, cast it to the C pointer type, and return this pointer to the user. We will need to store a reference to this just like we do with the actual data slices to prevent it from being gc'ed (go garbage collector does not track variables passed to CGO).

ret := C.tiledb_query_set_buffer(q.context.tiledbContext, q.tiledbQuery, cAttribute, cbuffer, &cbufferSize)
    if ret != C.TILEDB_OK {
        return fmt.Errorf("Error setting query buffer: %s", q.context.LastError())
    }

Also it would be good to add a unit test in query_test.go for incomplete queries to validate this new functionality is working as expected.

Implement ResultBufferElements

Probably the most user friendly way to handle result buffer size for reads is to create a function for ResultBufferElements that returns the map of name -> {offset_elements, data_elements} just like c++. We just have to store these uint64 pointers internally (similar to how we store a reference to the data slice so it doesn't get GC'ed).

This is a follow on issue after #51

Remove compressor and add filters

Remove compressor and add filters as part of core tiledb 1.4 release

Add quickstart examples

Add quickstart examples.

dense array
sparse array

Update README for v0.1.0 release

Add:

Update compatibility to reflect expected tag of v0.1.0
Add Platforms Supported section
Add Missing Functionality section

In quickstart (sparse), setting attribute datatype to UINT32 leads to failed tests

In quickstart_sparse_test.go changing the following line

a, _ := tiledb.NewAttribute(ctx, "a", tiledb.TILEDB_INT32)

a, _ := tiledb.NewAttribute(ctx, "a", tiledb.TILEDB_UINT32)

disallows the test from passing (data buffer has no elements)

error returned checked and it is

'tiledb/tiledb_serialization.h' file not found

OSX.
Installed tileDB via brew:
brew install tiledb-inc/stable/tiledb

Attempting to install go library via:
go get -v github.com/TileDB-Inc/TileDB-Go

Result:

$ go get -v github.com/TileDB-Inc/TileDB-Go
github.com/TileDB-Inc/TileDB-Go
# github.com/TileDB-Inc/TileDB-Go
../../go/src/github.com/TileDB-Inc/TileDB-Go/array_schema.go:7:10: fatal error: 'tiledb/tiledb_serialization.h' file not found
#include <tiledb/tiledb_serialization.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

C.tiledb_vfs_dir_size

After installing TileDB , I am downloading TileDB-Go using go get, but each time i run the go get , i receive the following error.

could not determine kind of name for

C.tiledb_vfs_dir_size

How to fix this issue , Please.

Implement ArraySchema Type

Implement ArraySchema Type via tiledb_array_schema_get_array_type. We are missing the ability to get the array type.

Error retrieving Data from Sparse Array

@Shelnutt2 @antalakas I have these two functions for insert and retrieve ,

Rows dimensions and columns dimensions :

func GetRowDimension() []int64 {
	return []int64{1,2}
}

func GetColumnDimension() []int64{

	return []int64{1,int64(utils.TotalGenomeSize())}
}

where TotalGenomeSize is 3 billions - Total Length of a linear Human DNA

Creating schema function is like this:

func (manager *HiveDBManager) CreateSchema(rowDimensions []int64 , columnDimensions []int64,
	cellOrder tiledb.Layout , tileOrder tiledb.Layout,tile_size int) (err error){



	if manager.databaseName == "" || len(manager.databaseName) <= 0 {
		return fmt.Errorf("Please Set Database Name First")
	}
	manager.context , _ = tiledb.NewContext(nil)
	domain , err := tiledb.NewDomain(manager.context)
	rowDim , err := tiledb.NewDimension(manager.context,"rows",rowDimensions,int64(tile_size))
	colDim , err := tiledb.NewDimension(manager.context,"cols",columnDimensions,int64(tile_size))
	domain.AddDimensions(rowDim,colDim)
	//Now create the schema for this database
	schema , err := tiledb.NewArraySchema(manager.context,tiledb.TILEDB_SPARSE)
	schema.SetDomain(domain)
	schema.SetCellOrder(tiledb.TILEDB_ROW_MAJOR)
	schema.SetTileOrder(tiledb.TILEDB_ROW_MAJOR)
	//Chromosome Information
	//chromosomes , err := tiledb.NewAttribute(manager.context,CHROMOSOME_BUFFER,tiledb.TILEDB_STRING_ASCII)
	//ref , err := tiledb.NewAttribute(manager.context,REF_BUFFER,tiledb.TILEDB_STRING_ASCII)
	//alt , err := tiledb.NewAttribute(manager.context,ALT_BUFFER,tiledb.TILEDB_STRING_ASCII)
	variants , err := tiledb.NewAttribute(manager.context,CALL_BUFFER,tiledb.TILEDB_INT32)
	//schema.AddAttributes(chromosomes,ref,alt,variants)
	schema.AddAttributes(variants)
	//Create the array
	array, err := tiledb.NewArray(manager.context,manager.databaseName)
	return array.Create(schema)
}

Writing data function is like

func (manager *HiveDBManager) WriteData(row sqldb.TableRow,chromosomes []string,
	references []string,alternatives []string,coordinates []int64, Variants []int32) (err error){

	if manager.MetaData == nil {
		return fmt.Errorf("You have to initialize the HiveDB Manager first.")
	}
	//first write the row into the metadata table in sqlite
	err = manager.MetaData.InsertIntoDB(row)
	if err != nil {
		return
	}
	// Write the variants into tiledb
	array , _ := tiledb.NewArray(manager.context,manager.databaseName)
	array.Open(tiledb.TILEDB_WRITE) // we open the array for writing
	query , _ := tiledb.NewQuery(manager.context,array)
	query.SetLayout(tiledb.TILEDB_UNORDERED)
	//query.SetBuffer(CHROMOSOME_BUFFER,chromosomes)
	//query.SetBuffer(REF_BUFFER,references)
	//query.SetBuffer(ALT_BUFFER,alternatives)
	query.SetBuffer(CALL_BUFFER,Variants)
	query.SetCoordinates(coordinates)
	err = query.Submit()
	defer logging.Println(fmt.Sprintf("Database/Array : %s has been written and closed.",manager.databaseName))
	return array.Close()
}

and the function responsible for reading the data back is

func (manager *HiveDBManager) ReadData(coordinates []int64) ([]int64,error) {

	if manager.context == nil || manager.MetaData == nil {
		return nil , fmt.Errorf("You have to init the hiveDB manager first.")
	}

	array , _ := tiledb.NewArray(manager.context,manager.databaseName)
	array.Open(tiledb.TILEDB_READ)
	maxElements, err := array.MaxBufferElements(coordinates)
	if err != nil {
		logging.Println(err)
		return nil , err
	}
	size := maxElements[CALL_BUFFER][1]
	data := make([]int64, size)
	coords := make([]int64,maxElements[tiledb.TILEDB_COORDS][1])
	// Clear the slice or empty the slice
	query , _ := tiledb.NewQuery(manager.context,array)
	query.SetSubArray(coordinates)
	query.SetLayout(tiledb.TILEDB_ROW_MAJOR)
	query.SetBuffer(CALL_BUFFER,data)
	query.SetCoordinates(coords)
	err = query.Submit()
	if err != nil {
		logging.Println(err)
		return nil , err
	}
	hasResults , _ := query.HasResults()
	fmt.Println(data)
	logging.Println("Has Results : ", hasResults)
	elements, err := query.ResultBufferElements()
	if err != nil {
		logging.Println(err)
		return nil , err
	}
	fmt.Println(elements)
	resultNum := elements[CALL_BUFFER][1]
	for r := 0; r < int(resultNum); r++ {
		i := coords[2*r]
		j := coords[2*r+1]
		a := data[r]
		fmt.Printf("Cell (%d, %d) has data %d\n", i, j, a)
	}

	err = array.Close()
	if err != nil {
		return nil , err
	}
	return data , nil
}

Elements variable contents is always like this:
map[__coords:[0 2]]

and there are no results returned.

I don't know why i can't get the data back

@Shelnutt2 @antalakas It seems as if call_buffer which is the attribute for tiledb schema is not stored , because elements only return __coords , it doesn't include "call_buffer" element inside.

I really don't know how to fix this , any help will be greatly appreciated.

Add support for Query operations

Add support for Query operations by implementing a query struct similar to c++ api.

undefined references

I am getting these problems while downloading TileDB-Go using go get command.

github.com/TileDB-Inc/TileDB-Go

/tmp/go-build191067676/github.com/TileDB-Inc/TileDB-Go/_obj/_cgo_main.o:(.data.rel+0x8): undefined reference to TILEDB_VAR_NUM' /tmp/go-build191067676/github.com/TileDB-Inc/TileDB-Go/_obj/_cgo_main.o:(.data.rel+0x10): undefined reference to TILEDB_OFFSET_SIZE'
/tmp/go-build191067676/github.com/TileDB-Inc/TileDB-Go/_obj/_cgo_main.o:(.data.rel+0x18): undefined reference to `TILEDB_COORDS'
collect2: error: ld returned 1 exit status

What should i do to fix this, Please help.

tests are failing for currently latest c/go lib versions

installed c library and then ran go test. i am observing following error message:
c library version: TileDB v2.2.3
go library version: v0.11.4

could you please recommend the combination of c library version and go library version that I should be using?

=== RUN   TestEpoch
--- PASS: TestEpoch (0.00s)
=== RUN   TestFragmentInfo
    fragment_info_test.go:145: 
        	Error Trace:	fragment_info_test.go:145
        	Error:      	Not equal: 
        	            	expected: 0x7
        	            	actual  : 0x8
        	Test:       	TestFragmentInfo
--- FAIL: TestFragmentInfo (0.00s)
=== RUN   TestFragmentInfoEncryption
    fragment_info_test.go:263: 
        	Error Trace:	fragment_info_test.go:263
        	Error:      	Not equal: 
        	            	expected: 0x7
        	            	actual  : 0x8
        	Test:       	TestFragmentInfoEncryption
--- FAIL: TestFragmentInfoEncryption (0.00s)
=== RUN   TestGroupCreate
--- PASS: TestGroupCreate (0.00s)

tried with go library version v0.11.0 and c lib version at v2.2.3 per this link and see a panic:

--- FAIL: ExampleVariableLengthArray (0.02s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x54f946]

goroutine 1 [running]:
testing.(*InternalExample).processRunResult(0xc0000e1d28, 0xc00032e000, 0x82, 0x13384f1, 0x5a5c20, 0x6dd6f0, 0xc000000180)
	/usr/local/go/src/testing/example.go:89 +0x648
testing.runExample.func2(0xbfff7258aa241850, 0x1f1d038e3, 0x6f7f20, 0xc0001a61b8, 0xc0000a4008, 0xc0001aa1e0, 0xc0000e1d28, 0xc0000e1d58)
	/usr/local/go/src/testing/run_example.go:58 +0x10d
panic(0x5a5c20, 0x6dd6f0)
	/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/TileDB-Inc/TileDB-Go.(*Query).ResultBufferElements(0xc0001ac640, 0x0, 0x0, 0x41)
	/home/sdeoras/go/src/github.com/TileDB-Inc/TileDB-Go/query.go:1773 +0x2e6
github.com/TileDB-Inc/TileDB-Go/examples.readVariableLengthArray()
	/home/sdeoras/go/src/github.com/TileDB-Inc/TileDB-Go/examples/variable_length_test.go:247 +0x936
github.com/TileDB-Inc/TileDB-Go/examples.ExampleVariableLengthArray()
	/home/sdeoras/go/src/github.com/TileDB-Inc/TileDB-Go/examples/variable_length_test.go:258 +0x30
testing.runExample(0x5d16d0, 0x1a, 0x5dacb0, 0x5d9b5c, 0xdb, 0x0, 0x0)
	/usr/local/go/src/testing/run_example.go:62 +0x209
testing.runExamples(0xc0000e1ed0, 0x6e2b80, 0x1b, 0x1b, 0xbfff72ec952d54d7)
	/usr/local/go/src/testing/example.go:44 +0x1af
testing.(*M).Run(0xc0000c8000, 0x0)
	/usr/local/go/src/testing/testing.go:1346 +0x273
main.main()
	_testmain.go:95 +0x138
FAIL	github.com/TileDB-Inc/TileDB-Go/examples	8.386s
FAIL

Add VFS Binding Support

Add VFS Binding Support, as a prequesit context and config bindings are needed.

Config bindings
Context bindings
VFS bindings
Travis CI tests

Add support for encryption

Add support for new encryption in 1.4

Better Handling of Variable Length Attributes

We should add high level functions to handle variable length attributes in queries better. Right now variable length attributes that are being written but have their variable length squashed into a single vector. When reading the user is expect to provide/use a squashed vector with a second vector holding the offsets. It would be a nicer experience if we supported ingesting an slice of go strings, or a slice of slices for other variable length datatypes. In a higher level function the flattening can be handled.

The same idea could be used for reads, attribute buffers can be converted from flat buffers to more usable slices in go.

Go client error

github.com/!tile!d!b-!inc/[email protected]/array.go:6:10: fatal error: 'tiledb/tiledb.h' file not found
#include <tiledb/tiledb.h>

OS: MacOS
Golang version 1.14

Minor Inconsistency

This is a list of minor inconsistency found during a review of TileDB-Go

Context
- IsFSSupported should be IsSupportedFS to be inline with c/c++ api
- Get* should drop the Get prefix, it was dropped for all other structs /c++ api
Domain:
- AddDimensions should take pointers to be consistant with what is returned from NewDimension()
- AddDimension should be renamed AddDimensions
Attribute:
- AddAttributes shoudl take pointers to be consistant with what is returned from NewAttribute()

Add support for Key-Value Map

Thanks for writing this amazing library! (Both the Go wrapper and the underlying C/C++ code.) It's exciting research and I'm glad to see it applied so well.

Is there a roadmap for KV support in Go? I'd try to implement it myself based on the docs for the C++ version, but it sounds like you have special logic to make it faster than it would be otherwise.

Thanks!

MaxBufferElements should return all values as element counts

MaxBufferElements should return all values as element counts not size in bytes.

Panic whilst attempting to Seek using VFS module

Hi, I've just started learning Go and the TileDB-Go API, and I came across an issue when using VFS to read a raw binary file. I'm testing with TileDB-2.8.0 and TileDB-Go-0.15.0.

The issue occurred after I opened the file and immediately attempted to call seek on the file handler.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4c30a2]

goroutine 1 [running]:
github.com/TileDB-Inc/TileDB-Go.(*VFSfh).Seek(0xc000098000, 0xd4, 0xc00004fc58?)
	/home/user/go/pkg/mod/github.com/!tile!d!b-!inc/!tile!d!b-![email protected]/vfs.go:595 +0x1a2

I believe it is because the internal variable size on the file handler *VFSfh isn't set and defaults to nil once the file has been opened, as when I modify the Seek function and repeat this block before the offset check, the Seek operation works as expected.

The following is the general gist of what I'm doing:

    pathname := "<path to some file>"

    config, err := tiledb.NewConfig()
    checkError(err)
    defer config.Free()

    ctx, err := tiledb.NewContext(config)
    checkError(err)
    defer ctx.Free()

    vfs, err := tiledb.NewVFS(ctx, config)
    checkError(err)
    defer vfs.Free()

    stream, err := vfs.Open(pathname, tiledb.TILEDB_VFS_READ)
    checkError(err)
    defer stream.Close()
    defer stream.Free()

    fmt.Println(vfs.FileSize(pathname))
    fmt.Println(stream.IsClosed())

    fmt.Println("about to seek")
    pos, err := stream.Seek(int64(212), 0)
    checkError(err)

Setup Travis CI

Setup Travis CI by:

Move tiledb installation into a dedicated script
Remove sudo requirement for travis ci

Add vfs dir size

Add Support For Array

Add support for Array Schema:

go tests are failing

Hi, I am using tiledb core 2.3.4 and it looks like go get ... fetched tiledb-go version v0.15.1. When running go test github.com/TileDB-Inc/TileDB-Go I get the following:

aws_access_key_id: 
aws_external_id: 
aws_load_frequency: 
aws_role_arn: 
aws_secret_access_key: 
aws_session_name: 
aws_session_token: 
ca_file: 
ca_path: 
connect_max_tries: 5
connect_scale_factor: 25
connect_timeout_ms: 10800
endpoint_override: 
logging_level: Off
max_parallel_ops: 64
multipart_part_size: 5242880
proxy_host: 
proxy_password: 
proxy_port: 0
proxy_scheme: http
proxy_username: 
region: us-east-1
request_timeout_ms: 3000
requester_pays: false
scheme: https
skip_init: false
sse: 
sse_kms_key_id: 
use_multipart_upload: true
use_virtual_addressing: true
verify_ssl: true
--- FAIL: TestFragmentInfo (0.02s)
    fragment_info_test.go:23: 
        	Error Trace:	fragment_info_test.go:23
        	Error:      	Not equal: 
        	            	expected: 0x1055
        	            	actual  : 0x8a5
        	Test:       	TestFragmentInfo
--- FAIL: TestFragmentInfoEncryption (0.03s)
    fragment_info_test.go:43: 
        	Error Trace:	fragment_info_test.go:43
        	Error:      	Not equal: 
        	            	expected: 0x1cef
        	            	actual  : 0x8a5
        	Test:       	TestFragmentInfoEncryption
FAIL
FAIL	github.com/TileDB-Inc/TileDB-Go	0.592s
FAIL

Add Attributes() function for ArraySchema

Add Attributes() function for ArraySchema. The c++ api has this as an easy way to get a vector/slice of all attributes.

Add support for get_buffers

go test currently fails

~ »  go get -v github.com/TileDB-Inc/TileDB-Go
...

~ » go test -v github.com/TileDB-Inc/TileDB-Go
# github.com/TileDB-Inc/TileDB-Go
TileDB-Go/config_test.go:5:2: cannot find package "github.com/stretchr/testify/assert" in any of:
	/usr/local/Cellar/go/1.10.3/libexec/src/github.com/stretchr/testify/assert (from $GOROOT)
	/Users/jacobbolewski/go/src/github.com/stretchr/testify/assert (from $GOPATH)
FAIL	github.com/TileDB-Inc/TileDB-Go [setup failed]

Add support for Stats api

Add support for go modules

With golang 1.11 released we should support the new experimental go modules

ResultBufferElements incorrect

ResultBufferElements is currently based on the size of the buffer the user sets, not the size of the results after a query is submitted. This needs to be adjusted so when ResultBufferElements is called we get the effective size of each buffer and use that to return the element count.

Make URI variable of array struct public

Unable to build against libtiledb conda binary on macOS

Summary: I was able to build TileDB-Go against libtiledb installed as a conda binary on Ubuntu but not macOS. Getting this to work is not currently urgent; I am documenting my experience below in case it becomes a priority in the future.

cc: @ihnorton
xref: #251

Building TileDB-Go against the libtiledb conda binary was straightforward. I installed libtiledb (conda install -c conda-forge tiledb) and then set the documented env vars:

$ export CPATH=$CONDA_PREFIX/include
$ export LIBRARY_PATH=$CONDA_PREFIX/lib
$ export LD_LIBRARY_PATH=$CONDA_PREFIX/lib

If I don't manually set those env vars, then it can't find libtiledb:

$ go build .
# github.com/TileDB-Inc/TileDB-Go
./array.go:6:10: fatal error: tiledb/tiledb.h: No such file or directory
    6 | #include <tiledb/tiledb.h>
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.

As is expected, the above env vars are not sufficient for building on macOS. It cannot find the clang compiler installed in the conda env:

$ go test -v ./...
# runtime/cgo
cgo: C compiler "x86_64-apple-darwin13.4.0-clang" not found: exec: "x86_64-apple-darwin13.4.0-clang": executable file not found in $PATH
FAIL	github.com/TileDB-Inc/TileDB-Go [build failed]

When I replace LD_LIBRARY_PATH with DYLD_LIBRARY_PATH, then it can find the compiler, but then it unexpectedly fails the system libarchive is referenced instead of the one installed in the conda env.

$ go test -v ./...
dyld: Symbol not found: _iconv
  Referenced from: /usr/lib/libarchive.2.dylib
  Expected in: /Users/runner/micromamba-root/envs/nightly/lib/libiconv.2.dylib
 in /usr/lib/libarchive.2.dylib
/Users/runner/work/_temp/e8fbb84f-3bef-4e8d-8a75-22a209e66751.sh: line 16:  5184 Abort trap: 6           go test -v ./...

Notes on other things I tried:

Explicitly installing libarchive via conda had no effect
I tried replacing DYLD_LIBRARY_PATH wtih DYLD_FALLBACK_LIBRARY_PATH, but that had no effect (ie it was back to not being able to find the compiler)
I tried combining DYLD_FALLBACK_LIBRARY_PATH=$CONDA_PREFIX/lib and PATH=$CONDA_PREFIX/bin:$PATH, but it still couldn't find the compiler
I tested on a local osx-arm64. I couldn't even get the dyld error because DYLD_LIBRARY_PATH had no effect (presumably due to System Integrity Protection), and thus it couldn't find the compiler in the conda env

For anyone interested in troubleshooting this, here is my original GitHub Actions workflow that tested TileDB-Go against the nightly libtiledb conda binary on both Ubuntu and macOS. For PR #251, I switched to building libtiledb from source on macOS.

GitHub Actions workflow

name: TileDB-Go with nightly libtiledb

on:
  push:
    paths:
      - '.github/workflows/nightly.yml'
      - '.github/scripts/nightly/**'
  schedule:
    - cron: "0 4 * * *" # Every night at 4 AM UTC (11 PM EST; 12 AM EDT)
  workflow_dispatch:

jobs:
  build:
    runs-on: ${{ matrix.os }}
    name: nightly-${{ matrix.os }}-go-${{ matrix.go }}
    strategy:
      fail-fast: false
      matrix:
        go: ["1.18"]
        os: ["macos-11", "ubuntu-20.04"]
    defaults:
      run:
        shell: bash -l {0}
    steps:

    - uses: actions/checkout@v3

    - name: Install Conda environment with go and nightly libtiledb
      uses: mamba-org/provision-with-micromamba@main
      with:
        environment-file: false
        environment-name: nightly
        extra-specs: |
          sel(osx): clang
          sel(linux): gcc
          sel(osx): libarchive
          conda-forge::go=${{ matrix.go }}
          tiledb
        channels: tiledb/label/nightlies, conda-forge

    - name: Install dependencies
      run: go get -t .

    - name: Test TileDB-Go
      run: |
        export CPATH=$CONDA_PREFIX/include
        export LIBRARY_PATH=$CONDA_PREFIX/lib

        OS=$(uname)
        if [[ "$OS" == "Linux" ]]
        then
          export LD_LIBRARY_PATH=$CONDA_PREFIX/lib
        elif [[ "$OS" == "Darwin" ]]
        then
          export DYLD_LIBRARY_PATH=$CONDA_PREFIX/lib
        else
          echo "Unrecognized OS: $OS"
          exit 1
        fi

        go test -v ./...

Add Dump Support

Add Dump Support just like stats for:

[x ] array schema
[ x] attribute
[x ] domain
[x ] dimension

Windows support

Windows support is possible using a GCC for windows port (MinGW) as pointed out in:
TileDB-Inc/TileDB#1129.

Better documentation / CI testing against Windows is probably required to make this a first class supported feature.

C.int and C.int32_t

../../src/github.com/TileDB-Inc/TileDB-Go/array.go:269: cannot use &isEmpty (type *C.int) as type *C.int32_t in argument to _Cfunc_tiledb_array_get_non_empty_domain

../../src/github.com/TileDB-Inc/TileDB-Go/array.go:269: cannot use _Cfunc_tiledb_array_get_non_empty_domain(_cgoCheckPointer((*C.struct_tiledb_ctx_t)(a.context.tiledbContext)).(*C.struct_tiledb_ctx_t), _cgoCheckPointer((*C.struct_tiledb_array_t)(a.tiledbArray)).(*C.struct_tiledb_array_t), _cgoCheckPointer0(unsafe.Pointer(&tmpDomain[0]), tmpDomain), &isEmpty) (type C.int32_t) as type C.int in assignment

../../hivedblibs/src/github.com/TileDB-Inc/TileDB-Go/array.go:289: cannot use &isEmpty (type *C.int) as type *C.int32_t in argument to _Cfunc_tiledb_array_get_non_empty_domain
../../hivedblibs/src/github.com/TileDB-Inc/TileDB-Go/array.go:289: cannot use _Cfunc_tiledb_array_get_non_empty_domain(_cgoCheckPointer((*C.struct_tiledb_ctx_t)(a.context.tiledbContext)).(*C.struct_tiledb_ctx_t), _cgoCheckPointer((*C.struct_tiledb_array_t)(a.tiledbArray)).(*C.struct_tiledb_array_t), _cgoCheckPointer0(unsafe.Pointer(&tmpDomain[0]), tmpDomain), &isEmpty) (type C.int32_t) as type C.int in assignment
../../hivedblibs/src/github.com/TileDB-Inc/TileDB-Go/array.go:309: cannot use &isEmpty (type *C.int) as type *C.int32_t in argument to _Cfunc_tiledb_array_get_non_empty_domain
../../hivedblibs/src/github.com/TileDB-Inc/TileDB-Go/array.go:309: cannot use _Cfunc_tiledb_array_get_non_empty_domain(_cgoCheckPointer((*C.struct_tiledb_ctx_t)(a.context.tiledbContext)).(*C.struct_tiledb_ctx_t), _cgoCheckPointer((*C.struct_tiledb_array_t)(a.tiledbArray)).(*C.struct_tiledb_array_t), _cgoCheckPointer0(unsafe.Pointer(&tmpDomain[0]), tmpDomain), &isEmpty) (type C.int32_t) as type C.int in assignment
../../hivedblibs/src/github.com/TileDB-Inc/TileDB-Go/array.go:329: cannot use &isEmpty (type *C.int) as type *C.int32_t in argument to _Cfunc_tiledb_array_get_non_empty_domain
../../hivedblibs/src/github.com/TileDB-Inc/TileDB-Go/array.go:329: cannot use _Cfunc_tiledb_array_get_non_empty_domain(_cgoCheckPointer((*C.struct_tiledb_ctx_t)(a.context.tiledbContext)).(*C.struct_tiledb_ctx_t), _cgoCheckPointer((*C.struct_tiledb_array_t)(a.tiledbArray)).(*C.struct_tiledb_array_t), _cgoCheckPointer0(unsafe.Pointer(&tmpDomain[0]), tmpDomain), &isEmpty) (type C.int32_t) as type C.int in assignment
../../hivedblibs/src/github.com/TileDB-Inc/TileDB-Go/array.go:349: cannot use &isEmpty (type *C.int) as type *C.int32_t in argument to _Cfunc_tiledb_array_get_non_empty_domain
../../hivedblibs/src/github.com/TileDB-Inc/TileDB-Go/array.go:349: cannot use _Cfunc_tiledb_array_get_non_empty_domain(_cgoCheckPointer((*C.struct_tiledb_ctx_t)(a.context.tiledbContext)).(*C.struct_tiledb_ctx_t), _cgoCheckPointer((*C.struct_tiledb_array_t)(a.tiledbArray)).(*C.struct_tiledb_array_t), _cgoCheckPointer0(unsafe.Pointer(&tmpDomain[0]), tmpDomain), &isEmpty) (type C.int32_t) as type C.int in assignment
../../hivedblibs/src/github.com/TileDB-Inc/TileDB-Go/array.go:349: too many errors

TileDB-GO started failing build against nightly libtiledb

I run nightly tests against the dev branch of libtiledb in jdblischak/centralized-tiledb-nightlies. TileDB-GO started failing yesterday.

The last passing build was Monday night with libtiledb 62098dc and TileDB-GO b0fd1c9.

On Tuesday night (yesterday) it failed with libtiledb 42dbf36 and TileDB-GO b0fd1c9 (ie no change in TileDB-GO).

I happened to be running my nightlies yesterday in order to troubleshoot an unrelated problem, and TileDB-GO actually failed earlier in the day yesterday too with libtiledb 6aca860. So the error was likely introduced in one of these five commits.

The code I use to build and test TileDB-GO is here, but briefly it is:

go get -t .

# libtiledb.so is installed in ./install-libtiledb/lib
export CPATH=$(pwd)/install-libtiledb/include
export LIBRARY_PATH=$(pwd)/install-libtiledb/lib
export LD_LIBRARY_PATH=$(pwd)/install-libtiledb/lib

go test -v ./...

The lines with FAIL are:

FAIL	github.com/TileDB-Inc/TileDB-Go [build failed]
FAIL	github.com/TileDB-Inc/TileDB-Go/examples [build failed]

Click for full output

# github.com/TileDB-Inc/TileDB-Go
cgo-gcc-prolog: In function ‘_cgo_a04293bf5dcb_Cfunc_tiledb_array_delete_fragments’:
cgo-gcc-prolog:152:2: warning: ‘tiledb_array_delete_fragments’ is deprecated [-Wdeprecated-declarations]
In file included from ./install-libtiledb/include/tiledb/tiledb.h:4420,
                 from ./array.go:4:
?   	github.com/TileDB-Inc/TileDB-Go/bytesizes	[no test files]
./install-libtiledb/include/tiledb/tiledb_deprecated.h:533:34: note: declared here
  533 | TILEDB_DEPRECATED_EXPORT int32_t tiledb_array_delete_fragments(
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# github.com/TileDB-Inc/TileDB-Go
cgo-gcc-prolog: In function ‘_cgo_a04293bf5dcb_Cfunc_tiledb_query_add_range’:
cgo-gcc-prolog:76:2: warning: ‘tiledb_query_add_range’ is deprecated [-Wdeprecated-declarations]
In file included from ./install-libtiledb/include/tiledb/tiledb.h:4420,
                 from ./query.go:4:
./install-libtiledb/include/tiledb/tiledb_deprecated.h:1[8](https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/9088948929/job/25013512948#step:5:9)3:34: note: declared here
  183 | TILEDB_DEPRECATED_EXPORT int32_t tiledb_query_add_range(
      |                                  ^~~~~~~~~~~~~~~~~~~~~~
cgo-gcc-prolog: In function ‘_cgo_a042[9](https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/9088948929/job/25013512948#step:5:10)3bf5dcb_Cfunc_tiledb_query_add_range_by_name’:
cgo-gcc-prolog:100:2: warning: ‘tiledb_query_add_range_by_name’ is deprecated [-Wdeprecated-declarations]
In file included from ./install-libtiledb/include/tiledb/tiledb.h:4420,
                 from ./query.go:4:
./install-libtiledb/include/tiledb/tiledb_deprecated.h:216:34: note: declared here
  216 | TILEDB_DEPRECATED_EXPORT int32_t tiledb_query_add_range_by_name(
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cgo-gcc-prolog: In function ‘_cgo_a04293bf5dcb_Cfunc_tiledb_query_add_range_var’:
cgo-gcc-prolog:126:2: warning: ‘tiledb_query_add_range_var’ is deprecated [-Wdeprecated-declarations]
In file included from ./install-libtiledb/include/tiledb/tiledb.h:4420,
                 from ./query.go:4:
./install-libtiledb/include/tiledb/tiledb_deprecated.h:246:34: note: declared here
FAIL	github.com/TileDB-Inc/TileDB-Go [build failed]
FAIL	github.com/TileDB-Inc/TileDB-Go/examples [build failed]
  246 | TILEDB_DEPRECATED_EXPORT int32_t tiledb_query_add_range_var(
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~
cgo-gcc-prolog: In function ‘_cgo_a04293bf5dcb_Cfunc_tiledb_query_add_range_var_by_name’:
cgo-gcc-prolog:151:2: warning: ‘tiledb_query_add_range_var_by_name’ is deprecated [-Wdeprecated-declarations]
In file included from ./install-libtiledb/include/tiledb/tiledb.h:4420,
                 from ./query.go:4:
./install-libtiledb/include/tiledb/tiledb_deprecated.h:277:34: note: declared here
  277 | TILEDB_DEPRECATED_EXPORT int32_t tiledb_query_add_range_var_by_name(
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cgo-gcc-prolog: In function ‘_cgo_a04293bf5dcb_Cfunc_tiledb_query_get_range’:
cgo-gcc-prolog:478:2: warning: ‘tiledb_query_get_range’ is deprecated [-Wdeprecated-declarations]
In file included from ./install-libtiledb/include/tiledb/tiledb.h:4420,
                 from ./query.go:4:
./install-libtiledb/include/tiledb/tiledb_deprecated.h:355:34: note: declared here
  355 | TILEDB_DEPRECATED_EXPORT int32_t tiledb_query_get_range(
      |                                  ^~~~~~~~~~~~~~~~~~~~~~
cgo-gcc-prolog: In function ‘_cgo_a04293bf5dcb_Cfunc_tiledb_query_get_range_from_name’:
cgo-gcc-prolog:503:2: warning: ‘tiledb_query_get_range_from_name’ is deprecated [-Wdeprecated-declarations]
In file included from ./install-libtiledb/include/tiledb/tiledb.h:4420,
                 from ./query.go:4:
./install-libtiledb/include/tiledb/tiledb_deprecated.h:387:34: note: declared here
  387 | TILEDB_DEPRECATED_EXPORT int32_t tiledb_query_get_range_from_name(
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cgo-gcc-prolog: In function ‘_cgo_a04293bf5dcb_Cfunc_tiledb_query_get_range_num’:
cgo-gcc-prolog:526:2: warning: ‘tiledb_query_get_range_num’ is deprecated [-Wdeprecated-declarations]
In file included from ./install-libtiledb/include/tiledb/tiledb.h:4420,
                 from ./query.go:4:
./install-libtiledb/include/tiledb/tiledb_deprecated.h:303:34: note: declared here
  303 | TILEDB_DEPRECATED_EXPORT int32_t tiledb_query_get_range_num(
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~
cgo-gcc-prolog: In function ‘_cgo_a04293bf5dcb_Cfunc_tiledb_query_get_range_num_from_name’:
cgo-gcc-prolog:548:2: warning: ‘tiledb_query_get_range_num_from_name’ is deprecated [-Wdeprecated-declarations]
In file included from ./install-libtiledb/include/tiledb/tiledb.h:4420,
                 from ./query.go:4:
./install-libtiledb/include/tiledb/tiledb_deprecated.h:326:34: note: declared here
  326 | TILEDB_DEPRECATED_EXPORT int32_t tiledb_query_get_range_num_from_name(
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cgo-gcc-prolog: In function ‘_cgo_a04293bf5dcb_Cfunc_tiledb_query_get_range_var’:
cgo-gcc-prolog:573:2: warning: ‘tiledb_query_get_range_var’ is deprecated [-Wdeprecated-declarations]
In file included from ./install-libtiledb/include/tiledb/tiledb.h:4420,
                 from ./query.go:4:
./install-libtiledb/include/tiledb/tiledb_deprecated.h:475:34: note: declared here
  475 | TILEDB_DEPRECATED_EXPORT int32_t tiledb_query_get_range_var(
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~
cgo-gcc-prolog: In function ‘_cgo_a04293bf5dcb_Cfunc_tiledb_query_get_range_var_from_name’:
cgo-gcc-prolog:597:2: warning: ‘tiledb_query_get_range_var_from_name’ is deprecated [-Wdeprecated-declarations]
In file included from ./install-libtiledb/include/tiledb/tiledb.h:4420,
                 from ./query.go:4:
./install-libtiledb/include/tiledb/tiledb_deprecated.h:504:34: note: declared here
  504 | TILEDB_DEPRECATED_EXPORT int32_t tiledb_query_get_range_var_from_name(
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cgo-gcc-prolog: In function ‘_cgo_a04293bf5dcb_Cfunc_tiledb_query_get_range_var_size’:
cgo-gcc-prolog:622:2: warning: ‘tiledb_query_get_range_var_size’ is deprecated [-Wdeprecated-declarations]
In file included from ./install-libtiledb/include/tiledb/tiledb.h:4420,
                 from ./query.go:4:
./install-libtiledb/include/tiledb/tiledb_deprecated.h:417:34: note: declared here
  417 | TILEDB_DEPRECATED_EXPORT int32_t tiledb_query_get_range_var_size(
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cgo-gcc-prolog: In function ‘_cgo_a04293bf5dcb_Cfunc_tiledb_query_get_range_var_size_from_name’:
cgo-gcc-prolog:646:2: warning: ‘tiledb_query_get_range_var_size_from_name’ is deprecated [-Wdeprecated-declarations]
In file included from ./install-libtiledb/include/tiledb/tiledb.h:4420,
                 from ./query.go:4:
./install-libtiledb/include/tiledb/tiledb_deprecated.h:446:34: note: declared here
  446 | TILEDB_DEPRECATED_EXPORT int32_t tiledb_query_get_range_var_size_from_name(
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cgo-gcc-prolog: In function ‘_cgo_a04293bf5dcb_Cfunc_tiledb_query_set_subarray’:
cgo-gcc-prolog:905:2: warning: ‘tiledb_query_set_subarray’ is deprecated [-Wdeprecated-declarations]
In file included from ./install-libtiledb/include/tiledb/tiledb.h:4420,
                 from ./query.go:4:
./install-libtiledb/include/tiledb/tiledb_deprecated.h:[10](https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/9088948929/job/25013512948#step:5:11)1:34: note: declared here
  101 | TILEDB_DEPRECATED_EXPORT int32_t tiledb_query_set_subarray(
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~
cgo-gcc-prolog: In function ‘_cgo_a04293bf5dcb_Cfunc_tiledb_query_submit_async’:
cgo-gcc-prolog:991:2: warning: ‘tiledb_query_submit_async’ is deprecated [-Wdeprecated-declarations]
In file included from ./install-libtiledb/include/tiledb/tiledb.h:4420,
                 from ./query.go:4:
./install-libtiledb/include/tiledb/tiledb_deprecated.h:151:34: note: declared here
  151 | TILEDB_DEPRECATED_EXPORT int32_t tiledb_query_submit_async(
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~
# github.com/TileDB-Inc/TileDB-Go
./serialize.go:386:[14](https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/9088948929/job/25013512948#step:5:15)9: not enough arguments in call to (_Cfunc_tiledb_deserialize_array)
	have (*_Ctype_struct_tiledb_ctx_handle_t, *_Ctype_struct_tiledb_buffer_handle_t, _Ctype_tiledb_serialization_type_t, _Ctype_int, **_Ctype_struct_tiledb_array_t)
	want (*_Ctype_struct_tiledb_ctx_handle_t, *_Ctype_struct_tiledb_buffer_handle_t, _Ctype_tiledb_serialization_type_t, _Ctype_int, *_Ctype_char, **_Ctype_struct_tiledb_array_t)
=== RUN   TestGoldens
=== PAUSE TestGoldens
=== RUN   TestBadPackages
=== PAUSE TestBadPackages
=== RUN   TestBadName
=== PAUSE TestBadName
=== RUN   TestCantWrite
=== PAUSE TestCantWrite
=== CONT  TestGoldens
=== RUN   TestGoldens/defaults.txt
=== PAUSE TestGoldens/defaults.txt
=== RUN   TestGoldens/custom.txt
=== PAUSE TestGoldens/custom.txt
=== CONT  TestGoldens/defaults.txt
=== CONT  TestBadName
=== CONT  TestBadPackages
=== RUN   TestBadPackages/--out=/at-the-root.go
=== PAUSE TestBadPackages/--out=/at-the-root.go
=== RUN   TestBadPackages/--out=/tmp/TestBadPackages778995793/001/package-name-is/invalid.go
=== PAUSE TestBadPackages/--out=/tmp/TestBadPackages778995793/001/package-name-is/invalid.go
=== RUN   TestBadPackages/--out=/tmp/TestBadPackages778995793/001/some/valid/path.go_--pkg=invalid-package
=== PAUSE TestBadPackages/--out=/tmp/TestBadPackages778995793/001/some/valid/path.go_--pkg=invalid-package
=== CONT  TestBadPackages/--out=/at-the-root.go
=== CONT  TestGoldens/custom.txt
=== NAME  TestBadName
    sizegen_test.go:88: "I have spaces" is an invalid suffix
        exit status 2
        
--- PASS: TestBadName (0.25s)
=== CONT  TestBadPackages/--out=/tmp/TestBadPackages778995793/001/some/valid/path.go_--pkg=invalid-package
=== CONT  TestBadPackages/--out=/tmp/TestBadPackages778995793/001/package-name-is/invalid.go
=== CONT  TestCantWrite
--- PASS: TestGoldens (0.00s)
    --- PASS: TestGoldens/defaults.txt (0.25s)
    --- PASS: TestGoldens/custom.txt (0.26s)
=== NAME  TestCantWrite
    sizegen_test.go:104: could not open output file: open /tmp/TestCantWrite3755731938/001/somepkg/filename.go: no such file or directory
        exit status 1
        
--- PASS: TestCantWrite (0.[15](https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/9088948929/job/25013512948#step:5:16)s)
--- PASS: TestBadPackages (0.00s)
    --- PASS: TestBadPackages/--out=/at-the-root.go (0.26s)
    --- PASS: TestBadPackages/--out=/tmp/TestBadPackages778995793/001/some/valid/path.go_--pkg=invalid-package (0.[16](https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/9088948929/job/25013512948#step:5:17)s)
    --- PASS: TestBadPackages/--out=/tmp/TestBadPackages778995793/001/package-name-is/invalid.go (0.16s)
PASS
ok  	github.com/TileDB-Inc/TileDB-Go/tools/sizegen	0.4[18](https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/9088948929/job/25013512948#step:5:19)s
FAIL

xref: jdblischak/centralized-tiledb-nightlies#6

Release 0.5.0

Release 0.5.0:

Update README:
- with new version number
- Update links to quickstarts on readme
- Update example usage
After README is updated and merged, tag v0.5.0 from master branch

Issue with Install on NixOS

$  go get -v github.com/TileDB-Inc/TileDB-Go

github.com/TileDB-Inc/TileDB-Go
# github.com/TileDB-Inc/TileDB-Go
../../go/pkg/mod/github.com/!tile!d!b-!inc/[email protected]/array.go:6:10: fatal error: tiledb/tiledb.h: No such file or directory
    6 | #include <tiledb/tiledb.h>
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.

This could be some general cgo issue, but I'm struggling to find a solution.

Adding new Data

@Shelnutt2 @antalakas How can i append data to a tiledb using TileDB-Go , assume that i have 4 x 4 dense array matrix that was created in tiledb on disk , how can append a new 1 x 4 dense array into this tiledb database.

I am sorry, this is not an issue, but i created an issue because i don't know where should i write these questions.