awa-ai / awadb Goto Github PK

View Code? Open in Web Editor NEW

166.0 6.0 15.0 4.23 MB

AI Native database for embedding vectors

Home Page: https://ljeagle.github.io/awadb

License: Apache License 2.0

CMake 3.72% Python 23.75% C++ 70.06% C 0.30% Shell 0.11% Makefile 0.01% Dockerfile 0.02% Go 2.04%

ai-native aigc chatgpt embedding-vectors llm vectordb

awadb's People

Contributors

Stargazers

Watchers

Forkers

chinyy jiazhai gyd-a wangning19940904 lichao88 rouch789 taozhiwang ansjsun wukathy21 sarvex kkxiaotikk koy2000 ljeagle fellowtraveler wahello

awadb's Issues

restful get api not show vector field information even if vector field is specified in pack_fields

Error encountered by using awadb_client.Search

Hi!

I am having the following problem with awadb_client.Search. When I include the meta_filter in the call to search, it will report the error. Errors are reported at irregular intervals and there may be an underlying parallelism issue. Here is the code I am using to test.

import awadb
# 1. Initialize awadb client!
awadb_client = awadb.Client()

# 2. Create table!
awadb_client.Create("testdb")

# 3. Add docs to the table. Can also update and delete the doc!
awadb_client.Add([{'name':'jim'}, {'age':39}, {'desc':'hello'}, {'vec':[1, 3.5, 3]}])
awadb_client.Add([{'name':'vincent'}, {'age':28}, {'desc':'world'}, {'vec':[1, 3.4, 2]}])
awadb_client.Add([{'name':'david'}, {'age':45}, {'desc':'hi'},  {'vec':[1, 2.4, 4]}])
awadb_client.Add([{'name':'tom'}, {'age':25}, {'desc':'dolly'}, {'vec':[1.3, 2.9, 8.9]}])

# 4. Search by specified vector query and the most TopK similar result
results = awadb_client.Search("hello", 3, meta_filter={"max_age" : 30})

# Output the results
print(results)

Here are a few typical error messages I get. (It might have race conditions resulting from unsafe access to critical sections)

malloc error

Python(52184,0x1e12fe080) malloc: *** error for object 0x200000003: pointer being freed was not allocated
Python(52184,0x1e12fe080) malloc: *** set a breakpoint in malloc_error_break to debug
[1]    52184 abort      /opt/homebrew/bin/python3 
/opt/homebrew/Cellar/[email protected]/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Segmentation fault (noticed that it actually printed the correct output)

[{'ResultSize': 2, 'ResultItems': [{'vec': [1.0, 3.4000000953674316, 2.0], 'name': 'vincent', 'age': 28, 'desc': 'world'}, {'vec': [1.2999999523162842, 2.9000000953674316, 8.899999618530273], 'name': 'tom', 'age': 25, 'desc': 'dolly'}]}]
[1]    52496 segmentation fault  /opt/homebrew/bin/python3

bus error

/opt/homebrew/Cellar/[email protected]/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
[1]    52640 bus error  /opt/homebrew/bin/python3

add() function error

restful get api not show vector field information even if vector field is specified in pack_fields

Operation not permitted

version:
awadb v0.11
awadb-paoxy v-.11

I run command "docker-compose up" follow the describe: https://github.com/awa-ai/awadb/blob/main/docs/deploy.md，return error as flow:

docker-compose up

[+] Running 2/0
? Container composeyml-awadb-1 Created 0.0s
? Container composeyml-awadb_proxy-1 Created 0.0s
Attaching to composeyml-awadb-1, composeyml-awadb_proxy-1
composeyml-awadb-1 | OpenBLAS blas_thread_init: pthread_create failed for thread 1 of 4: Operation not permitted
composeyml-awadb-1 | OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
composeyml-awadb-1 | OpenBLAS blas_thread_init: pthread_create failed for thread 2 of 4: Operation not permitted
composeyml-awadb-1 | OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
composeyml-awadb-1 | OpenBLAS blas_thread_init: pthread_create failed for thread 3 of 4: Operation not permitted
composeyml-awadb-1 | OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
composeyml-awadb-1 | E1215 06:44:24.358007328 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.358239565 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.358372126 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.358839220 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.358862731 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.358891187 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.358909855 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.358928723 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.358951604 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.358972423 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.358991447 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.359009758 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.359030124 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.359049170 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.359067594 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.359085384 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.359104447 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.359124798 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.359148390 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.359204442 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | E1215 06:44:24.359271448 1 thd.cc:157] pthread_create failed: Operation not permitted
composeyml-awadb-1 | Server listening on 0.0.0.0:10000
composeyml-awadb_proxy-1 | runtime/cgo: pthread_create failed: Operation not permitted
composeyml-awadb_proxy-1 | SIGABRT: abort
composeyml-awadb_proxy-1 | PC=0x7f60e8ab9d3c m=0 sigcode=18446744073709551610
composeyml-awadb_proxy-1 |
composeyml-awadb_proxy-1 | goroutine 0 [idle]:
composeyml-awadb_proxy-1 | runtime: g 0: unknown pc 0x7f60e8ab9d3c
composeyml-awadb_proxy-1 | stack: frame={sp:0x7fff0ff2cc70, fp:0x0} stack=[0x7fff0f72e0f0,0x7fff0ff2d100)
composeyml-awadb_proxy-1 | 0x00007fff0ff2cb70: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cb80: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cb90: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cba0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cbb0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cbc0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cbd0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cbe0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cbf0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc00: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc10: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc20: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc30: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc40: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc50: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc60: 0x0000000000000000 0x00007f60e8ab9d2e
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc70: <0x0000000000000000 0xc7052ecdb7c77700
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc80: 0x0000000000000006 0x00007f60e8a2c740
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc90: 0x00007fff0ff2cf40 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cca0: 0x00000000010be9e0 0x00007f60e8a6af32
composeyml-awadb_proxy-1 | 0x00007fff0ff2ccb0: 0x00007f60e8c02e70 0x00007f60e8a55472
composeyml-awadb_proxy-1 | 0x00007fff0ff2ccc0: 0x0000000000000020 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2ccd0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cce0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2ccf0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cd00: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cd10: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cd20: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cd30: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cd40: 0x00007f60e8a2c740 0x00007f60e8aac87a
composeyml-awadb_proxy-1 | 0x00007fff0ff2cd50: 0x00007f60e8c02840 0xc7052ecdb7c77700
composeyml-awadb_proxy-1 | 0x00007fff0ff2cd60: 0x00007f60e8c02840 0x00007f60e8c02840
composeyml-awadb_proxy-1 | runtime: g 0: unknown pc 0x7f60e8ab9d3c
composeyml-awadb_proxy-1 | stack: frame={sp:0x7fff0ff2cc70, fp:0x0} stack=[0x7fff0f72e0f0,0x7fff0ff2d100)
composeyml-awadb_proxy-1 | 0x00007fff0ff2cb70: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cb80: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cb90: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cba0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cbb0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cbc0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cbd0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cbe0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cbf0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc00: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc10: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc20: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc30: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc40: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc50: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc60: 0x0000000000000000 0x00007f60e8ab9d2e
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc70: <0x0000000000000000 0xc7052ecdb7c77700
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc80: 0x0000000000000006 0x00007f60e8a2c740
composeyml-awadb_proxy-1 | 0x00007fff0ff2cc90: 0x00007fff0ff2cf40 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cca0: 0x00000000010be9e0 0x00007f60e8a6af32
composeyml-awadb_proxy-1 | 0x00007fff0ff2ccb0: 0x00007f60e8c02e70 0x00007f60e8a55472
composeyml-awadb_proxy-1 | 0x00007fff0ff2ccc0: 0x0000000000000020 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2ccd0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cce0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2ccf0: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cd00: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cd10: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cd20: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cd30: 0x0000000000000000 0x0000000000000000
composeyml-awadb_proxy-1 | 0x00007fff0ff2cd40: 0x00007f60e8a2c740 0x00007f60e8aac87a
composeyml-awadb_proxy-1 | 0x00007fff0ff2cd50: 0x00007f60e8c02840 0xc7052ecdb7c77700
composeyml-awadb_proxy-1 | 0x00007fff0ff2cd60: 0x00007f60e8c02840 0x00007f60e8c02840
composeyml-awadb_proxy-1 |
composeyml-awadb_proxy-1 | goroutine 1 [running]:
composeyml-awadb_proxy-1 | runtime.systemstack_switch()
composeyml-awadb_proxy-1 | /usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc000110740 sp=0xc000110730 pc=0x46ae88
composeyml-awadb_proxy-1 | runtime.main()
composeyml-awadb_proxy-1 | /usr/local/go/src/runtime/proc.go:169 +0x6d fp=0xc0001107e0 sp=0xc000110740 pc=0x43d56d
composeyml-awadb_proxy-1 | runtime.goexit()
composeyml-awadb_proxy-1 | /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001107e8 sp=0xc0001107e0 pc=0x46ce61
composeyml-awadb_proxy-1 |
composeyml-awadb_proxy-1 | rax 0x0
composeyml-awadb_proxy-1 | rbx 0x1
composeyml-awadb_proxy-1 | rcx 0xffffffffffffffff
composeyml-awadb_proxy-1 | rdx 0x6
composeyml-awadb_proxy-1 | rdi 0x1
composeyml-awadb_proxy-1 | rsi 0x1
composeyml-awadb_proxy-1 | rbp 0x7f60e8a2c740
composeyml-awadb_proxy-1 | rsp 0x7fff0ff2cc70
composeyml-awadb_proxy-1 | r8 0x0
composeyml-awadb_proxy-1 | r9 0x73
composeyml-awadb_proxy-1 | r10 0x8
composeyml-awadb_proxy-1 | r11 0x246
composeyml-awadb_proxy-1 | r12 0x6
composeyml-awadb_proxy-1 | r13 0x0
composeyml-awadb_proxy-1 | r14 0x10be9e0
composeyml-awadb_proxy-1 | r15 0x6
composeyml-awadb_proxy-1 | rip 0x7f60e8ab9d3c
composeyml-awadb_proxy-1 | rflags 0x246
composeyml-awadb_proxy-1 | cs 0x33
composeyml-awadb_proxy-1 | fs 0x0
composeyml-awadb_proxy-1 | gs 0x0
composeyml-awadb_proxy-1 exited with code 2
composeyml-awadb_proxy-1 exited with code 2
composeyml-awadb_proxy-1 exited with code 2
composeyml-awadb_proxy-1 exited with code 2

How to load a previously saved vector database?

I save the text vector using the following code:

vector_store = AwaDB.from_documents(docs,self.embeddings, table_name="langchain_awadb",log_and_data_dir = vs_path)

When I want to reload the vector database before the vs_path path, I try the load() function and load_local() function, but both of them return a value of type bool.
How can I reload the previously saved vector database?

The created table is not returned by /list api

awadb version: v0.15
awadb_proxy version: v0.15

I send /list by postman，the created tables are not all retutned.

picture 1: I send request with only param db

picturn 2: I send request with params db and table

Add function error faced when using inside a for loop

I've noticed that when I use the Add function in a for loop, python crashes and reports an error.

print(texts)

awadb_client = awadb.Client()
awadb_client.Create("testdb1")


for text in texts:
    print([text])
    awadb_client.Add([text])

The code that led me to this error was as above.
Here are the error information:

[1]    82822 trace trap  /opt/homebrew/bin/python3

I use python 3.11.4 and awadb 0.3.9.

Here are parts of the variables' information:
texts is a list of string.
[text] is a list which only contain one string.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.