milvus-io / milvus Goto Github PK
View Code? Open in Web Editor NEWA cloud-native vector database, storage for next generation AI applications
Home Page: https://milvus.io
License: Apache License 2.0
A cloud-native vector database, storage for next generation AI applications
Home Page: https://milvus.io
License: Apache License 2.0
Is your feature request related to a problem? Please describe.
Some developers don't have Nvidia gpu card on their machine, they cannot compile/run Milvus.
Describe the solution you'd like
Use pre-compiled macro to disable gpu function code
Describe the bug
Server no response
Steps/Code to reproduce behavior
Create index: ivf_sq8h with SDK-python
Expected behavior
Error code, with message "not support"
Environment details
branch: 0.5.0/master
As title.
Describe the solution you'd like
Add ELK for Jenkins CI
Is your feature request related to a problem? Please describe.
To test original FAISS benchmark.
Describe the solution you'd like
Add this benchmark under core/index/unittest
Describe alternatives you've considered
N/A
Additional context
test SQ8/SQ8H with L2/IP
Describe the bug
[ PASSED ] 39 tests. [ FAILED ] 1 test, listed below: [ FAILED ] DBTest2.DELETE_BY_RANGE_TEST 1 FAILED TEST
Expected behavior
All unit test pass
Environment details
CI environment
Screenshots
Additional context
Describe the bug
server_test failed in ConfigTest
Steps/Code to reproduce behavior
Run server_test in cmdline, it failed
Is your feature request related to a problem? Please describe.
I wish I could use Milvus to create partition within a table. User can insert vectors into a partition by specifying a partition tag, and search vectors from a certain partition of table. Partition can be dropped.
Describe the solution you'd like
to be discussed..
Additional context
Related issue:
#28 Add new api about vector deletion via generated date not insert date
Describe the solution you'd like
Add Codecov Yaml file
Describe the bug
The server start error messages lack vital information for troubleshooting.
Expected behavior
Current error message | Suggested error message |
---|---|
ERROR: mode specified in server_config is not one of ['single', 'cluster', 'read_only'] | Error: server_config.deploy_mode is not one of single, cluster_readonly, and cluster_writable. |
ERROR! Failed to create database root path: {path} | Error: Failed to create database primary path: {path}. Possible reason: db_config.primary_path is wrong or not available. |
ERROR! Failed to create database slave path: | Error: Failed to create database secondary path:{path}. Possible reason: db_config.secondary_path is wrong or not available. |
ERROR! Failed to open database: {error message} | Error: Failed to open database. Possible reason: {error message} |
ERROR: invalid server IP address: {it} | Error: Invalid server IP address: {address}. Possible reason: server_config.address is invalid. |
ERROR: port xxx is not a number | Error: Port {port} is not a number. Possible reason: server_config.port is invalid. |
ERROR: port xxx out of range [1025, 65534] | Error: Port {port} is not in range [1025, 65534]. Possible reason: server_config.port is invalid. |
ERROR: db_path is empty | Error: db_path is empty. Possible reason: db_config.db_path is empty. |
ERROR: invalid db_backend_url: {url} | Error: Invalid db_backend_url: {url}. Possible reason: db_config.db_backend_url is invalid. The correct format should be like sqlite://:@:/ or mysql://root:[email protected]:3306/milvus. |
ERROR: Invalid server config time_zone: {time zone} | ERROR: Invalid server config time_zone: {time zone} |
ERROR: Invalid insert_buffer_size: {buffer size} | ERROR: Invalid insert buffer size: {buffer size}. Possible reason: db_config.insert_buffer_sizel is not a positive integer. |
ERROR: Invalid metric config auto_bootup: {enable metric} | ERROR: Invalid metric config: {enable metric}. Possible reason: metric_config.enable_monitor is not a boolean. |
ERROR: Invalid metric collector: {metric collector} | ERROR: Invalid metric config: {metric collector}. Possible reason: metric_config.collector is invalid. |
ERROR: Invalid metric config prometheus_port: {port} | ERROR: Invalid metric config: {port}. Possible reason: metric_config.prometheus_config.port is not in range [1025, 65534]. |
ERROR: Invalid cache config cpu_cache_capacity: {capacity} | ERROR: Invalid cpu cache capacity: {capacity}. Possible reason: cache_config.cpu_cache_capacity is not a positive integer. |
ERROR: Cache config cpu_cache_capacity exceed system memory: | ERROR: Invalid cpu cache capacity: {capacity}. Possible reason: cache_config.cpu_cache_capacity exceeds system memory. |
ERROR: Sum of cpu_cache_capacity and buffer_size exceed system memory | ERROR: Invalid cpu cache capacity: {capacity}. Possible reason: sum of cache_config.cpu_cache_capacity and db_config.insert_buffer_size exceeds system memory. |
ERROR: Invalid cache config cpu_cache_threshold: {threshold} | ERROR: Invalid cpu cache threshold: {threshold}. Possible reason: cache_config.cpu_cache_threshold is not in range (0.0, 1.0]. |
ERROR: Invalid cache config gpu_cache_capacity: {capacity} | ERROR: Invalid gpu cache capacity: {capacity}. Possible reason: cache_config.gpu_cache_capacity is not a positive integer. |
ERROR: Fail to get GPU memory for GPU device: {device} | ERROR: Fail to get GPU memory for GPU device: {device} |
ERROR: Cache config gpu_cache_capacity exceed GPU memory: {capacity} | ERROR: Invalid gpu cache capacity: {capacity}. Possible reason: cache_config.gpu_cache_capacity exceeds GPU memory. |
ERROR: Invalid cache config gpu_cache_threshold: {threshold} | ERROR: Invalid gpu cache threshold: {threshold}. Possible reason: cache_config.gpu_cache_threshold is not in range (0.0, 1.0]. |
ERROR: Invalid cache config cache_insert_data: {bool} | ERROR: Invalid cache insert option: {bool}. Possible reason: cache_config.cache_insert_data is not a boolean. |
ERROR: Invalid engine config use_blas_threshold: {threshold} | ERROR: Invalid blas threshold: {threshold}. Possible reason: engine_config.use_blas_threshold is not a positive integer. |
ERROR: Invalid gpu device: {device} | ERROR: Invalid gpu device: {device}. Possible reason: resource_config.search_resources does not match your hardware. |
ERROR: Empty resource config search_resources | ERROR: Invalid search resource. Possible reason: resource_config.search_resources is empty. |
ERROR: Invalid resource config index_build_device: {device} | ERROR: Invalid index build device: {device}. Possible reason: resource_config.index_build_device does not match your hardware. |
"Thanks" section in this project looks very well. Maybe we can do it in our readme.
https://github.com/nlohmann/json
Add dumpable interface, make class inherit from it.
Describe the solution you'd like
Add code coverage for Milvus Jenkins CI
Release 0.5.1 need new CHANGELOG.md
I have add milvus into codebeat. Now I need add a codebeat icon into README.md
Describe the bug
Expected behavior
Make sure that a container does not exist that is using the image.
Describe the bug
Start Milvus server, then run C++ sdk example(sdk_simple) from command line. The example program get grpc error.
Steps/Code to reproduce behavior
1 .Start Milvus server
2. Run C++ sdk example(sdk_simple) from command line
Expected behavior
The example should run successfully.
Environment details
Is your feature request related to a problem? Please describe.
I wish I could use Milvus to delete vectors via their generated date,the generated date means vector's actual production date that different from the vector's insert date.
For example : Some vectors come from pictures which generated between 2019-05-01 and 2019-05-25 , but we import the vectors to milvus database at 2019-06-01. If we want delete the vectors between 2019-05-01 and 2019-05-10 , existing apis can not support.
Describe the solution you'd like
Add a api which like "milvus.delete_vectors_by_range('test01', '2019-06-01', '2020-01-01')" but the date means vector's actual production date not vector's insert date.
release notes 链接无效。
Step 1 Install dependencies
$ cd [Milvus sourcecode path]/core
./ubuntu_build_deps.sh
第二行是不是缺了一个$
?
Is your feature request related to a problem? Please describe.
Default gpu_cache_capacity is 0,it makes SQ8H index work slow.
Describe the solution you'd like
Set default gpu_cache_capacity in server_config.template.
Is your feature request related to a problem? Please describe.
I wish I could know Milvus version from log files.
Describe the solution you'd like
Print version information into global.log file in Server::Start()
Describe the bug
Topk result is incorrect for small dataset.
Steps/Code to reproduce behavior
Expected behavior
the topk result for no.2 vector should be like no.1
Additional context
Change k from 10 to 5, the result is correct
Describe the solution you'd like
Create Mysql container on Jenkins CI build stage
Is your feature request related to a problem? Please describe.
Build time is too long.
Describe the solution you'd like
Remove some redundant third party packages in Milvus
Change Arrow's configure arguments and use AUTO approach to build Boost
Is your feature request related to a problem? Please describe.
I wish I could use Milvus to find and delete vector by it's corresponding ID. Currently the SDK supports only removing the vectors using the date-time, which is a little bit useless for the following cases; suppose I've inserted a huge dataset of animals and the last image vector of a person was mistakenly added, so in order to remove the last vector I need to remove the whole table.
Describe the solution you'd like
Delete vector by its corresponding ID.
Describe alternatives you've considered
Actually this feature is already implemented in FAISS search engine and the other ones. I think you know that it is possible to remove the ID from the index, using remove_ids() in FAISS. Is this possible to add the same feature for the Milvus?
Additional context
https://github.com/facebookresearch/faiss/wiki/Special-operations-on-indexes
Describe the bug
-rw-r--r-- 1 root root Oct 13 05:36 libfaiss.a
-rw-r--r-- 1 root root Oct 19 11:49 libknowhere.a
remove these files in compiled libs
Steps/Code to reproduce behavior
N/A
Expected behavior
No more files not related to milvus-libs after build
Environment details
N/A
Screenshots
N/A
Additional context
N/A
Describe the bug
make clang-format failed after run build.sh -l
Steps/Code to reproduce behavior
cd milvus/core
build.sh -l
cd cmake_build
make clang-format
result:failed to format the code
Expected behavior
make clang-format successfully after build.sh -l
Is your feature request related to a problem? Please describe.
Currently Milvus keep origin data in disk so that user can change index type freely. But it cost too much disk space. I wish I could drop origin data after index is built successfully.
Describe the solution you'd like
Add a parameter in create_index sdk api. "drop_origin"(boolean), default is false.
If create_index(drop_origin=true), the table only contain index files, and user cannot specify any other index type. Call drop_index to this table, the table will become an empty table.
If create_index(drop_origin=false), keep old behavior.
Describe the bug
Now, each CI test triggered by PR never pass totally.
Steps/Code to reproduce behavior
Just issue a PR without 'skip ci' message
Expected behavior
Normally, all cases can pass now.
Environment details
CI environment
Screenshots
N/A
Additional context
N/A
Describe the bug
Some troubleshoot messages in the Milvus software do not provide enough information. Users still need to refer to the documentation for details. This might have a negative effect on user experience. We need to improve these messages in the software.
Expected behavior
Please refer to the suggested text below:
Topic | Old Message | New Message |
---|---|---|
General | Invalid table name: xxx | Invalid table name: xxx. A table name can only contain numbers, letters, and underscores. The first character of a table name must not be a number. The length of a table name must be less than 255 characters. |
General | Table xxx not exist | Table xxx does not exist. Use milvus.has_table to verify whether the table exists. You also can check if the table name exists. |
CreateTable | Invalid table dimension: xxx | Invalid table dimension: xxx. The table dimension must be within the range of 1 ~ 16384. |
CreateTable | Invalid index file size: xxx | Invalid index file size: xxx. The index file size must be within the range of 1 ~ 4096. |
CreateTable | Invalid index metric type: xxx | Invalid index metric type: xxx. Make sure the metric type is either MetricType.L2 or MetricType.IP. |
CreateIndex | Invalid index type: xxx | Invalid index type: xxx. Make sure the index type is among FLAT, IVFLAT, and IVF_SQ8. |
CreateIndex | Invalid index nlist: xxx | Invalid index nlist: xxx. The index nlist must be greater than 0. |
Insert | Row record array is empty | The row record array is empty. Make sure you have entered vector records. |
Insert | Size of vector ids is not equal to row record array size | The size of vector ID array must be equal to the size of the vector. |
Insert | Table vector ids are user defined, please provide id for this batch | Table vector IDs are user-defined. Please provide IDs for all vectors of this table. |
Insert | Table vector ids are auto generated, no need to provide id for this batch | Table vector IDs are auto-generated. All vectors of this table must use auto-generated IDs. |
Insert | Row record float array is empty | The row record float array must not be empty. |
Insert | Invalid row record dimension: xxx vs. table dimension: xxx | The row record dimension must be equal to the table dimension. |
Search | Invalid topk: xxx | Invalid topk: xxx. The topk must be within the range of 1~2048. |
Search | Invalid nprobe: xxx | Invalid nprobe: xxx. The nprobe must be within the range of 1 ~ index nlist. |
Search | Query record float array is empty | The query record float array is empty. Make sure the vectors you want to search have values. |
Search | Invalid query record dimension: xxx vs. table dimension: xxx | The vector dimension must be equal to the table dimension. |
Refer to Milvus Documentation for all troubleshooting messages documented in the current release.
Note that the original descriptions are based on the current Milvus documentation. If there are any software changes that are not captured by the documentation or the suggested text is not technically accurate, please let me know.
Describe the solution you'd like
Add Ubuntu18.04 Build and Runtime environment
Is your feature request related to a problem? Please describe.
I wish I could use Milvus to check meta version before server start. For old version meta, if it is migrate-able, Milvus migrate it automatically, otherwise, Milvus should give an error message and exit.
Describe the solution you'd like
**Is your feature request related to a problem?
No
Describe the solution you'd like
Just involve all tests in projects
Describe alternatives you've considered
May create another REPO to store test cases, but in same REPO will be much earier to synchronize server/test version.
Additional context
N/A
Describe the bug
Memory usage increased slowly during searching vectors
Steps/Code to reproduce behavior
Keep searching about 1000 times
Expected behavior
Memory usage keep stable
Environment details
N/A
Screenshots
N/A
Additional context
version 0.5.1 and version before 0.5.1
Location of incorrect documentation
README
Describe the problems or issues found in the documentation
Code format and code coverage should be in CONTRIBUTING.md
Suggested fix for documentation
Move the content from README to CONTRIBUTING
Describe the bug
test_scheduler core_dump when run ./codecoverage
Steps/Code to reproduce behavior
NA
Expected behavior
NA
Environment details
NA
Screenshots
NA
Additional context
NA
Describe the solution you'd like
Add Jenkins CI for Milvus
Describe the bug
A clear and concise description of what the bug is.
Steps/Code to reproduce behavior
Follow this guide to craft a minimal bug report. This helps us reproduce the issue you're having and resolve the issue more quickly.
Expected behavior
A clear and concise description of what you expected to happen.
Environment details
server_config.yaml
)Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Describe the bug
Get github milvus, build and run, create SQ8H index, milvus hangs.
Steps/Code to reproduce behavior
Get github milvus, build and run, create SQ8H index, milvus hangs.
Expected behavior
Milvus should give a message 'Unsupported index type' for this case
Describe the bug
Currently, whole compiling time is too long. Downloading boost library source code is one of the reasons.
Steps/Code to reproduce behavior
Each clean compiling
Expected behavior
Shorten the compiling time.
There is a directory named "external" used for external project which source code embedded in project.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.