Comments (3)
我也遇到了一样的问题了,INFO:debug_logger:milvus delete files_id: ['804d5229f04d4fdbb76fa565a53aa994']
INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(**http://es-container-local:9200**)> (force=False)
INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544 [status:200 duration:0.002s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has been marked alive after a successful request
INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)> (force=False)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.002s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 1 times in a row, putting on 1 second timeout
WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3)
INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)> (force=False)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.001s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 2 times in a row, putting on 2 second timeout
WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.001s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 3 times in a row, putting on 4 second timeout
WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.002s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 4 times in a row, putting on 8 second timeout
[2024-07-10 14:51:31 +0800] [678] [ERROR] Exception occurred while handling uri: 'http://10.10.0.232:8777/api/local_doc_qa/delete_files'
http://es-container-local:9200这个链接一看就是错,还有milvus的容器,用docker ps看,是没有暴露端口的,我也想不明白为什么
from qanything.
我也遇到了同样的问题,请问如何解决
from qanything.
我也遇到了一样的问题了,INFO:debug_logger:milvus delete files_id: ['804d5229f04d4fdbb76fa565a53aa994'] INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(**http://es-container-local:9200**)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544 [status:200 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout [2024-07-10 14:51:31 +0800] [678] [ERROR] Exception occurred while handling uri: 'http://10.10.0.232:8777/api/local_doc_qa/delete_files'
http://es-container-local:9200这个链接一看就是错,还有milvus的容器,用docker ps看,是没有暴露端口的,我也想不明白为什么
是否已有关于该错误的issue或讨论? | Is there an existing issue / discussion for this?
* [x] 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions
该问题是否在FAQ中有解答? | Is there an existing answer for this in FAQ?
* [x] 我已经搜索过FAQ | I have searched FAQ
当前行为 | Current Behavior
按照5.17日的安装文档,docker部署,run.sh安装未报错,启动容器终端输出未见异常,访问UI可以正常访问,遇见两个问题: 1.无法正常上传文件,上传第一个文件一直报错:milvus插入失败,请稍后再试,第二个第三个没问题,但出现该问题后,该知识库无法正常删除,报错请求失败,该文件同样无法删除,docx,ppt,txt均尝试了。 2.无法正常问答,问答一直报错请刷新后重试。
测试了ollama,本地Qwen,应该不是模型问题,看sanic.log,问题应该出在es上面。
期望行为 | Expected Behavior
解决BUG
运行环境 | Environment
- OS:Centos8 - NVIDIA Driver:535 - CUDA:11.8 - docker:2.24.0 - docker-compose:2.27.0 - NVIDIA GPU:4090 - NVIDIA GPU Memory:24GB
QAnything日志 | QAnything logs
新建知识库时应该还是正常的:
INFO:debug_logger:check_user_exist [] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB7e4842c2615c40649e78754c8e39203b', 'kb_name': '默认知识库'}] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}
随后在插入文件时发现有一个es的404(已用########突出):
INFO:debug_logger:XXX-wxs.pptx, cd4eca5871ac4bb7a43fb7ca0058815d, success INFO:debug_logger:success init localfile XXX-wxs.pptx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:debug_logger:before 2nd split doc lens: 53 INFO:debug_logger:after 2nd split doc lens: 44 INFO:debug_logger:langchain analysis content head: XXX INFO:debug_logger:split time: 0.3504297733306885 44 INFO:debug_logger:embedding number: 3
0%| | 0/3 [00:00<?, ?it/s] 33%|███▎ | 1/3 [00:00<00:00, 8.08it/s] 100%|██████████| 3/3 [00:00<00:00, 24.23it/s] INFO:debug_logger:embedding time: 0.12532377243041992 44 INFO:debug_logger:now inser_file XXX-wxs.pptx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:XXX-wxs.pptx (insert count: 44, delete count: 0, upsert count: 0, timestamp: 450132530566529026, success count: 44, err count: 0) INFO:debug_logger:now inser_file for es: XXX-wxs.pptx INFO:debug_logger:Inserting into es ... #############INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:404 duration:0.003s] INFO:debug_logger:##ES## - Creating index zzp++kb6fda41a1c95f4605ab812263d921740a with: mappings: {'properties': {'file_id': {'type': 'keyword', 'index': True}, 'content': {'type': 'text', 'similarity': 'custom_bm25', 'index': True, 'analyzer': 'ik_smart', 'search_analyzer': 'ik_smart'}}} settings: {'index': {'similarity': {'custom_bm25': {'type': 'BM25', 'k1': '1.3', 'b': '0.6'}}}} INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:N/A duration:10.010s] ##############WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout ERROR:debug_logger:ES insert file_id: cd4eca5871ac4bb7a43fb7ca0058815d file_name: XXX-wxs.pptx failed: Connection timed out INFO:debug_logger:insert time: 10.019953966140747 INFO:debug_logger:insert_to_milvus: success num: 0, failed num: 1 INFO:debug_logger:upload_files zzp INFO:debug_logger:mode: strong WARNING:root:missing use_local_file in request INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)]
第二次再次插入一个文档,前端UI显示成功,但日志中有一堆warning:
INFO:debug_logger:ori name: 离线部署centos文档.docx INFO:debug_logger:decode name: 离线部署centos文档.docx INFO:debug_logger:cleaned name: 离线部署centos文档.docx INFO:debug_logger:check_user_exist [('zzp',)] INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:add_file: bbd20c38f53b473babbafe01f5aab6b7 INFO:debug_logger:离线部署centos文档.docx, bbd20c38f53b473babbafe01f5aab6b7, success INFO:debug_logger:success init localfile 离线部署centos文档.docx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:before 2nd split doc lens: 170 INFO:debug_logger:after 2nd split doc lens: 170 INFO:debug_logger:langchain analysis content head: 搭建私有yum仓库,安装gcc gcc-c++,ffmpeg(三步法),nginx(nginx在centos8. INFO:debug_logger:split time: 0.04621243476867676 170 INFO:debug_logger:embedding number: 11
0%| | 0/11 [00:00<?, ?it/s] 100%|██████████| 11/11 [00:00<00:00, 286.36it/s] INFO:debug_logger:embedding time: 0.04653787612915039 170 INFO:debug_logger:now inser_file 离线部署centos文档.docx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:离线部署centos文档.docx (insert count: 170, delete count: 0, upsert count: 0, timestamp: 450132533227028483, success count: 170, err count: 0) INFO:debug_logger:now inser_file for es: 离线部署centos文档.docx INFO:debug_logger:Inserting into es ... INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.003s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:debug_logger:##ES## - Index zzp++kb6fda41a1c95f4605ab812263d921740a already exists. Skipping creation. INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base zzp INFO:debug_logger:check_kb_exist [('KB7e4842c2615c40649e78754c8e39203b',)] INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:elastic_transport.transport:DELETE http://es-container-local:9200/zzp%2B%2Bkb7e4842c2615c40649e78754c8e39203b?ignore_unavailable=true [status:200 duration:0.001s] INFO:debug_logger:##ES## - success to delete index: ['zzp++kb7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:##ES## - success delete kb_ids: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB6fda41a1c95f4605ab812263d921740a', 'kb_name': 'aaa'}] INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_bulk?refresh=false [status:N/A duration:10.010s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout INFO:debug_logger:离线部署centos文档.docx Error adding texts: Connection timed out INFO:debug_logger:insert time: 10.025246858596802 INFO:debug_logger:insert_to_milvus: success num: 1, failed num: 0 INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:delete_docs zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:check_file_exist [('cd4eca5871ac4bb7a43fb7ca0058815d', 'yellow')] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:milvus delete files_id: ['cd4eca5871ac4bb7a43fb7ca0058815d'] INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.006s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout
随后删除报错log如下:
[2024-05-31 09:35:57 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)
尝试问答,报错log如下:
INFO:debug_logger:local_doc_chat zzp INFO:debug_logger:rerank True INFO:debug_logger:history: [] INFO:debug_logger:question: 我想知道 INFO:debug_logger:kb_ids: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:debug_logger:user_id: zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:streaming: True INFO:debug_logger:start generate answer INFO:debug_logger:start generate... INFO:debug_logger:embedding number: 1 函数 get_knowledge_based_answer 执行耗时: 7.152557373046875e-07 秒
0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:00<00:00, 52.80it/s] INFO:debug_logger:milvus group number: 1 INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout [2024-05-31 09:36:07 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)
复现方法 | Steps To Reproduce
按照文档下载源码,bash run.sh -c cloud -i 0 下载镜像,起容器,出现上述问题
备注 | Anything else?
No response
是否已有关于该错误的issue或讨论? | Is there an existing issue / discussion for this?
* [x] 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions
该问题是否在FAQ中有解答? | Is there an existing answer for this in FAQ?
* [x] 我已经搜索过FAQ | I have searched FAQ
当前行为 | Current Behavior
按照5.17日的安装文档,docker部署,run.sh安装未报错,启动容器终端输出未见异常,访问UI可以正常访问,遇见两个问题: 1.无法正常上传文件,上传第一个文件一直报错:milvus插入失败,请稍后再试,第二个第三个没问题,但出现该问题后,该知识库无法正常删除,报错请求失败,该文件同样无法删除,docx,ppt,txt均尝试了。 2.无法正常问答,问答一直报错请刷新后重试。
测试了ollama,本地Qwen,应该不是模型问题,看sanic.log,问题应该出在es上面。
期望行为 | Expected Behavior
解决BUG
运行环境 | Environment
- OS:Centos8 - NVIDIA Driver:535 - CUDA:11.8 - docker:2.24.0 - docker-compose:2.27.0 - NVIDIA GPU:4090 - NVIDIA GPU Memory:24GB
QAnything日志 | QAnything logs
新建知识库时应该还是正常的:
INFO:debug_logger:check_user_exist [] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB7e4842c2615c40649e78754c8e39203b', 'kb_name': '默认知识库'}] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}
随后在插入文件时发现有一个es的404(已用########突出):
INFO:debug_logger:XXX-wxs.pptx, cd4eca5871ac4bb7a43fb7ca0058815d, success INFO:debug_logger:success init localfile XXX-wxs.pptx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:debug_logger:before 2nd split doc lens: 53 INFO:debug_logger:after 2nd split doc lens: 44 INFO:debug_logger:langchain analysis content head: XXX INFO:debug_logger:split time: 0.3504297733306885 44 INFO:debug_logger:embedding number: 3
0%| | 0/3 [00:00<?, ?it/s] 33%|███▎ | 1/3 [00:00<00:00, 8.08it/s] 100%|██████████| 3/3 [00:00<00:00, 24.23it/s] INFO:debug_logger:embedding time: 0.12532377243041992 44 INFO:debug_logger:now inser_file XXX-wxs.pptx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:XXX-wxs.pptx (insert count: 44, delete count: 0, upsert count: 0, timestamp: 450132530566529026, success count: 44, err count: 0) INFO:debug_logger:now inser_file for es: XXX-wxs.pptx INFO:debug_logger:Inserting into es ... #############INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:404 duration:0.003s] INFO:debug_logger:##ES## - Creating index zzp++kb6fda41a1c95f4605ab812263d921740a with: mappings: {'properties': {'file_id': {'type': 'keyword', 'index': True}, 'content': {'type': 'text', 'similarity': 'custom_bm25', 'index': True, 'analyzer': 'ik_smart', 'search_analyzer': 'ik_smart'}}} settings: {'index': {'similarity': {'custom_bm25': {'type': 'BM25', 'k1': '1.3', 'b': '0.6'}}}} INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:N/A duration:10.010s] ##############WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout ERROR:debug_logger:ES insert file_id: cd4eca5871ac4bb7a43fb7ca0058815d file_name: XXX-wxs.pptx failed: Connection timed out INFO:debug_logger:insert time: 10.019953966140747 INFO:debug_logger:insert_to_milvus: success num: 0, failed num: 1 INFO:debug_logger:upload_files zzp INFO:debug_logger:mode: strong WARNING:root:missing use_local_file in request INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)]
第二次再次插入一个文档,前端UI显示成功,但日志中有一堆warning:
INFO:debug_logger:ori name: 离线部署centos文档.docx INFO:debug_logger:decode name: 离线部署centos文档.docx INFO:debug_logger:cleaned name: 离线部署centos文档.docx INFO:debug_logger:check_user_exist [('zzp',)] INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:add_file: bbd20c38f53b473babbafe01f5aab6b7 INFO:debug_logger:离线部署centos文档.docx, bbd20c38f53b473babbafe01f5aab6b7, success INFO:debug_logger:success init localfile 离线部署centos文档.docx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:before 2nd split doc lens: 170 INFO:debug_logger:after 2nd split doc lens: 170 INFO:debug_logger:langchain analysis content head: 搭建私有yum仓库,安装gcc gcc-c++,ffmpeg(三步法),nginx(nginx在centos8. INFO:debug_logger:split time: 0.04621243476867676 170 INFO:debug_logger:embedding number: 11
0%| | 0/11 [00:00<?, ?it/s] 100%|██████████| 11/11 [00:00<00:00, 286.36it/s] INFO:debug_logger:embedding time: 0.04653787612915039 170 INFO:debug_logger:now inser_file 离线部署centos文档.docx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:离线部署centos文档.docx (insert count: 170, delete count: 0, upsert count: 0, timestamp: 450132533227028483, success count: 170, err count: 0) INFO:debug_logger:now inser_file for es: 离线部署centos文档.docx INFO:debug_logger:Inserting into es ... INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.003s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:debug_logger:##ES## - Index zzp++kb6fda41a1c95f4605ab812263d921740a already exists. Skipping creation. INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base zzp INFO:debug_logger:check_kb_exist [('KB7e4842c2615c40649e78754c8e39203b',)] INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:elastic_transport.transport:DELETE http://es-container-local:9200/zzp%2B%2Bkb7e4842c2615c40649e78754c8e39203b?ignore_unavailable=true [status:200 duration:0.001s] INFO:debug_logger:##ES## - success to delete index: ['zzp++kb7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:##ES## - success delete kb_ids: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB6fda41a1c95f4605ab812263d921740a', 'kb_name': 'aaa'}] INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_bulk?refresh=false [status:N/A duration:10.010s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout INFO:debug_logger:离线部署centos文档.docx Error adding texts: Connection timed out INFO:debug_logger:insert time: 10.025246858596802 INFO:debug_logger:insert_to_milvus: success num: 1, failed num: 0 INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:delete_docs zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:check_file_exist [('cd4eca5871ac4bb7a43fb7ca0058815d', 'yellow')] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:milvus delete files_id: ['cd4eca5871ac4bb7a43fb7ca0058815d'] INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.006s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout
随后删除报错log如下:
[2024-05-31 09:35:57 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)
尝试问答,报错log如下:
INFO:debug_logger:local_doc_chat zzp INFO:debug_logger:rerank True INFO:debug_logger:history: [] INFO:debug_logger:question: 我想知道 INFO:debug_logger:kb_ids: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:debug_logger:user_id: zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:streaming: True INFO:debug_logger:start generate answer INFO:debug_logger:start generate... INFO:debug_logger:embedding number: 1 函数 get_knowledge_based_answer 执行耗时: 7.152557373046875e-07 秒
0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:00<00:00, 52.80it/s] INFO:debug_logger:milvus group number: 1 INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout [2024-05-31 09:36:07 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)
复现方法 | Steps To Reproduce
按照文档下载源码,bash run.sh -c cloud -i 0 下载镜像,起容器,出现上述问题
备注 | Anything else?
_No respons
是否已有关于该错误的issue或讨论? | Is there an existing issue / discussion for this?
* [x] 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions
该问题是否在FAQ中有解答? | Is there an existing answer for this in FAQ?
* [x] 我已经搜索过FAQ | I have searched FAQ
当前行为 | Current Behavior
按照5.17日的安装文档,docker部署,run.sh安装未报错,启动容器终端输出未见异常,访问UI可以正常访问,遇见两个问题: 1.无法正常上传文件,上传第一个文件一直报错:milvus插入失败,请稍后再试,第二个第三个没问题,但出现该问题后,该知识库无法正常删除,报错请求失败,该文件同样无法删除,docx,ppt,txt均尝试了。 2.无法正常问答,问答一直报错请刷新后重试。
测试了ollama,本地Qwen,应该不是模型问题,看sanic.log,问题应该出在es上面。
期望行为 | Expected Behavior
解决BUG
运行环境 | Environment
- OS:Centos8 - NVIDIA Driver:535 - CUDA:11.8 - docker:2.24.0 - docker-compose:2.27.0 - NVIDIA GPU:4090 - NVIDIA GPU Memory:24GB
QAnything日志 | QAnything logs
新建知识库时应该还是正常的:
INFO:debug_logger:check_user_exist [] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB7e4842c2615c40649e78754c8e39203b', 'kb_name': '默认知识库'}] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}
随后在插入文件时发现有一个es的404(已用########突出):
INFO:debug_logger:XXX-wxs.pptx, cd4eca5871ac4bb7a43fb7ca0058815d, success INFO:debug_logger:success init localfile XXX-wxs.pptx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:debug_logger:before 2nd split doc lens: 53 INFO:debug_logger:after 2nd split doc lens: 44 INFO:debug_logger:langchain analysis content head: XXX INFO:debug_logger:split time: 0.3504297733306885 44 INFO:debug_logger:embedding number: 3
0%| | 0/3 [00:00<?, ?it/s] 33%|███▎ | 1/3 [00:00<00:00, 8.08it/s] 100%|██████████| 3/3 [00:00<00:00, 24.23it/s] INFO:debug_logger:embedding time: 0.12532377243041992 44 INFO:debug_logger:now inser_file XXX-wxs.pptx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:XXX-wxs.pptx (insert count: 44, delete count: 0, upsert count: 0, timestamp: 450132530566529026, success count: 44, err count: 0) INFO:debug_logger:now inser_file for es: XXX-wxs.pptx INFO:debug_logger:Inserting into es ... #############INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:404 duration:0.003s] INFO:debug_logger:##ES## - Creating index zzp++kb6fda41a1c95f4605ab812263d921740a with: mappings: {'properties': {'file_id': {'type': 'keyword', 'index': True}, 'content': {'type': 'text', 'similarity': 'custom_bm25', 'index': True, 'analyzer': 'ik_smart', 'search_analyzer': 'ik_smart'}}} settings: {'index': {'similarity': {'custom_bm25': {'type': 'BM25', 'k1': '1.3', 'b': '0.6'}}}} INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:N/A duration:10.010s] ##############WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout ERROR:debug_logger:ES insert file_id: cd4eca5871ac4bb7a43fb7ca0058815d file_name: XXX-wxs.pptx failed: Connection timed out INFO:debug_logger:insert time: 10.019953966140747 INFO:debug_logger:insert_to_milvus: success num: 0, failed num: 1 INFO:debug_logger:upload_files zzp INFO:debug_logger:mode: strong WARNING:root:missing use_local_file in request INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)]
第二次再次插入一个文档,前端UI显示成功,但日志中有一堆warning:
INFO:debug_logger:ori name: 离线部署centos文档.docx INFO:debug_logger:decode name: 离线部署centos文档.docx INFO:debug_logger:cleaned name: 离线部署centos文档.docx INFO:debug_logger:check_user_exist [('zzp',)] INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:add_file: bbd20c38f53b473babbafe01f5aab6b7 INFO:debug_logger:离线部署centos文档.docx, bbd20c38f53b473babbafe01f5aab6b7, success INFO:debug_logger:success init localfile 离线部署centos文档.docx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:before 2nd split doc lens: 170 INFO:debug_logger:after 2nd split doc lens: 170 INFO:debug_logger:langchain analysis content head: 搭建私有yum仓库,安装gcc gcc-c++,ffmpeg(三步法),nginx(nginx在centos8. INFO:debug_logger:split time: 0.04621243476867676 170 INFO:debug_logger:embedding number: 11
0%| | 0/11 [00:00<?, ?it/s] 100%|██████████| 11/11 [00:00<00:00, 286.36it/s] INFO:debug_logger:embedding time: 0.04653787612915039 170 INFO:debug_logger:now inser_file 离线部署centos文档.docx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:离线部署centos文档.docx (insert count: 170, delete count: 0, upsert count: 0, timestamp: 450132533227028483, success count: 170, err count: 0) INFO:debug_logger:now inser_file for es: 离线部署centos文档.docx INFO:debug_logger:Inserting into es ... INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.003s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:debug_logger:##ES## - Index zzp++kb6fda41a1c95f4605ab812263d921740a already exists. Skipping creation. INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base zzp INFO:debug_logger:check_kb_exist [('KB7e4842c2615c40649e78754c8e39203b',)] INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:elastic_transport.transport:DELETE http://es-container-local:9200/zzp%2B%2Bkb7e4842c2615c40649e78754c8e39203b?ignore_unavailable=true [status:200 duration:0.001s] INFO:debug_logger:##ES## - success to delete index: ['zzp++kb7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:##ES## - success delete kb_ids: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB6fda41a1c95f4605ab812263d921740a', 'kb_name': 'aaa'}] INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_bulk?refresh=false [status:N/A duration:10.010s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout INFO:debug_logger:离线部署centos文档.docx Error adding texts: Connection timed out INFO:debug_logger:insert time: 10.025246858596802 INFO:debug_logger:insert_to_milvus: success num: 1, failed num: 0 INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:delete_docs zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:check_file_exist [('cd4eca5871ac4bb7a43fb7ca0058815d', 'yellow')] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:milvus delete files_id: ['cd4eca5871ac4bb7a43fb7ca0058815d'] INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.006s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout
随后删除报错log如下:
[2024-05-31 09:35:57 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)
尝试问答,报错log如下:
INFO:debug_logger:local_doc_chat zzp INFO:debug_logger:rerank True INFO:debug_logger:history: [] INFO:debug_logger:question: 我想知道 INFO:debug_logger:kb_ids: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:debug_logger:user_id: zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:streaming: True INFO:debug_logger:start generate answer INFO:debug_logger:start generate... INFO:debug_logger:embedding number: 1 函数 get_knowledge_based_answer 执行耗时: 7.152557373046875e-07 秒
0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:00<00:00, 52.80it/s] INFO:debug_logger:milvus group number: 1 INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout [2024-05-31 09:36:07 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)
复现方法 | Steps To Reproduce
按照文档下载源码,bash run.sh -c cloud -i 0 下载镜像,起容器,出现上述问题
备注 | Anything else?
No response
我也遇到了该问题。定位为磁盘容量超过95%时elasticsearch将转为只读模式,清理磁盘至95%以下后该问题解决。
from qanything.
Related Issues (20)
- Qwen模型文件存放的地址在哪里呢
- response返回bug
- 替换为Qwen-14B-Chat模型的时候,出现以下报错,修改了模型文件中的config.seq_length也没用
- [BUG] <title>纯图片型的PDF文件解析会出现数组下标越界的问题 HOT 1
- [BUG] <title> Python版本GPU设置
- [BUG] <title>
- [E:onnxruntime:, sequential_executor.cc:514 ExecuteKernel] Non-zero status code returned while running Gather node. Name:'Gather_112' Status Message: /onnxruntime_src/onnxruntime/core/framework/bfc_arena.cc:376 void* onnxruntime::BFCArena::AllocateRawInternal(size_t, bool, onnxruntime::Stream*, bool, onnxruntime::WaitNotificationFn) Failed to allocate memory for requested buffer of size 12582912
- 问知识库是pdf的内容,理解能力差 HOT 2
- [BUG] 双3060显卡运行时报错 HOT 1
- 优化 docker 镜像大小,删除/model_repos/QAEnsemble/base
- 上传文件最大30M,如何修改为100M?
- 多个worker时显存翻倍,怎么共享一份模型参数?
- [BUG] <title>develop1.5版本 milvus insert error上传文件失败 HOT 1
- 内网环境进行对话时报错Connection to openaipublic.blob.core.windows.net timed out
- 拉取镜像时报错
- [BUG] <title>启动后端时间过长,多次测试,总耗时永远为805秒
- [BUG] <title> 启动Qanything-api服务报错
- 前端代码什么时候开源呢 HOT 1
- [BUG] <title> python最新版pdf无法解析,已经下载了pdf模型文件
- embedding, rerank 模型的部署问题
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from qanything.