docker部署出错，无法正常问答和知识库应用 about qanything

[yanghp86]

我也遇到了一样的问题了，INFO:debug_logger:milvus delete files_id: ['804d5229f04d4fdbb76fa565a53aa994']
INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(**http://es-container-local:9200**)> (force=False)
INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544 [status:200 duration:0.002s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has been marked alive after a successful request
INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)> (force=False)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.002s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 1 times in a row, putting on 1 second timeout
WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3)
INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)> (force=False)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.001s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 2 times in a row, putting on 2 second timeout
WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.001s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 3 times in a row, putting on 4 second timeout
WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.002s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 4 times in a row, putting on 8 second timeout
[2024-07-10 14:51:31 +0800] [678] [ERROR] Exception occurred while handling uri: 'http://10.10.0.232:8777/api/local_doc_qa/delete_files'

http://es-container-local:9200这个链接一看就是错，还有milvus的容器，用docker ps看，是没有暴露端口的，我也想不明白为什么

[yangpeng666]

我也遇到了同样的问题，请问如何解决

[yangpeng666]

我也遇到了一样的问题了，INFO:debug_logger:milvus delete files_id: ['804d5229f04d4fdbb76fa565a53aa994'] INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(**http://es-container-local:9200**)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544 [status:200 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout [2024-07-10 14:51:31 +0800] [678] [ERROR] Exception occurred while handling uri: 'http://10.10.0.232:8777/api/local_doc_qa/delete_files'

http://es-container-local:9200这个链接一看就是错，还有milvus的容器，用docker ps看，是没有暴露端口的，我也想不明白为什么

是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?

* [x]  我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?

* [x]  我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

按照5.17日的安装文档，docker部署，run.sh安装未报错，启动容器终端输出未见异常，访问UI可以正常访问，遇见两个问题： 1.无法正常上传文件，上传第一个文件一直报错：milvus插入失败，请稍后再试，第二个第三个没问题，但出现该问题后，该知识库无法正常删除，报错请求失败，该文件同样无法删除，docx，ppt，txt均尝试了。 2.无法正常问答，问答一直报错请刷新后重试。

测试了ollama，本地Qwen，应该不是模型问题，看sanic.log，问题应该出在es上面。

期望行为 | Expected Behavior

解决BUG

运行环境 | Environment

- OS:Centos8
- NVIDIA Driver:535
- CUDA:11.8
- docker:2.24.0
- docker-compose:2.27.0
- NVIDIA GPU:4090
- NVIDIA GPU Memory:24GB

QAnything日志 | QAnything logs

新建知识库时应该还是正常的：

INFO:debug_logger:check_user_exist [] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB7e4842c2615c40649e78754c8e39203b', 'kb_name': '默认知识库'}] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}

随后在插入文件时发现有一个es的404（已用########突出）：

INFO:debug_logger:XXX-wxs.pptx, cd4eca5871ac4bb7a43fb7ca0058815d, success INFO:debug_logger:success init localfile XXX-wxs.pptx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:debug_logger:before 2nd split doc lens: 53 INFO:debug_logger:after 2nd split doc lens: 44 INFO:debug_logger:langchain analysis content head: XXX INFO:debug_logger:split time: 0.3504297733306885 44 INFO:debug_logger:embedding number: 3

0%| | 0/3 [00:00<?, ?it/s] 33%|███▎ | 1/3 [00:00<00:00, 8.08it/s] 100%|██████████| 3/3 [00:00<00:00, 24.23it/s] INFO:debug_logger:embedding time: 0.12532377243041992 44 INFO:debug_logger:now inser_file XXX-wxs.pptx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:XXX-wxs.pptx (insert count: 44, delete count: 0, upsert count: 0, timestamp: 450132530566529026, success count: 44, err count: 0) INFO:debug_logger:now inser_file for es: XXX-wxs.pptx INFO:debug_logger:Inserting into es ... #############INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:404 duration:0.003s] INFO:debug_logger:##ES## - Creating index zzp++kb6fda41a1c95f4605ab812263d921740a with: mappings: {'properties': {'file_id': {'type': 'keyword', 'index': True}, 'content': {'type': 'text', 'similarity': 'custom_bm25', 'index': True, 'analyzer': 'ik_smart', 'search_analyzer': 'ik_smart'}}} settings: {'index': {'similarity': {'custom_bm25': {'type': 'BM25', 'k1': '1.3', 'b': '0.6'}}}} INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:N/A duration:10.010s] ##############WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout ERROR:debug_logger:ES insert file_id: cd4eca5871ac4bb7a43fb7ca0058815d file_name: XXX-wxs.pptx failed: Connection timed out INFO:debug_logger:insert time: 10.019953966140747 INFO:debug_logger:insert_to_milvus: success num: 0, failed num: 1 INFO:debug_logger:upload_files zzp INFO:debug_logger:mode: strong WARNING:root:missing use_local_file in request INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)]

第二次再次插入一个文档，前端UI显示成功，但日志中有一堆warning：

INFO:debug_logger:ori name: 离线部署centos文档.docx INFO:debug_logger:decode name: 离线部署centos文档.docx INFO:debug_logger:cleaned name: 离线部署centos文档.docx INFO:debug_logger:check_user_exist [('zzp',)] INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:add_file: bbd20c38f53b473babbafe01f5aab6b7 INFO:debug_logger:离线部署centos文档.docx, bbd20c38f53b473babbafe01f5aab6b7, success INFO:debug_logger:success init localfile 离线部署centos文档.docx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:before 2nd split doc lens: 170 INFO:debug_logger:after 2nd split doc lens: 170 INFO:debug_logger:langchain analysis content head: 搭建私有yum仓库，安装gcc gcc-c++，ffmpeg（三步法），nginx（nginx在centos8. INFO:debug_logger:split time: 0.04621243476867676 170 INFO:debug_logger:embedding number: 11

0%| | 0/11 [00:00<?, ?it/s] 100%|██████████| 11/11 [00:00<00:00, 286.36it/s] INFO:debug_logger:embedding time: 0.04653787612915039 170 INFO:debug_logger:now inser_file 离线部署centos文档.docx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:离线部署centos文档.docx (insert count: 170, delete count: 0, upsert count: 0, timestamp: 450132533227028483, success count: 170, err count: 0) INFO:debug_logger:now inser_file for es: 离线部署centos文档.docx INFO:debug_logger:Inserting into es ... INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.003s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:debug_logger:##ES## - Index zzp++kb6fda41a1c95f4605ab812263d921740a already exists. Skipping creation. INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base zzp INFO:debug_logger:check_kb_exist [('KB7e4842c2615c40649e78754c8e39203b',)] INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:elastic_transport.transport:DELETE http://es-container-local:9200/zzp%2B%2Bkb7e4842c2615c40649e78754c8e39203b?ignore_unavailable=true [status:200 duration:0.001s] INFO:debug_logger:##ES## - success to delete index: ['zzp++kb7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:##ES## - success delete kb_ids: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB6fda41a1c95f4605ab812263d921740a', 'kb_name': 'aaa'}] INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_bulk?refresh=false [status:N/A duration:10.010s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout INFO:debug_logger:离线部署centos文档.docx Error adding texts: Connection timed out INFO:debug_logger:insert time: 10.025246858596802 INFO:debug_logger:insert_to_milvus: success num: 1, failed num: 0 INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:delete_docs zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:check_file_exist [('cd4eca5871ac4bb7a43fb7ca0058815d', 'yellow')] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:milvus delete files_id: ['cd4eca5871ac4bb7a43fb7ca0058815d'] INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.006s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout

随后删除报错log如下：

[2024-05-31 09:35:57 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)

尝试问答，报错log如下：

INFO:debug_logger:local_doc_chat zzp INFO:debug_logger:rerank True INFO:debug_logger:history: [] INFO:debug_logger:question: 我想知道 INFO:debug_logger:kb_ids: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:debug_logger:user_id: zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:streaming: True INFO:debug_logger:start generate answer INFO:debug_logger:start generate... INFO:debug_logger:embedding number: 1 函数 get_knowledge_based_answer 执行耗时: 7.152557373046875e-07 秒

0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:00<00:00, 52.80it/s] INFO:debug_logger:milvus group number: 1 INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout [2024-05-31 09:36:07 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)

复现方法 | Steps To Reproduce

按照文档下载源码，bash run.sh -c cloud -i 0 下载镜像，起容器，出现上述问题

备注 | Anything else?

No response

是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?

* [x]  我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?

* [x]  我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

按照5.17日的安装文档，docker部署，run.sh安装未报错，启动容器终端输出未见异常，访问UI可以正常访问，遇见两个问题： 1.无法正常上传文件，上传第一个文件一直报错：milvus插入失败，请稍后再试，第二个第三个没问题，但出现该问题后，该知识库无法正常删除，报错请求失败，该文件同样无法删除，docx，ppt，txt均尝试了。 2.无法正常问答，问答一直报错请刷新后重试。

测试了ollama，本地Qwen，应该不是模型问题，看sanic.log，问题应该出在es上面。

期望行为 | Expected Behavior

解决BUG

运行环境 | Environment

- OS:Centos8
- NVIDIA Driver:535
- CUDA:11.8
- docker:2.24.0
- docker-compose:2.27.0
- NVIDIA GPU:4090
- NVIDIA GPU Memory:24GB

QAnything日志 | QAnything logs

新建知识库时应该还是正常的：

INFO:debug_logger:check_user_exist [] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB7e4842c2615c40649e78754c8e39203b', 'kb_name': '默认知识库'}] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}

随后在插入文件时发现有一个es的404（已用########突出）：

INFO:debug_logger:XXX-wxs.pptx, cd4eca5871ac4bb7a43fb7ca0058815d, success INFO:debug_logger:success init localfile XXX-wxs.pptx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:debug_logger:before 2nd split doc lens: 53 INFO:debug_logger:after 2nd split doc lens: 44 INFO:debug_logger:langchain analysis content head: XXX INFO:debug_logger:split time: 0.3504297733306885 44 INFO:debug_logger:embedding number: 3

0%| | 0/3 [00:00<?, ?it/s] 33%|███▎ | 1/3 [00:00<00:00, 8.08it/s] 100%|██████████| 3/3 [00:00<00:00, 24.23it/s] INFO:debug_logger:embedding time: 0.12532377243041992 44 INFO:debug_logger:now inser_file XXX-wxs.pptx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:XXX-wxs.pptx (insert count: 44, delete count: 0, upsert count: 0, timestamp: 450132530566529026, success count: 44, err count: 0) INFO:debug_logger:now inser_file for es: XXX-wxs.pptx INFO:debug_logger:Inserting into es ... #############INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:404 duration:0.003s] INFO:debug_logger:##ES## - Creating index zzp++kb6fda41a1c95f4605ab812263d921740a with: mappings: {'properties': {'file_id': {'type': 'keyword', 'index': True}, 'content': {'type': 'text', 'similarity': 'custom_bm25', 'index': True, 'analyzer': 'ik_smart', 'search_analyzer': 'ik_smart'}}} settings: {'index': {'similarity': {'custom_bm25': {'type': 'BM25', 'k1': '1.3', 'b': '0.6'}}}} INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:N/A duration:10.010s] ##############WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout ERROR:debug_logger:ES insert file_id: cd4eca5871ac4bb7a43fb7ca0058815d file_name: XXX-wxs.pptx failed: Connection timed out INFO:debug_logger:insert time: 10.019953966140747 INFO:debug_logger:insert_to_milvus: success num: 0, failed num: 1 INFO:debug_logger:upload_files zzp INFO:debug_logger:mode: strong WARNING:root:missing use_local_file in request INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)]

第二次再次插入一个文档，前端UI显示成功，但日志中有一堆warning：

INFO:debug_logger:ori name: 离线部署centos文档.docx INFO:debug_logger:decode name: 离线部署centos文档.docx INFO:debug_logger:cleaned name: 离线部署centos文档.docx INFO:debug_logger:check_user_exist [('zzp',)] INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:add_file: bbd20c38f53b473babbafe01f5aab6b7 INFO:debug_logger:离线部署centos文档.docx, bbd20c38f53b473babbafe01f5aab6b7, success INFO:debug_logger:success init localfile 离线部署centos文档.docx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:before 2nd split doc lens: 170 INFO:debug_logger:after 2nd split doc lens: 170 INFO:debug_logger:langchain analysis content head: 搭建私有yum仓库，安装gcc gcc-c++，ffmpeg（三步法），nginx（nginx在centos8. INFO:debug_logger:split time: 0.04621243476867676 170 INFO:debug_logger:embedding number: 11

0%| | 0/11 [00:00<?, ?it/s] 100%|██████████| 11/11 [00:00<00:00, 286.36it/s] INFO:debug_logger:embedding time: 0.04653787612915039 170 INFO:debug_logger:now inser_file 离线部署centos文档.docx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:离线部署centos文档.docx (insert count: 170, delete count: 0, upsert count: 0, timestamp: 450132533227028483, success count: 170, err count: 0) INFO:debug_logger:now inser_file for es: 离线部署centos文档.docx INFO:debug_logger:Inserting into es ... INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.003s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:debug_logger:##ES## - Index zzp++kb6fda41a1c95f4605ab812263d921740a already exists. Skipping creation. INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base zzp INFO:debug_logger:check_kb_exist [('KB7e4842c2615c40649e78754c8e39203b',)] INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:elastic_transport.transport:DELETE http://es-container-local:9200/zzp%2B%2Bkb7e4842c2615c40649e78754c8e39203b?ignore_unavailable=true [status:200 duration:0.001s] INFO:debug_logger:##ES## - success to delete index: ['zzp++kb7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:##ES## - success delete kb_ids: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB6fda41a1c95f4605ab812263d921740a', 'kb_name': 'aaa'}] INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_bulk?refresh=false [status:N/A duration:10.010s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout INFO:debug_logger:离线部署centos文档.docx Error adding texts: Connection timed out INFO:debug_logger:insert time: 10.025246858596802 INFO:debug_logger:insert_to_milvus: success num: 1, failed num: 0 INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:delete_docs zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:check_file_exist [('cd4eca5871ac4bb7a43fb7ca0058815d', 'yellow')] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:milvus delete files_id: ['cd4eca5871ac4bb7a43fb7ca0058815d'] INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.006s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout

随后删除报错log如下：

[2024-05-31 09:35:57 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)

尝试问答，报错log如下：

INFO:debug_logger:local_doc_chat zzp INFO:debug_logger:rerank True INFO:debug_logger:history: [] INFO:debug_logger:question: 我想知道 INFO:debug_logger:kb_ids: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:debug_logger:user_id: zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:streaming: True INFO:debug_logger:start generate answer INFO:debug_logger:start generate... INFO:debug_logger:embedding number: 1 函数 get_knowledge_based_answer 执行耗时: 7.152557373046875e-07 秒

0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:00<00:00, 52.80it/s] INFO:debug_logger:milvus group number: 1 INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout [2024-05-31 09:36:07 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)

复现方法 | Steps To Reproduce

按照文档下载源码，bash run.sh -c cloud -i 0 下载镜像，起容器，出现上述问题

备注 | Anything else?

_No respons

是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?

* [x]  我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?

* [x]  我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

按照5.17日的安装文档，docker部署，run.sh安装未报错，启动容器终端输出未见异常，访问UI可以正常访问，遇见两个问题： 1.无法正常上传文件，上传第一个文件一直报错：milvus插入失败，请稍后再试，第二个第三个没问题，但出现该问题后，该知识库无法正常删除，报错请求失败，该文件同样无法删除，docx，ppt，txt均尝试了。 2.无法正常问答，问答一直报错请刷新后重试。

测试了ollama，本地Qwen，应该不是模型问题，看sanic.log，问题应该出在es上面。

期望行为 | Expected Behavior

解决BUG

运行环境 | Environment

- OS:Centos8
- NVIDIA Driver:535
- CUDA:11.8
- docker:2.24.0
- docker-compose:2.27.0
- NVIDIA GPU:4090
- NVIDIA GPU Memory:24GB

QAnything日志 | QAnything logs

新建知识库时应该还是正常的：

INFO:debug_logger:check_user_exist [] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB7e4842c2615c40649e78754c8e39203b', 'kb_name': '默认知识库'}] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}

随后在插入文件时发现有一个es的404（已用########突出）：

INFO:debug_logger:XXX-wxs.pptx, cd4eca5871ac4bb7a43fb7ca0058815d, success INFO:debug_logger:success init localfile XXX-wxs.pptx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:debug_logger:before 2nd split doc lens: 53 INFO:debug_logger:after 2nd split doc lens: 44 INFO:debug_logger:langchain analysis content head: XXX INFO:debug_logger:split time: 0.3504297733306885 44 INFO:debug_logger:embedding number: 3

0%| | 0/3 [00:00<?, ?it/s] 33%|███▎ | 1/3 [00:00<00:00, 8.08it/s] 100%|██████████| 3/3 [00:00<00:00, 24.23it/s] INFO:debug_logger:embedding time: 0.12532377243041992 44 INFO:debug_logger:now inser_file XXX-wxs.pptx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:XXX-wxs.pptx (insert count: 44, delete count: 0, upsert count: 0, timestamp: 450132530566529026, success count: 44, err count: 0) INFO:debug_logger:now inser_file for es: XXX-wxs.pptx INFO:debug_logger:Inserting into es ... #############INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:404 duration:0.003s] INFO:debug_logger:##ES## - Creating index zzp++kb6fda41a1c95f4605ab812263d921740a with: mappings: {'properties': {'file_id': {'type': 'keyword', 'index': True}, 'content': {'type': 'text', 'similarity': 'custom_bm25', 'index': True, 'analyzer': 'ik_smart', 'search_analyzer': 'ik_smart'}}} settings: {'index': {'similarity': {'custom_bm25': {'type': 'BM25', 'k1': '1.3', 'b': '0.6'}}}} INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:N/A duration:10.010s] ##############WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout ERROR:debug_logger:ES insert file_id: cd4eca5871ac4bb7a43fb7ca0058815d file_name: XXX-wxs.pptx failed: Connection timed out INFO:debug_logger:insert time: 10.019953966140747 INFO:debug_logger:insert_to_milvus: success num: 0, failed num: 1 INFO:debug_logger:upload_files zzp INFO:debug_logger:mode: strong WARNING:root:missing use_local_file in request INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)]

第二次再次插入一个文档，前端UI显示成功，但日志中有一堆warning：

INFO:debug_logger:ori name: 离线部署centos文档.docx INFO:debug_logger:decode name: 离线部署centos文档.docx INFO:debug_logger:cleaned name: 离线部署centos文档.docx INFO:debug_logger:check_user_exist [('zzp',)] INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:add_file: bbd20c38f53b473babbafe01f5aab6b7 INFO:debug_logger:离线部署centos文档.docx, bbd20c38f53b473babbafe01f5aab6b7, success INFO:debug_logger:success init localfile 离线部署centos文档.docx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:before 2nd split doc lens: 170 INFO:debug_logger:after 2nd split doc lens: 170 INFO:debug_logger:langchain analysis content head: 搭建私有yum仓库，安装gcc gcc-c++，ffmpeg（三步法），nginx（nginx在centos8. INFO:debug_logger:split time: 0.04621243476867676 170 INFO:debug_logger:embedding number: 11

0%| | 0/11 [00:00<?, ?it/s] 100%|██████████| 11/11 [00:00<00:00, 286.36it/s] INFO:debug_logger:embedding time: 0.04653787612915039 170 INFO:debug_logger:now inser_file 离线部署centos文档.docx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:离线部署centos文档.docx (insert count: 170, delete count: 0, upsert count: 0, timestamp: 450132533227028483, success count: 170, err count: 0) INFO:debug_logger:now inser_file for es: 离线部署centos文档.docx INFO:debug_logger:Inserting into es ... INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.003s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:debug_logger:##ES## - Index zzp++kb6fda41a1c95f4605ab812263d921740a already exists. Skipping creation. INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base zzp INFO:debug_logger:check_kb_exist [('KB7e4842c2615c40649e78754c8e39203b',)] INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:elastic_transport.transport:DELETE http://es-container-local:9200/zzp%2B%2Bkb7e4842c2615c40649e78754c8e39203b?ignore_unavailable=true [status:200 duration:0.001s] INFO:debug_logger:##ES## - success to delete index: ['zzp++kb7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:##ES## - success delete kb_ids: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB6fda41a1c95f4605ab812263d921740a', 'kb_name': 'aaa'}] INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_bulk?refresh=false [status:N/A duration:10.010s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout INFO:debug_logger:离线部署centos文档.docx Error adding texts: Connection timed out INFO:debug_logger:insert time: 10.025246858596802 INFO:debug_logger:insert_to_milvus: success num: 1, failed num: 0 INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:delete_docs zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:check_file_exist [('cd4eca5871ac4bb7a43fb7ca0058815d', 'yellow')] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:milvus delete files_id: ['cd4eca5871ac4bb7a43fb7ca0058815d'] INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.006s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout

随后删除报错log如下：

[2024-05-31 09:35:57 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)

尝试问答，报错log如下：

INFO:debug_logger:local_doc_chat zzp INFO:debug_logger:rerank True INFO:debug_logger:history: [] INFO:debug_logger:question: 我想知道 INFO:debug_logger:kb_ids: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:debug_logger:user_id: zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:streaming: True INFO:debug_logger:start generate answer INFO:debug_logger:start generate... INFO:debug_logger:embedding number: 1 函数 get_knowledge_based_answer 执行耗时: 7.152557373046875e-07 秒

0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:00<00:00, 52.80it/s] INFO:debug_logger:milvus group number: 1 INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout [2024-05-31 09:36:07 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)

复现方法 | Steps To Reproduce

按照文档下载源码，bash run.sh -c cloud -i 0 下载镜像，起容器，出现上述问题

备注 | Anything else?

No response

我也遇到了该问题。定位为磁盘容量超过95%时elasticsearch将转为只读模式，清理磁盘至95%以下后该问题解决。