milvus standalone时不时崩溃 about history_rag

[flash201524]

RPC error: [insert_rows], <MilvusException: (code=1100, message=the length (74246) of dynamic field exceeds max length (65536): invalid parameter[expected=valid length dynamic field][actual=length exceeds max length])>, <Time:{'RPC start': '2024-04-11 23:49:01.457415', 'RPC error': '2024-04-11 23:49:02.505110'}>
Traceback (most recent call last):
File "E:\OneDrive\history_rag-master\cli.py", line 120, in
cli.run()
File "E:\OneDrive\history_rag-master\cli.py", line 53, in run
self.parse_input(command_text)
File "E:\OneDrive\history_rag-master\cli.py", line 65, in parse_input
self.build_index(path=commands[1], overwrite=False)
File "E:\OneDrive\history_rag-master\cli.py", line 92, in build_index
self._executor.build_index(path, overwrite)
File "E:\OneDrive\history_rag-master\executor.py", line 186, in build_index
self.index = VectorStoreIndex(nodes, storage_context=storage_context, show_progress=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\software\Anaconda\envs\pytorch\Lib\site-packages\llama_index\indices\vector_store\base.py", line 53, in init
super().init(
File "E:\software\Anaconda\envs\pytorch\Lib\site-packages\llama_index\indices\base.py", line 75, in init
index_struct = self.build_index_from_nodes(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\software\Anaconda\envs\pytorch\Lib\site-packages\llama_index\indices\vector_store\base.py", line 274, in build_index_from_nodes
return self._build_index_from_nodes(nodes, **insert_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\software\Anaconda\envs\pytorch\Lib\site-packages\llama_index\indices\vector_store\base.py", line 246, in _build_index_from_nodes
self._add_nodes_to_index(
File "E:\software\Anaconda\envs\pytorch\Lib\site-packages\llama_index\indices\vector_store\base.py", line 200, in _add_nodes_to_index
new_ids = self._vector_store.add(nodes_batch, **insert_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\software\Anaconda\envs\pytorch\Lib\site-packages\llama_index\vector_stores\milvus.py", line 199, in add
self.collection.insert(insert_list)
File "E:\software\Anaconda\envs\pytorch\Lib\site-packages\pymilvus\orm\collection.py", line 508, in insert
return conn.insert_rows(
^^^^^^^^^^^^^^^^^
File "E:\software\Anaconda\envs\pytorch\Lib\site-packages\pymilvus\decorators.py", line 147, in handler
raise e from e
File "E:\software\Anaconda\envs\pytorch\Lib\site-packages\pymilvus\decorators.py", line 143, in handler
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "E:\software\Anaconda\envs\pytorch\Lib\site-packages\pymilvus\decorators.py", line 182, in handler
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\software\Anaconda\envs\pytorch\Lib\site-packages\pymilvus\decorators.py", line 122, in handler
raise e from e
File "E:\software\Anaconda\envs\pytorch\Lib\site-packages\pymilvus\decorators.py", line 87, in handler
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "E:\software\Anaconda\envs\pytorch\Lib\site-packages\pymilvus\client\grpc_handler.py", line 514, in insert_rows
check_status(response.status)
File "E:\software\Anaconda\envs\pytorch\Lib\site-packages\pymilvus\client\utils.py", line 60, in check_status
raise MilvusException(status.code, status.reason, status.error_code)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1100, message=the length (74246) of dynamic field exceeds max length (65536): invalid parameter[expected=valid length dynamic field][actual=length exceeds max length])>

[wxywb]

这不是崩溃的，因为milvus存储text的字符串长度是有限制的，如图是65536，而由于你的切分方式导致，切出了一个74246的长文本，事实上，长文本的embedding效果一般也不好，因为信息过多让特征不再明显。因为history_rag的文本切分是针对史料，所以不一定适合你的文本，使用更通用的文本切分方式，请参考#63

[flash201524]

这不是崩溃的，因为milvus存储text的字符串长度是有限制的，如图是65536，而由于你的切分方式导致，切出了一个74246的长文本，事实上，长文本的embedding效果一般也不好，因为信息过多让特征不再明显。因为history_rag的文本切分是针对史料，所以不一定适合你的文本，使用更通用的文本切分方式，请参考#63

明白了，但是我没办法把已经切好的文本删除了，输入remove 文件夹名之后说有多少条但是删除0条，有没有什么其他删除的指令呢

[wxywb]

from pymilvus import Collection, connections
connections.connect("default", host="localhost", port="19530")
#在cfgs/config.yaml中的默认值
col_name = "history_rag" 
col = Collection(col_name)
col.load()
col.drop()