Comments (1)
感谢关注ModelCache项目,下面是针对上述问题的逐条回复:
问题1:
Cache会同时缓存query和prompt(如果存在), 在多轮对话中甚至会缓存LLM的回答(因为对话存在连续性)。
问题2:
对于“现在几点了?”这样的问题,大模型不会直接回答,举例:
- GPT4_Turbo_128K的回复为:抱歉,作为一个AI模型,我无法提供实时的日期和时间信息。要获取当前的时间,请查看您的设备(电脑、手机等)上的时钟功能,或者使用操作系统提供的时间查询服务。
- GPT3.5_Turbo的回复为:抱歉,作为一个人工智能,我无法提供实时时间。您可以查看您的电子设备或者手表来获取当前时间。祝您有美好的一天!如果您还有其他问题,我很乐意帮助您。
因此,Cache不会影响回答效果,如果在你的项目中使用了tools api来解决此类问题,调整Cache的位置(例如:Cache能力放在时钟tools后),也不会产生影响。
问题3:
考虑到向量数据库特点和检索性能等因素,将query-result pairs中的result存储到关系数据库(OceanBase,也可以是mysql、sqlite等),以code场景为例:
- query:“使用python实现快速排序”
- result:“在Python中实现快速排序的一种方法如下:\n def quicksort(arr): ...(代码省略)”
关系数据库中存储的是result内容,当新的用户query与缓存的query一致或相似时,会从关系数据库中检索到对应的result,直接返回给用户,而无需调用大模型,从而节省推理资源并提升响应速度。
from codefuse-modelcache.
Related Issues (2)
- 非常感谢蚂蚁开源code模型 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from codefuse-modelcache.