Comments (5)
Currently just goes text by text. Should be easy to expose a batch method though.
what’s the use case? Is it just to add it onto the LLM class or do you want it as part of a chain as well?
from langchain.
Every text in a batch is independent with the same prompt templet, but it is slow to call llm() text by text now.So I wonder if there exits better method to get all the results faster :)
from langchain.
Every text in a batch is independent with the same prompt templet, but it is slow to call llm() text by text now.So I wonder if there exits better method to get all the results faster :)
You can ask LLM to process your list of entities one by one, so you are “equivalently” getting the batch result, but you are also bearing the risk of getting unwanted output on a list of large number of elements.
from langchain.
there is now a generate endpoint, which LLMs can use to batch requests: https://langchain.readthedocs.io/en/latest/examples/prompts/llm_functionality.html
the batching is implemented only for openai at the moment
what LLM provider are you using?
from langchain.
Hi, @Bookraint! I'm here to help the LangChain team manage their backlog and I wanted to let you know that we are marking this issue as stale.
From what I understand, you were asking if LangChain can process a batch of prompts or if it can only process one text at a time. hwchase17 responded that currently LangChain only goes text by text, but it should be easy to expose a batch method. You mentioned that you have multiple texts with the same prompt template and calling llm() text by text is slow. tonyabracadabra suggested asking LLM to process the list of entities one by one, but warned about the risk of getting unwanted output.
However, there seems to be a resolution to the issue. hwchase17 mentioned that there is now a generate endpoint for batching requests, although it is currently only implemented for OpenAI.
Now, we would like to know if this issue is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.
Thank you for your understanding and contribution to the LangChain project!
from langchain.
Related Issues (20)
- AzureSearch delete method does not use the variable FIELDS_ID therefore it does not override the value
- Hi @dosu, HOT 2
- FileNotFoundError in PineconeVectorStore.from_documents on AWS Lambda HOT 3
- Weavite "Containsany" Filter returns error
- Structured output with ChatOpenAI is not working when structure class has a list of strings HOT 2
- langchain-openai==0.1.8 is now broken HOT 9
- Logit Bias is not having the desired effect when using ChatOpenAI - it doesn't seem like it's propagating to OpenAI call properly HOT 1
- Langchain using chain.ainvoke for async breaks with OpenAI>=1.0: AttributeError: 'NoneType' object has no attribute 'create HOT 3
- Creating proxy using ChatTongyi, unable to return results properly HOT 1
- [2024-05-31 11:06:20,219: WARNING] langchain_core.callbacks.manager: Error in LangChainTracer.on_chain_end callback: AttributeError("'NoneType' object has no attribute 'append'") HOT 7
- AIMessage played before invoking a tool is not registered in the Agent memory HOT 2
- Unexpected Behaviour with Human as a tool HOT 1
- Chroma returns the same document more than once when use as a retriver HOT 2
- Plugin execution not working anymore HOT 1
- Error when Choosing between multiple schemas
- ConversationBufferWindowMemory HOT 2
- Tools do not work with HuggingFace - Issue either with tutorial or library
- GraphCypherQAChain cannot generate correct Cypher commands
- REGRESSION: ChatLiteLLM: ValidationError only when using cache HOT 1
- ChatMistralAI with_structured_output does not recognize BaseModel subclass HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from langchain.