Comments (2)
Describe the bug Very slow inference during agent work in comparison to usual LLM interaction I'm using local setup with API connection to TextGen WebUI in local network Each iteration of TaskWeaver is very-very slow generation speed is drastically decreased to around 1-2 t/s (usual speed on same setup 15-20 t/s)
At this communication rate this tool is net very useful, simple coding task like print numbers executed in 20-30 mins. Is there any tweak to solve it. I guess it could because of relatively large context in each request?
To Reproduce Steps to reproduce the behavior:
- Start the service
- Type the user query "any listed query from example description"
- Wait for the response forever
Expected behavior Similar inference speed as Autogen
Environment Information (please complete the following information):
- OS: MacOS
- Python Version 3.11
- LLM that you're using: number of different 7b models
hi bro, how to run with local llm
from taskweaver.
Close inactive issues.
from taskweaver.
Related Issues (20)
- debug error in latest version HOT 1
- Track or count the number of tokens being used HOT 2
- Handling Korean Font Issues in Matplotlib Visualizations with Taskweaver and Docker HOT 2
- Not able to leverage session event handler in my code HOT 4
- Request for Scatter plot default marker modify HOT 2
- Permission Errors and Plugin Execution Failures in Dockerized Environment
- I can't import packages which needs to be installed HOT 1
- Failed loading plugin, 'gbk' codec can't decode? HOT 2
- Not able run Taskweaver with LLM Qwen1.5-72B-Chat HOT 2
- Does Taskweaver works with llama3? HOT 2
- Gets stuck in repeating Board messages HOT 1
- Running Ollama with LLama3 and Phi3 HOT 3
- Having trouble getting packages installed for plugins to use HOT 2
- No memory using only code_interperter (without planner) HOT 1
- Couldn't able to change the port number. HOT 1
- After executing the sql_pull_data plugin, the final error displayed is “No such file or directory”. HOT 4
- Front-End Attachment Feature Missing and OpenAI API Key Connection Error in Application HOT 13
- Is it possible to use ollama embedding model while using OpenAI model for agents? HOT 2
- Can Telemetry tracking be used when using TaskWeaver as a library? HOT 6
- Multiple RAG as plugins HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from taskweaver.