Comments (10)
I should say, I am more than happy to make a PR for this, if in fact, threaded execution is not possible with retries, since this is very important for us at the moment.
Will also be happy to work on Async support which would simplify a lot of this. Let me know if and how I can help :)
from dspy.
It seems that there is a threading lock in the backrack_handler wrapper . Is it possible to build our own backtrack_handler and use that? How should we go about this? Thanks in advance!
from dspy.
I am also looking forward to see multi-threaded assertions.
I see. But doesn't the settings singleton handle threads correctly?
Line 51 in 55510ee
Couldn't this be leveraged to get the relevant config for the current thread in the backtrack_handler?
I think this is a good observation as I also think the only thing making assertions not supporting threads is the sharing of global config (which is not really "config", but rather storing the module call traces and backtracking information globally). However, I also note that each thread does get its copy of the config:
Lines 51 to 65 in af5186c
This implementation is not without its problems though, as pointed out in this part of the evaluator code also leveraging threads:
dspy/dspy/evaluate/evaluate.py
Lines 153 to 157 in af5186c
You can see that as the thread-local config only copies from the main thread, it won't work if threads spawn more threads. Therefore, I can see the problem of maybe using multi-threaded module with assertions inside multi-threaded evaluator.
However, it appears that settings_v2.py is created to address this issue? It would be better for the developers to offer more insight here.
What is the most problematic at hand is that I guess even if we can create our own custom_backtrack_handler
, then how are we supposed to build our own thread-safe version of dsp.settings.trace
? I mean the access of dsp.settings.trace
might be thread-safe, but having multiple threads might mess up the traces.
from dspy.
Any updates? I am still facing this problem, which effectively prevents making many requests in parallel with the retry mechanism. Since the library does not support async requests yet, threads are the main way to achieve this concurrency. Any help would be much appreciated.
from dspy.
Hi @dvd42 , would love to see a PR for this! Currently, DSPy assertions require the lock to ensure the Retries are mapped to the correct requests (and were breaking otherwise) and have not been tested with async. activate_assertions
uses the default backtrack_handler, but you can configure a custom backtrack_handler
logic compatible with assertions.py
and pass it to the assert_transform_module
:
Example:
from dspy.primitives.assertions import assert_transform_module#, backtrack_handler
# define custom_backtrack_handler
program_with_assertions = assert_transform_module(ProgramWithAssertions(), custom_backtrack_handler)
from dspy.
I see. But doesn't the settings singleton handle threads correctly?
Line 51 in 55510ee
Couldn't this be leveraged to get the relevant config for the current thread in the backtrack_handler?
Let me know if I am on the right track here and I'll create a PR :)
from dspy.
@arnavsinghvi11 Am also facing this problem.
Can you show an example of custom_backtrack_handler. What all things should be considered in a custom_backtrack_handler
from dspy.
following looking to do the same
from dspy.
Facing the same issue here. Hope that multi-thread support would be available soon so that we won't need to do work-arounds.
from dspy.
This seems like a solution.
def funcion_that_your_threads_execute()
model = your_llm()
dspy.configure(
lm=model,
trace=[],
assert_failures=0,
suggest_failures=0,
backtrack_to=None,
predictor_feedbacks={},
backtrack_to_args={},
)
If I replace the lock with nullcontext
(dspy.settings.lock = nullcontext()
). This seems to work for me, resulting in consistent outputs with multithreading.
@arnavsinghvi11 @okhat could you confirm if I am missing something?
If this is correct, I could make a PR updating/creating an example wherever you point me to with this use case.
from dspy.
Related Issues (20)
- Integrate LiteLLM HOT 1
- Issue: Integration of Langfuse with DSPy Framework - Zero Values for Cost and Token Usage HOT 1
- ModuleNotFoundError: No module named 'dspy.predict.llamaindex'
- Support for AWS Credentials Input in dspy.Bedrock
- MIPRO does not return demonstrations
- Request for OpenAI Structured Output Support HOT 2
- InvalidDimensionException: Embedding dimension X does not match collection dimensionality Y
- Add support for default_headers in OpenAI client
- Need more info or documentation on dspy.settings.experimental HOT 1
- Use of a Retriever with the attribute with_metadata leads to an unexpected behaviour.
- Mising inport for Mongo retriever HOT 1
- optimization: add dedent for __doc__ HOT 1
- Guidance on use case
- How to save the traces as a file locally? HOT 1
- Developing a Dockerfile for dspy
- Pydantic package upgrade HOT 2
- Grammar-Based Prompt Format Optimization
- Error in trying to save compiled dspy program using optimized_program.save(YOUR_SAVE_PATH) HOT 4
- `LangchainPredict` class not compatible with `COPRO` optimizer
- [ERROR] MIPROv2 + multi hop module results in IndexError HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dspy.