Dspy Assertions make execution sequential about dspy

[dvd42]

I should say, I am more than happy to make a PR for this, if in fact, threaded execution is not possible with retries, since this is very important for us at the moment.

Will also be happy to work on Async support which would simplify a lot of this. Let me know if and how I can help :)

[dvd42]

It seems that there is a threading lock in the backrack_handler wrapper . Is it possible to build our own backtrack_handler and use that? How should we go about this? Thanks in advance!

[theta-lin]

I am also looking forward to see multi-threaded assertions.

I see. But doesn't the settings singleton handle threads correctly?

dspy/dsp/utils/settings.py

Line 51 in 55510ee

thread_id = threading.get_ident()

Couldn't this be leveraged to get the relevant config for the current thread in the backtrack_handler?

I think this is a good observation as I also think the only thing making assertions not supporting threads is the sharing of global config (which is not really "config", but rather storing the module call traces and backtracking information globally). However, I also note that each thread does get its copy of the config:

dspy/dsp/utils/settings.py

Lines 51 to 65 in af5186c

	@property
	def config(self):
	thread_id = threading.get_ident()
	if thread_id not in self.stack_by_thread:
	self.stack_by_thread[thread_id] = [self.main_stack[-1].copy()]
	return self.stack_by_thread[thread_id][-1]
	def __getattr__(self, name):
	if hasattr(self.config, name):
	return getattr(self.config, name)
	if name in self.config:
	return self.config[name]
	super().__getattr__(name)

This implementation is not without its problems though, as pointed out in this part of the evaluator code also leveraging threads:

dspy/dspy/evaluate/evaluate.py

Lines 153 to 157 in af5186c

	# NOTE: TODO: Won't work if threads create threads!
	thread_stacks = dspy.settings.stack_by_thread
	creating_new_thread = threading.get_ident() not in thread_stacks
	if creating_new_thread:
	thread_stacks[threading.get_ident()] = list(dspy.settings.main_stack)

You can see that as the thread-local config only copies from the main thread, it won't work if threads spawn more threads. Therefore, I can see the problem of maybe using multi-threaded module with assertions inside multi-threaded evaluator.

However, it appears that settings_v2.py is created to address this issue? It would be better for the developers to offer more insight here.

What is the most problematic at hand is that I guess even if we can create our own custom_backtrack_handler, then how are we supposed to build our own thread-safe version of dsp.settings.trace? I mean the access of dsp.settings.trace might be thread-safe, but having multiple threads might mess up the traces.

[dvd42]

Any updates? I am still facing this problem, which effectively prevents making many requests in parallel with the retry mechanism. Since the library does not support async requests yet, threads are the main way to achieve this concurrency. Any help would be much appreciated.

[arnavsinghvi11]

Hi @dvd42 , would love to see a PR for this! Currently, DSPy assertions require the lock to ensure the Retries are mapped to the correct requests (and were breaking otherwise) and have not been tested with async. activate_assertions uses the default backtrack_handler, but you can configure a custom backtrack_handler logic compatible with assertions.py and pass it to the assert_transform_module:

Example:

from dspy.primitives.assertions import assert_transform_module#, backtrack_handler

# define custom_backtrack_handler

program_with_assertions = assert_transform_module(ProgramWithAssertions(), custom_backtrack_handler)

[dvd42]

I see. But doesn't the settings singleton handle threads correctly?

dspy/dsp/utils/settings.py

Line 51 in 55510ee

thread_id = threading.get_ident()

Couldn't this be leveraged to get the relevant config for the current thread in the backtrack_handler?

Let me know if I am on the right track here and I'll create a PR :)

[RahulBalan]

@arnavsinghvi11 Am also facing this problem.
Can you show an example of custom_backtrack_handler. What all things should be considered in a custom_backtrack_handler

[chiragshah285]

following looking to do the same

[enchyisle]

Facing the same issue here. Hope that multi-thread support would be available soon so that we won't need to do work-arounds.

[dvd42]

This seems like a solution.

def funcion_that_your_threads_execute()
  model = your_llm()
  dspy.configure(
      lm=model,
      trace=[],
      assert_failures=0,
      suggest_failures=0,
      backtrack_to=None,
      predictor_feedbacks={},
      backtrack_to_args={},
  )

If I replace the lock with nullcontext (dspy.settings.lock = nullcontext()). This seems to work for me, resulting in consistent outputs with multithreading.

@arnavsinghvi11 @okhat could you confirm if I am missing something?

If this is correct, I could make a PR updating/creating an example wherever you point me to with this use case.