Comments (8)
You can definitely use it for what you described! Just change the training examples, the Template, and the program’s generate steps!
from dspy.
Thanks for the quick answer, I will give it a try and reach out for any help. Also in your videos on youtube, I see you mentioning that do not throw away unlabelled questions from the demo, instead feed them as unlabelled examples. Is there some documentation that explains how this works? How does the model learn from unlabelled samples?
Regards,
Karrtik
from dspy.
Hey @okhat, if we are looking at answers which are long passages, how to do we demonstrate train samples? The passage_match and answer_match always returns none for them right?
from dspy.
@ksgr5566 Use a dsp.generate call to do the approximate passage_match and anwer_match!
from dspy.
@ksgr5566 Use a dsp.generate call to do the approximate passage_match and anwer_match!
@okhat I don't understand on how that would work. Let me be more clear about my question:
@dsp.transformation
def multihop_attempt(d: dsp.Example) -> dsp.Example:
# Prepare unaugmented demonstrations for the example.
x = dsp.Example(question=d.question, demos=dsp.all_but(train, d))
# Search. And skip examples where search fails.
# Annotate demonstrations for multihop_search_v2 with the simpler multihop_search_v1 pipeline.
x = multihop_search_v1(x)
if not dsp.passage_match(x.context, d.answer): return None
# Predict. And skip examples where predict fails.
x = QA_predict(x, sc=False)
if not dsp.answer_match(x.answer, d.answer): return None
return d.copy(**x)
@dsp.transformation
def multihop_demonstrate(x: dsp.Example) -> dsp.Example:
x.demos = dsp.annotate(multihop_attempt)(demos, k=4, return_all=True)
return x
My answers (d.answer
) here are not factoids, instead a passage which is two to three sentences long. I observed that dsp.passage_match
and dsp.answer_match
are made for factoid answers from here.
I want to be able to do multihop_demonstrate when demos consists of answers which are 2-3 sentences long. Could you be more clear on how to do this?
from dspy.
Write a DSP program whose job is to implement passage match and answer match using dsp.generate (using the LLM)
from dspy.
ohkk.. I made another answer_match like this:
True_Answer = dsp.Type(prefix="The correct answer: ", desc="${The true answer}")
Predicted_Answer = dsp.Type(prefix="Another answer: ", desc="${The answer to compare to}")
Resp = dsp.Type(prefix="Response: ", desc="${True (or) False}")
answer_match_template = dsp.Template(
instructions="Return True if the essence of both answers is same else return False. Respond with only 'True' and 'False'.",
true_answer=True_Answer(), predicted_answer=Predicted_Answer(), response=Resp()
)
def answer_match(true_ans, pred_ans):
match_example = dsp.Example(true_answer=true_ans, predicted_answer=pred_ans, demos=[])
_, completions = dsp.generate(answer_match_template)(match_example, stage='answer_match')
return completions[0].response
This works for answer_match.
One other doubt, why do we need passage_match in between, because finally answer_match does the work right?
from dspy.
Excellent
from dspy.
Related Issues (20)
- Getting this backing off problem when I use COPRO HOT 1
- Can't load Llama Model HOT 7
- Add Support to FAISS Retiever! HOT 2
- MultiChainComparison input from ChainOfThought HOT 3
- Update Google and Google VertexAI for Gemini 1.5 Pro / Flash HOT 2
- How to optimize only a part of instruction? HOT 1
- GROQ client not available in packages HOT 2
- Predict forward() does not use lm in kwargs HOT 1
- DSPY Copro Tutorial for Hotpot QA doesn't work with Bedrock Claude Sonnet Model HOT 4
- ETA on new release? HOT 2
- RAG.forward() missing 1 required positional argument: 'question' HOT 5
- Enhancement: Would be great if we can load dspy dataset directly from pandas dataframe
- Clarification on the use of hard-written output field prompt in TextToSQLAnswer Signature HOT 1
- dspy and langfuse integration HOT 6
- about openai version
- Best practice for train and validation set separation HOT 2
- Triggering the Azure OpenAi (GPT 4) Triggers content warning and DSPy exception
- DSPy COPRO Tutorial for Hotpot QA doesn't work with VertexAI Gemini
- ZeroDivisionError: division by zero when using MIPROv2 HOT 2
- DatabricksRM retrieval using dspy.Retrieve() throws TypeError HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dspy.