Comments (5)
I see that you read the doc on improving answer quality already. Have you identified if the incorrect answers are due to the retrieval step or the LLM step?
If they are due to the retrieval step, then you may want to try out different chunking strategies or try a content chunking strategy like concatenating the document title to each chunk. You can also try increasing the retrieval count and/or setting a minimum reranker score. Have you tried running evaluations with https://github.com/Azure-Samples/ai-rag-chat-evaluator to see if any parameters increase answer quality for you?
You can also try changing to a more powerful model like gpt-4. That won't help with the retrieval step, but that could help if the problem is that the model isn't distinguishing well between helpful results and irrelevant results in the sources.
from azure-search-openai-demo.
As for whether you can teach the bot: I would first start off with taking those examples that it does poorly on, and add them to an evaluation data set. Then you can run evaluations with different parameters and see whether you can improve results.
There's only so much context that you can fit into the prompt, so I'm not sure it'd work to include stuff in the system prompt like "vacation policy is in doc X". Perhaps it would if your number of documents is small enough.
The way to actually teach these models is fine-tuning, which you can certainly try, but the expense may not be worth it. There's a technique called RAFT from Berkeley which is designed for improving results for RAG applications. The retrieval still needs to be good, however.
from azure-search-openai-demo.
@pamelafox It's with retrieval. I believe it's because there are some documents with overlapping content. For example, another one that the bot often gets wrong is confusing the "Company credit card" doc with the "Business card" doc.
How can I concatenate the document title to each chunk? Is there documentation on this? Or how else could I address the bot getting confused in retrieval with overlapping content? Already added metadata to these PDFs but that isn't seeming to help.
I don't know how complicated this would be (I'm just a novice IT intern college student), but ideally, I'd be able to tell the bot if the user says certain keywords like "company card" don't even bother going to the semantic search ranker thing, just skip that step, I'll tell you the exact file to reference and then you can take that info, send to LLM and complete process. This might be something I have to figure out on my own, but please let me know if you think there's a way that could work.
Thank you for the responses and helpful information, it is much appreciated.
from azure-search-openai-demo.
@pamelafox Waiting to hear back on this, let me know thank you!
from azure-search-openai-demo.
To concatenate the title to each chunk, you would modify the code here:
Specifically, instead of "content": section.split_page.text, you would set that content field to something like "Document title: {title}\n Document chunk: {section.split_page.text}"
If you wanted it included in the vector embedding, you'd pass that string to the embeddings as well:
As for skipping a search directly and going directly to that file, check out this blog post where I describe one way to do that:
https://blog.pamelafox.org/2024/03/rag-techniques-using-function-calling.html
from azure-search-openai-demo.
Related Issues (20)
- Need help with authentication setup (HTTP ERROR 500 but works locally) HOT 6
- Long deployment time HOT 6
- azd pipeline config command fails
- Broken Relative Paths HOT 1
- 403 Forbidden when ai search and azure openai public networking is disabled HOT 2
- Deploying GPT-4o mini instead of 3.5 Turbo HOT 7
- Feature: Generate text to speech on demand HOT 8
- Getting "TypeError: Cannot read properties of undefined (reading 'role')" HOT 2
- After Deployment, I am not able to see upload or Login features HOT 2
- Changes to CSS keeping getting overwritten HOT 1
- Getting long path error, while installing requirements.txt HOT 2
- Enable Local Deployment with Login/ACLs and Uploads On HOT 2
- Bypass Cognitive Service
- Unable to use Azure Speech SDK with custom subdomain HOT 1
- HTTP Error 500- This page isn't working at the moment. 500 Unable to download OpenID Connect Configuration.
- Question about logging
- Solution for indexing JSON attribute values: AI Search Index and Indexer via Integrated Vectorization
- [Issue] Wrong Pricing Strategy in Cost Estimate HOT 1
- Broken citations
- Answers outside knowledge base
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azure-search-openai-demo.