Comments (7)
@dtee1 yes please :) let me know if you have any questions - you're welcome to join our slack as well.
from openllmetry.
Interested
from openllmetry.
This change definitely has bugs. I updated to the latest version of traceloop-sdk today, 0.16.6, but even though gpt4's responses are normally output to the console without any filtering issues, it still shows FILTERED on traceloop.
from openllmetry.
According to the Azure official documentation https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/content-filter?tabs=warning%2Cpython-new
data: {"id":"","object":"","created":0,"model":"","prompt_annotations":[{"prompt_index":0,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}],"choices":[],"usage":null}
data: {"id":"chatcmpl-7rCNsVeZy0PGnX3H6jK8STps5nZUY","object":"chat.completion.chunk","created":1692913344,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"role":"assistant"}}],"usage":null}
data: {"id":"chatcmpl-7rCNsVeZy0PGnX3H6jK8STps5nZUY","object":"chat.completion.chunk","created":1692913344,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":"Color"}}],"usage":null}
data: {"id":"chatcmpl-7rCNsVeZy0PGnX3H6jK8STps5nZUY","object":"chat.completion.chunk","created":1692913344,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":" is"}}],"usage":null}
data: {"id":"chatcmpl-7rCNsVeZy0PGnX3H6jK8STps5nZUY","object":"chat.completion.chunk","created":1692913344,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":" a"}}],"usage":null}
...
data: {"id":"","object":"","created":0,"model":"","choices":[{"index":0,"finish_reason":null,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},"content_filter_offsets":{"check_offset":44,"start_offset":44,"end_offset":198}}],"usage":null}
...
data: {"id":"chatcmpl-7rCNsVeZy0PGnX3H6jK8STps5nZUY","object":"chat.completion.chunk","created":1692913344,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":"stop","delta":{}}],"usage":null}
data: {"id":"","object":"","created":0,"model":"","choices":[{"index":0,"finish_reason":null,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},"content_filter_offsets":{"check_offset":506,"start_offset":44,"end_offset":571}}],"usage":null}
data: [DONE]
I believe you need to check whether the finish_reason is 'stop' or 'content_filter', rather than checking if the content_filter_results is null.
from openllmetry.
@sukidesuka thanks for flagging! I was lazy and didn't test it properly for Azure. I've now added proper testing specifically for Azure and fixed the issue so this won't happen. See #886
from openllmetry.
I can help!
from openllmetry.
Sounds good @lauradowell!
@dtee1, are you still on this?
from openllmetry.
Related Issues (20)
- 🐛 Bug Report: Langchain ChromaDB RAG template does not seem to work HOT 1
- 🚀 Feature: Support Google's alternative package for Gemini HOT 5
- 🚀 Feature: cost metrics across all instrumentations HOT 2
- TypeError: '<' not supported between instances of 'NoneType' and 'int'
- 🐛 Bug Report: Build issue with Module Not Found Error HOT 11
- 🐛 Bug Report: Requirements for Langchain example HOT 7
- 🚀 Feature: Add support for streaming in Langchain instrumentation HOT 1
- 🚀 Feature: Add parameter to control trace behavior in workflow decorator
- 🐛 Bug Report: Missing Token Counts for Anthropic Claude 3 Models with Amazon Bedrock and Langchain
- 🐛 Bug Report: missing package dependencies HOT 4
- 🚀 Feature: Support DSPy
- 🐛 Bug Report: Haystack example does not produce prompts in the spans HOT 1
- 🐛 Bug Report: warnings on LangChain HOT 6
- 🚀 Feature: provide llm provider information in payload of metrics HOT 4
- 🐛 Bug Report: Bump opentelemetry-instrumentation beta packages from 0.45b0 to 0.46b0 HOT 4
- 🐛 Bug Report: Error in llm streaming when using the latest langchain-openai
- 🐛 Bug Report: Generation crashes because `finish_reason` is not always present in the async delta HOT 1
- 🚀 Feature: Tracing for Streamed OpenAI Tool / Function Calls
- 🚀 Feature: Support for pgvector
- 🐛 Bug Report: Not getting token counts for Databricks (both foundation & external models)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openllmetry.