traceloop / openllmetry Goto Github PK
View Code? Open in Web Editor NEWOpen-source observability for your LLM application, based on OpenTelemetry
Home Page: https://www.traceloop.com/openllmetry
License: Apache License 2.0
Open-source observability for your LLM application, based on OpenTelemetry
Home Page: https://www.traceloop.com/openllmetry
License: Apache License 2.0
Traceloop SDK
from dotenv import load_dotenv
import os
load_dotenv()
from traceloop.sdk import Traceloop
Traceloop.init()
I have created a .env file which contains my trace loop api key.
When run above program, it always report error as follows, am I doing anything wrong? Thanks
(py310) gyliu@guangyas-air openllmetry % /Users/gyliu/py310/bin/python /Users/gyliu/go/src/github.com/traceloop/openllmetry/packages/sample-app/sample_app/test.py
Traceback (most recent call last):
File "/Users/gyliu/go/src/github.com/traceloop/openllmetry/packages/sample-app/sample_app/test.py", line 6, in <module>
Traceloop.init()
File "/Users/gyliu/py310/lib/python3.10/site-packages/traceloop/sdk/__init__.py", line 64, in init
Traceloop.__fetcher.run()
File "/Users/gyliu/py310/lib/python3.10/site-packages/traceloop/sdk/fetcher.py", line 55, in run
refresh_data(
File "/Users/gyliu/py310/lib/python3.10/site-packages/traceloop/sdk/fetcher.py", line 144, in refresh_data
response = fetch_url(f"{base_url}/v1/traceloop/prompts", api_key)
File "/Users/gyliu/py310/lib/python3.10/site-packages/tenacity/__init__.py", line 289, in wrapped_f
return self(f, *args, **kw)
File "/Users/gyliu/py310/lib/python3.10/site-packages/tenacity/__init__.py", line 379, in __call__
do = self.iter(retry_state=retry_state)
File "/Users/gyliu/py310/lib/python3.10/site-packages/tenacity/__init__.py", line 314, in iter
return fut.result()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/Users/gyliu/py310/lib/python3.10/site-packages/tenacity/__init__.py", line 382, in __call__
result = fn(*args, **kwargs)
File "/Users/gyliu/py310/lib/python3.10/site-packages/traceloop/sdk/fetcher.py", line 101, in fetch_url
raise requests.exceptions.HTTPError(response=response)
requests.exceptions.HTTPError
Run the python script as above
Run this program
from dotenv import load_dotenv
import os
load_dotenv()
from traceloop.sdk import Traceloop
Traceloop.init()
error as follows
(py310) gyliu@guangyas-air openllmetry % /Users/gyliu/py310/bin/python /Users/gyliu/go/src/github.com/traceloop/openllmetry/packages/sample-app/sample_app/test.py
Traceback (most recent call last):
File "/Users/gyliu/go/src/github.com/traceloop/openllmetry/packages/sample-app/sample_app/test.py", line 6, in <module>
Traceloop.init()
File "/Users/gyliu/py310/lib/python3.10/site-packages/traceloop/sdk/__init__.py", line 64, in init
Traceloop.__fetcher.run()
File "/Users/gyliu/py310/lib/python3.10/site-packages/traceloop/sdk/fetcher.py", line 55, in run
refresh_data(
File "/Users/gyliu/py310/lib/python3.10/site-packages/traceloop/sdk/fetcher.py", line 144, in refresh_data
response = fetch_url(f"{base_url}/v1/traceloop/prompts", api_key)
File "/Users/gyliu/py310/lib/python3.10/site-packages/tenacity/__init__.py", line 289, in wrapped_f
return self(f, *args, **kw)
File "/Users/gyliu/py310/lib/python3.10/site-packages/tenacity/__init__.py", line 379, in __call__
do = self.iter(retry_state=retry_state)
File "/Users/gyliu/py310/lib/python3.10/site-packages/tenacity/__init__.py", line 314, in iter
return fut.result()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/Users/gyliu/py310/lib/python3.10/site-packages/tenacity/__init__.py", line 382, in __call__
result = fn(*args, **kwargs)
File "/Users/gyliu/py310/lib/python3.10/site-packages/traceloop/sdk/fetcher.py", line 101, in fetch_url
raise requests.exceptions.HTTPError(response=response)
requests.exceptions.HTTPError
3.10
No response
None
OpenAI Instrumentation
OpenAI's v1 SDK has a with_raw_responses
redirect that returns a different type, LegacyAPIResponse.
The code assumes it's a pydantic model and crashes in this mode.
from opentelemetry import trace
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.richconsole import RichConsoleSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.instrumentation.openai import OpenAIInstrumentor
from openai import AsyncAzureOpenAI, AsyncOpenAI
from azure.identity.aio import DefaultAzureCredential, get_bearer_token_provider
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(RichConsoleSpanExporter()))
OpenAIInstrumentor().instrument()
azure_credential = DefaultAzureCredential(exclude_shared_token_cache_credential=True)
token_provider = get_bearer_token_provider(azure_credential, "https://cognitiveservices.azure.com/.default")
openai_client = AsyncAzureOpenAI(
api_version="2023-07-01-preview",
azure_endpoint="https://<redacted>.openai.azure.com",
azure_ad_token_provider=token_provider,
)
async def test():
# THIS next line
response = await openai_client.embeddings.with_raw_response.create(
model="embedding",
input="Ground control to Major Tom",
)
response.close()
import asyncio
asyncio.run(test())
It should either not trace, but definitely not crash.
Will submit test to reproduce
3.11
No response
Yes I am willing to submit a PR!
Traceloop SDK
We need a way to override and specify the packages to auto-instrument. Right now we do this automatically based on installed packages, which might not fit everyone.
Issues with transformer package we saw with litellm maintainer - BerriAI/litellm#1160
No response
No response
None
OpenAI Instrumentation
Support OpenAI Embeddings API. Note that since outputs are extremely big, we probably shouldn't add them to a span, just the metadata
Complete visibility into usage of LLMs
Similar to current instrumentation for chat / completion APIs
No response
None
OpenAI Instrumentation
Want to log a variable's value in the trace of llm. Where llm is called from a async task. Currently the association_properties are shared by all the tasks.
So basically a way to report a variable's value.
To add more details to the LLM trace that are specific to each llm call made in the same context.
Give a way to log variables inside a llm traces. This should only be set for local scope of the function or the linked functions/processes. Should not be updated globally.
I tried setting the association_properties within each task those were called from a common parent. But the variable's value was getting overwritten globally. Each time I was getting the most recent value.
None
Anthropic Instrumentation
Can we update the data format to MM/DD/YY, current format is DD/MM/YY
Check filter on the UI page
Can we update the data format to MM/DD/YY, current format is DD/MM/YY
No response
No response
None
All Packages
I've noticed some instrumentations only check for the kwargs
passed by the method. For instance, in ChromaDB:
def _set_add_attributes(span, kwargs):
_set_span_attribute(span, "db.chroma.add.ids_count", count_or_none(kwargs.get("ids")))
_set_span_attribute(span, "db.chroma.add.embeddings_count", count_or_none(kwargs.get("embeddings")))
_set_span_attribute(span, "db.chroma.add.metadatas_count", count_or_none(kwargs.get("metadatas")))
_set_span_attribute(span, "db.chroma.add.documents_count", count_or_none(kwargs.get("documents")))
This has the implication that depending on the user calls the function, he might miss some attributes from being added to the trace.
For instance:
chromadb.some_function("1", "books")
Would not instrument the arguments, while calling:
chromadb.some_function(id="1", collection="books")
Would successfully include attributes id
and collection
in the trace.
Make instrumentations more robust / reliable / predictable. We don't want the behavior to change depending on the user calls a method.
We should make this is baked-in in the instrumentation tooling, so that this always handled correctly. We need some code design proposal done and experimented with. This is what I've implemented in weaviate instrumentation to avoid the issue and could serve as a starting point:
class ArgsGetter:
"""Helper to make sure we get arguments regardless
of whether they were passed as args or as kwargs.
Additionally, cast serializes dicts to JSON string.
"""
def __init__(self, args, kwargs):
self.args = args
self.kwargs = kwargs
def __call__(self, index, name):
try:
obj = self.args[index]
except IndexError:
obj = self.kwargs.get(name)
if obj:
try:
return json.dumps(obj)
except json.decoder.JSONDecodeError:
logger.warning(
"Failed to decode argument (%s) (%s) to JSON", index, name
)
class _Instrumentor:
def map_attributes(self, span, method_name, attributes, args, kwargs):
getter = ArgsGetter(args, kwargs)
for idx, attribute in enumerate(attributes):
_set_span_attribute(
span,
f"{self.namespace}.{method_name}.{attribute}",
getter(idx, attribute),
)
def instrument(self, method_name, span, args, kwargs):
attributes = self.mapped_attributes.get(method_name)
if attributes:
self.map_attributes(span, method_name, attributes, args, kwargs)
Note, this could also be labeled as a bug, but I think the goal here is not to simple fix cases where it might misbehave. It's to prevent it from even happening again in the future.
Yes I am willing to submit a PR!
Traceloop SDK
adding ibm_watsonx_ai
package support to IBM watsonx traceloop instrumentation
IBM watsonx has ibm_watsonx_ai
and ibm_watson_machine_learning
packages, the current instrumentation only support instrumentation of ibm_watson_machine_learning
. plus the ibm_watsonx_ai
package is also what langchain_community llms supports, so this support is also needed to get langchain watsonx trace.
I will create a PR based on https://github.com/huang-cn/traceloop-openllmetry/tree/watson_ml_moduel_support/packages/opentelemetry-instrumentation-watsonx
No response
Yes I am willing to submit a PR!
Anthropic Instrumentation
Similar to what we have for OpenAI
It will help us stabilize semantic conversions as we'll have another example of an LLM call
No response
None
Anthropic Instrumentation
Integrating LanceDB
Flexibility to switch between VectorDBs
In addition with other VectorDBs, LanceDB can also be integrated in same way.
No response
Yes I am willing to submit a PR!
Anthropic Instrumentation
I have backend opentelemetry-server, how to use opentelemetry-instrumentation-langchain to collector our langchain tracing
I can see opentelemetry-instrumentation-langchain , but not found readme
use our langchain project
No response
None
All Packages
Some values, like token usage, are better reported (also) as otel metrics. We should update all instrumentations to send metrics as well as traces.
More ways to use the outputted data. Observability platforms can use metrics to set alerts on or to create dashboards.
This needs to be researched and defined. Please consult on slack before starting to work on this.
No response
None
Traceloop SDK
Right now, we only allow setting the service name when initializing the Traceloop SDK. We need to allow providing an arbitrary list of attributes (which is a dictionary) in addition so that users can set other resource attributes.
For Splunk integration - traceloop/docs#4
As an initialization variable to the SDK
No response
Yes I am willing to submit a PR!
Traceloop SDK
Traceloop instrumentation only have tracer, how about add meter and logger to expose metrics and logs?
opentelemetry support log, metrics and tracing, we should support all those.
enable meter and logger.
No response
Yes I am willing to submit a PR!
Anthropic Instrumentation
If there are sth wrong with LLM Provider instrumentation code, the Traceloop.init() will be failed. One example is I want to run a watsonx service, and llamaindex init failed, this caused I cannot run my watsonx service.
(py311) gyliu@Guangyas-MacBook-Air openllmetry % /Users/gyliu/py311/bin/python /Users/gyliu/go/src/github.com/traceloop/openllmetry/
packages/sample-app/sample_app/watsonx_generate.py
Traceloop syncing configuration and prompts
Traceloop exporting traces to https://api.traceloop.com authenticating with bearer token
/Users/gyliu/py311/lib/python3.11/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.chat_models import ChatAnyscale`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/Users/gyliu/py311/lib/python3.11/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.chat_models import ChatOpenAI`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/Users/gyliu/py311/lib/python3.11/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.embeddings import HuggingFaceBgeEmbeddings`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/Users/gyliu/py311/lib/python3.11/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.embeddings import HuggingFaceEmbeddings`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/Users/gyliu/py311/lib/python3.11/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.llms import AI21`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/Users/gyliu/py311/lib/python3.11/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.llms import Cohere`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/Users/gyliu/py311/lib/python3.11/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.llms import FakeListLLM`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
/Users/gyliu/py311/lib/python3.11/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.llms import OpenAI`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
Traceback (most recent call last):
File "/Users/gyliu/go/src/github.com/traceloop/openllmetry/packages/sample-app/sample_app/watsonx_generate.py", line 9, in <module>
Traceloop.init()
File "/Users/gyliu/py311/lib/python3.11/site-packages/traceloop/sdk/__init__.py", line 152, in init
Traceloop.__tracer_wrapper = TracerWrapper(
^^^^^^^^^^^^^^
File "/Users/gyliu/py311/lib/python3.11/site-packages/traceloop/sdk/tracing/tracing.py", line 111, in __new__
init_instrumentations()
File "/Users/gyliu/py311/lib/python3.11/site-packages/traceloop/sdk/tracing/tracing.py", line 266, in init_instrumentations
init_llama_index_instrumentor()
File "/Users/gyliu/py311/lib/python3.11/site-packages/traceloop/sdk/tracing/tracing.py", line 360, in init_llama_index_instrumentor
from opentelemetry.instrumentation.llamaindex import LlamaIndexInstrumentor
File "/Users/gyliu/py311/lib/python3.11/site-packages/opentelemetry/instrumentation/llamaindex/__init__.py", line 14, in <module>
from opentelemetry.instrumentation.llamaindex.custom_llm_instrumentor import CustomLLMInstrumentor
File "/Users/gyliu/py311/lib/python3.11/site-packages/opentelemetry/instrumentation/llamaindex/custom_llm_instrumentor.py", line 14, in <module>
from llama_index.llms.custom import CustomLLM
File "/Users/gyliu/py311/lib/python3.11/site-packages/llama_index/__init__.py", line 17, in <module>
from llama_index.embeddings.langchain import LangchainEmbedding
File "/Users/gyliu/py311/lib/python3.11/site-packages/llama_index/embeddings/__init__.py", line 18, in <module>
from llama_index.embeddings.huggingface import (
File "/Users/gyliu/py311/lib/python3.11/site-packages/llama_index/embeddings/huggingface.py", line 17, in <module>
from llama_index.llms.huggingface import HuggingFaceInferenceAPI
File "/Users/gyliu/py311/lib/python3.11/site-packages/llama_index/llms/__init__.py", line 25, in <module>
from llama_index.llms.litellm import LiteLLM
File "/Users/gyliu/py311/lib/python3.11/site-packages/llama_index/llms/litellm.py", line 28, in <module>
from llama_index.llms.litellm_utils import (
File "/Users/gyliu/py311/lib/python3.11/site-packages/llama_index/llms/litellm_utils.py", line 4, in <module>
from openai.openai_object import OpenAIObject
ModuleNotFoundError: No module named 'openai.openai_object'
Enable init can only init a specified llm provider.
as Reproduction steps
No response
No response
Yes I am willing to submit a PR!
Traceloop SDK
Currently, all prompts are logged by default. This might be problematic for some use cases, so we need an option to disable that for a specific task / workflow.
Sometimes prompts can be sensitive, or just too big in size and disabling it is required by the user.
We need to properly design this, as we need to propagate this option from the SDK to the specific instrumentations.
No response
None
All Packages
Instrument calls to Google's Vertex AI, including adding attributes for input parameters, model, etc. - similarly to our Anthropic instrumentation. The instrumentation should support all types of calls - streaming, non streaming, async, etc.
This should specifically work with Google's new Gemini model.
Completeness of OpenLLMetry
Similar to other instrumentations we have in this repo.
None
Anthropic Instrumentation
The supported destination do not have IBM Instana yet, we will enable Instana as well.
Instana already support OTEL, and the instrumentation for AI can help Instana support AI Observability.
Test with IBM Watsonx and langchain first, depend on #341
No response
Yes I am willing to submit a PR!
LlamaIndex Instrumentation
Right now, we monkey-patch classes and methods in LlamaIndex which requires endless work and constant maintenance. LlamaIndex has a system for callbacks that can potentially be used to create/end spans without being too coupled with with the framework's inner structure.
Support LlamaIndex entirely and be future-proof to internal API changes
Look into LlamaIndex callback_manager
and how other frameworks are using it.
No response
None
Traceloop SDK
I'm not sure if this bug report should be in the docs
repo or here, but I am trying to send traces to Azure Application Insights using traceloop and getting different results that what is documented.
Run the sample code from https://www.traceloop.com/docs/openllmetry/integrations/azure
I expected to see a trace but instead am seeing the call show up as a dependency in Application Insights
The screenshot from the docs shows there are 4 traces and there seems to be an extra llm_span
that I don't have on my end.
When I ran the code form the docs, Traces
are coming through as Dependencies
. From the screenshot below, you can see that Traces
are 0.
3.11.5
In addition, the code from the documentation needs some updates:
os
importresponse['choices'][0]['message']['content']
should be updated to response.choices[0].message.content
.note I would normally just submit a PR with these changes but since I am running into different behavior than what is documented, I chose to open an issue instead.
Yes I am willing to submit a PR!
All Packages
We need to extract and report embedding vectors that are sent and returned for query
calls to vector DBs like Chroma, Weaviate, Pinecone, etc.
More visiblity into what's happening when calling a vector DB
This isn't straightforward - these might be large, so sending them as attributes might not make sense and we'll need to think of the proper OTEL format to use (logs? span events?)
Consult on slack before working on it.
None
All Packages
Instrument calls to Milvus, including adding attributes, similarly to our Chroma instrumentation. The instrumentation should support all types of calls - streaming, non streaming, async, etc.
Completness of OpenLLMetry
Similarily to other instrumentations we have in this repo.
None
Traceloop SDK
When providing the wrong TRACELOOP_API_KEY
the following error is thrown, making it hard to trace back the reason (which is a bad API key). We should output a better log for that.
Traceback (most recent call last):
File "/Users/nirga/vecinity/openllmetry/packages/sample-app/sample_app/prompt_registry_example_app.py", line 10, in <module>
Traceloop.init(app_name="prompt_registry_example_app")
File "/Users/nirga/vecinity/openllmetry/packages/traceloop-sdk/traceloop/sdk/__init__.py", line 62, in init
Traceloop.__fetcher.run()
File "/Users/nirga/vecinity/openllmetry/packages/traceloop-sdk/traceloop/sdk/fetcher.py", line 55, in run
refresh_data(
File "/Users/nirga/vecinity/openllmetry/packages/traceloop-sdk/traceloop/sdk/fetcher.py", line 144, in refresh_data
response = fetch_url(f"{base_url}/v1/traceloop/prompts", api_key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/nirga/vecinity/openllmetry/packages/sample-app/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 289, in wrapped_f
return self(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/Users/nirga/vecinity/openllmetry/packages/sample-app/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 379, in __call__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/nirga/vecinity/openllmetry/packages/sample-app/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 326, in iter
raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x11492b910 state=finished raised HTTPError>]
Choose a random TRACELOOP_API_KEY
and set TRACELOOP_BASE_URL
to api.traceloop.com
(or leave empty).
Some output saying something like Authorization error: invalid API key.
See log message in the description
No response
No response
None
All Packages
We should improve our instrumentation wrappers to reduce the required amount of boilerplate per project.
The end-goal is to drive the instrumentations towards a more declarative approach, as much as possible. There will always be exceptions, so the solution also needs to be flexible with this.
Less maintenance overhead.
We'll need to write some custom tooling for this.
In the weaviate instrumentation PR, I wrote some simple classes to reduce boilerplate, for instance:
class _BatchInstrumentor(_Instrumentor):
namespace = "db.weaviate.batch"
mapped_attributes = {
"add_data_object": [
"data_object",
"class_name",
"uuid",
"vector",
"tenant",
],
"flush": [],
}
We could try and take this further. Python offers an amazing introspection module in the standard library (inspect
), so in theory just by passing the function name we should be able to automatically extract all arguments. That should be the default way to instrument methods.
Using such resources, we can then simply map in a declarative way most of the standard objects / functions, like we already do:
WRAPPED_METHODS = [
{
"package": chromadb.api.segment,
"object": "SegmentAPI",
"method": "_query",
"span_name": "chroma.query.segment._query"
},
{
"package": chromadb,
"object": "Collection",
"method": "add",
"span_name": "chroma.add"
},
But without the need to explicitly add calls to set span attributes, as our instrumentation tooling should automatically take care of that:
def _set_add_attributes(span, kwargs):
_set_span_attribute(span, "db.chroma.add.ids_count", count_or_none(kwargs.get("ids")))
_set_span_attribute(span, "db.chroma.add.embeddings_count", count_or_none(kwargs.get("embeddings")))
_set_span_attribute(span, "db.chroma.add.metadatas_count", count_or_none(kwargs.get("metadatas")))
_set_span_attribute(span, "db.chroma.add.documents_count", count_or_none(kwargs.get("documents")))
No response
Yes I am willing to submit a PR!
All Packages
As the instrumentations grew, there are some semantic conventions that should be extracted out of the individual packages into the common one, namely:
Order in the repo :)
.
No response
None
VertexAI Instrumentation
Following #413, I had to disable the VertexAI tests since vcr.py doesn't support GRPC. We need to figure out how to mock those requests to avoid making actual calls to GRPC.
N/A
N/A
N/A
No response
No response
None
Traceloop SDK
Combined with the haystack implementation, traces sometime do not appear in the dashboard.
This is a simple example that when run in a colab, do not always product traces:
import os
from haystack.nodes import PromptNode, PromptTemplate, AnswerParser
from haystack.pipelines import Pipeline
from traceloop.sdk import Traceloop
from traceloop.sdk.decorators import workflow
Traceloop.init(app_name="haystack_app")
prompt = PromptTemplate(
prompt="Tell me a joke about {query}\n",
output_parser=AnswerParser(),
)
prompt_node = PromptNode(
model_name_or_path="gpt-4",
api_key=os.getenv("OPENAI_API_KEY"),
default_prompt_template=prompt,
)
pipeline = Pipeline()
pipeline.add_node(component=prompt_node, name="PromptNode", inputs=["Query"])
query = "OpenTelemetry"
result = pipeline.run(query)
print(result["answers"][0].answer)
Traces should be produced
See deepset-ai/haystack-integrations#51 (comment)
No response
This doesn't happen if we explicitly annotate methods with @workflow
. The main difference is that this annotation forces flush of traces at the end of traces. We did recently add an auth-flush on app shutdown - but this probably never gets called on notebooks.
Yes I am willing to submit a PR!
OpenAI Instrumentation
In addition to the destinations currently supported to catch traces, it would be great to extend and include Azure Application Insights. OpenTelemetry is a key recommendation for almost any solution deployed on Azure as they should be tracking traces and metrics, and it would be great to have native support.
I was going to add links to Azure documentation for OTEL + App Insights... Realized it might just be easier to send this ChatGPT instruction for a sample Python app instrumented with OTEL, sending traces to App Insights.
https://chat.openai.com/share/8b6cc9a7-b8c5-4dea-8e0b-7c6728ff40a0
Holistic Azure solution, e.g. using Azure OpenAI and sending metrics to Azure Application Insights. All data would be airgapped and stored in your own subscription.
Support Azure App Insights as a trace export destination, using instrumentation key and endpoint.
No response
Yes I am willing to submit a PR!
Chromadb Instrumentation
When I try to use the sample app based on OpenAI, I get a deprecated error and the script fails to execute.
python sample_app/chroma_app.py
Script should run without errors. It should support the same OpenAI version defined in the test dependencies.
(venv) (base) paolo@paolo-MS-7D08:~/dev/openllmetry/packages/sample-app$ python sample_app/chroma_app.py
Traceloop syncing configuration and prompts
Traceloop exporting traces to https://api.traceloop.com authenticating with bearer token
Traceback (most recent call last):
File "/home/paolo/dev/openllmetry/packages/sample-app/sample_app/chroma_app.py", line 97, in <module>
assess_claims(samples["claim"].tolist())
File "/home/paolo/dev/openllmetry/packages/traceloop-sdk/traceloop/sdk/decorators/__init__.py", line 102, in wrap
return fn(*args, **kwargs)
File "/home/paolo/dev/openllmetry/packages/sample-app/sample_app/chroma_app.py", line 82, in assess_claims
response = openai.ChatCompletion.create(
File "/home/paolo/dev/openllmetry/packages/traceloop-sdk/venv/lib/python3.10/site-packages/openai/lib/_old_api.py", line 39, in __call__
raise APIRemovedInV1(symbol=self._symbol)
openai.lib._old_api.APIRemovedInV1:
You tried to access openai.ChatCompletion, but this is no longer supported in openai>=1.0.0 - see the README at https://github.com/openai/openai-python for the API.
You can run `openai migrate` to automatically upgrade your codebase to use the 1.0.0 interface.
Alternatively, you can pin your installation to the old version, e.g. `pip install openai==0.28`
A detailed migration guide is available here: https://github.com/openai/openai-python/discussions/742
3.10.10
This is the same issue I've encountered when working in this PR: #368
Specifically, I had to update this file:
packages/sample-app/sample_app/pinecone_app.py
Using the existing code from packages/traceloop-sdk/tests/test_pinecone_instrumentation.py
was a quick way to do it, since in the tests the code is up-to-date.
It might be that this affects additional packages, I haven't checked other cases.
Yes I am willing to submit a PR!
Traceloop SDK
Add a setting called trace_content
(which will default to true
) and will control whether all the instrumentations are sending sensitive content (like prompts and completions). This should override the env var behavior (similar to other flags we have).
More control on this sensitive feature
No response
None
Traceloop SDK
Enable traceloop can collect metrics and tracing for IBM Watsonx, we have a prototype at https://github.com/gyliu513/langX101/tree/main/otel/opentelemetry-instrumentation-watsonx
Before OTEL community reach to an agreement of the AI Semantic Convention at open-telemetry/semantic-conventions#639 , this repo is the best place to host most of the instrumentation for different llm providers.
I will create a PR based on https://github.com/gyliu513/langX101/tree/main/otel/opentelemetry-instrumentation-watsonx
No response
Yes I am willing to submit a PR!
All Packages
Could you please update pydantic to v2?
Version resolving fails if any of the packages has a contstraint for >2 eg. pydantic-settings.
Because no versions of traceloop-sdk match >0.3.0,<0.4.0
and traceloop-sdk (0.3.0) depends on pydantic (>=1.10.12,<2.0.0), traceloop-sdk (>=0.3.0,<0.4.0) requires pydantic (>=1.10.12,<2.0.0).
And because pydantic-settings (2.0.3) depends on pydantic (>=2.0.1)
and no versions of pydantic-settings match >2.0.3,<3.0.0, traceloop-sdk (>=0.3.0,<0.4.0) is incompatible with pydantic-settings (>=2.0.3,<3.0.0).
So, because theydo-journey-ai depends on both pydantic-settings (^2.0.3) and traceloop-sdk (^0.3.0), version solving failed.
It'll need some work probably.
No response
Yes I am willing to submit a PR!
All Packages
If an HTTP error is returned from a foundation model API, we don't properly log it as a failed span.
N/A
N/A
N/A
No response
No response
None
All Packages
Right now our entire testing suite is more like "integration tests" - we make actual calls to models and check the created spans. This is way too complex and flaky and we should have simple unit tests that we can run in PRs to test that we didn't break each instrumentation. These should be part of each instrumentation package by itself, where we mock the responses from the likes of OpenAI instead of making actual calls.
Lighter, less flaky tests.
See feature decsription
No response
None
All Packages
When a chain is built using LCEL, traces are not emitted.
Traces should be emitted for langchain blocks.
Only traces emitted :
{
"name": "PATCH",
"context": {
"trace_id": "0x5d226499992625813276d417daf03aad",
"span_id": "0x52cf0fa0514c33a5",
"trace_state": "[]"
},
"kind": "SpanKind.CLIENT",
"parent_id": null,
"start_time": "2024-01-29T11:06:14.970928Z",
"end_time": "2024-01-29T11:06:15.404928Z",
"status": {
"status_code": "UNSET"
},
"attributes": {
"http.method": "PATCH",
"http.url": "https://api.smith.langchain.com/runs/97d3bdd0-8a1e-4437-9401-84e2004f16d8",
"http.status_code": 200
},
"events": [],
"links": [],
"resource": {
"attributes": {
"service.name": "src/streamlit_main.py"
},
"schema_url": ""
}
}
{
"name": "POST",
"context": {
"trace_id": "0xe6cf84bd7a5a72d6220d76528fc6fa32",
"span_id": "0x0aa04c6a4dbf1bfd",
"trace_state": "[]"
},
"kind": "SpanKind.CLIENT",
"parent_id": null,
"start_time": "2024-01-29T11:06:14.970581Z",
"end_time": "2024-01-29T11:06:15.430682Z",
"status": {
"status_code": "UNSET"
},
"attributes": {
"http.method": "POST",
"http.url": "https://api.smith.langchain.com/runs",
"http.status_code": 200
},
"events": [],
"links": [],
"resource": {
"attributes": {
"service.name": "src/streamlit_main.py"
},
"schema_url": ""
}
}
{
"name": "POST",
"context": {
"trace_id": "0xfa2c996295cfda4103f37090627791c7",
"span_id": "0x5c3d0c3dd6bd1d51",
"trace_state": "[]"
},
"kind": "SpanKind.CLIENT",
"parent_id": null,
"start_time": "2024-01-29T11:06:14.971143Z",
"end_time": "2024-01-29T11:06:15.441099Z",
"status": {
"status_code": "UNSET"
},
"attributes": {
"http.method": "POST",
"http.url": "https://api.smith.langchain.com/runs",
"http.status_code": 200
},
"events": [],
"links": [],
"resource": {
"attributes": {
"service.name": "src/streamlit_main.py"
},
"schema_url": ""
}
}
{
"name": "POST",
"context": {
"trace_id": "0xeff8835537de179427d32f928d2b244c",
"span_id": "0xa010e98b7ec99bc6",
"trace_state": "[]"
},
"kind": "SpanKind.CLIENT",
"parent_id": null,
"start_time": "2024-01-29T11:06:14.971049Z",
"end_time": "2024-01-29T11:06:15.471814Z",
"status": {
"status_code": "UNSET"
},
"attributes": {
"http.method": "POST",
"http.url": "https://api.smith.langchain.com/runs",
"http.status_code": 200
},
"events": [],
"links": [],
"resource": {
"attributes": {
"service.name": "src/streamlit_main.py"
},
"schema_url": ""
}
}
INFO: HTTP Request: POST https://navira-poc-02.openai.azure.com//openai/deployments/gpt-35-turbo/chat/completions?api-version=2023-09-01-preview "HTTP/1.1 200 OK"
{
"name": "openai.chat",
"context": {
"trace_id": "0xa3b49a57b3990fdd94f44b647ab7f4fb",
"span_id": "0xe925e3010d0e482a",
"trace_state": "[]"
},
"kind": "SpanKind.CLIENT",
"parent_id": null,
"start_time": "2024-01-29T11:06:14.955512Z",
"end_time": "2024-01-29T11:06:15.976070Z",
"status": {
"status_code": "UNSET"
},
"attributes": {
"llm.request.type": "chat",
"openai.api_version": "2023-09-01-preview",
"llm.vendor": "OpenAI",
"llm.request.model": "gpt-3.5-turbo",
"llm.temperature": 0.0,
"llm.headers": "None",
"llm.prompts.0.role": "user",
"llm.prompts.0.content": "You are an AWS cloud consultant tasked with recognizing the intent of users' queries in the context of cloud cost governance. Pay close attention to the exact definitions of allowed categories & services. DO NOT deviate from them STRICTLY.\n\nContext: \nThe intent consists of 2 parts : category and service.\n\nThe possible categories of user's intent. \n- RESOURCE_METADATA_DISCOVERY: Involves querying metadata or attributes of user provisioned AWS cloud resources. This can include details like configurations, provisioned capacity, tags and status.\n- RESOURCE_USAGE_DISCOVERY: Focuses on analyzing resource utilization metrics of the user's provisioned resources such as performance metrics and usage patterns across different metrics.\n- RESOURCE_COST_DISCOVERY: It involves evaluating cost-related data of user's provisioned resources to gain insights into expenditure patterns and identify cost drivers.\n- PRICING_DISCOVERY: Dedicated to exploring the general public pricing details of AWS services (not related to the user's data). This includes obtaining information about service rates and understanding different pricing models (e.g., on-demand, reserved instances).\n- RECOMMENDATION: Includes providing suggestions and actionable advice for optimizing AWS cloud resource usage, costs & performance.\n\nThe services related to the user's intent.\n- EC2: Amazon Elastic Compute Cloud\n- RDS: Amazon Relational Database Service\n- S3: Amazon Simple Storage Service\n- DYNAMODB: Amazon DynamoDB\n- EBS: Amazon Elastic Block Storage\n- OPENSEARCH_SERVICE: Amazon OpenSearch Service\n- CLOUDWATCH: Amazon CloudWatch\n- ELASTICACHE: Amazon ElastiCache\n- VPC: Amazon Virtual Private Cloud\n- ELB: Amazon Elastic Load Balancing\n- CLOUDFRONT: Amazon CloudFront\nOutput Format:The output should be formatted as a JSON instance that conforms to the JSON schema below.\n\nAs an example, for the schema {"properties": {"foo": {"title": "Foo", "description": "a list of strings", "type": "array", "items": {"type": "string"}}}, "required": ["foo"]}\nthe object {"foo": ["bar", "baz"]} is a well-formatted instance of the schema. The object {"properties": {"foo": ["bar", "baz"]}} is not well-formatted.\n\nHere is the output schema:\n\n{\"description\": \"Serializable base class.\", \"properties\": {\"category\": {\"description\": \"The category of the user's intent.\", \"allOf\": [{\"$ref\": \"#/definitions/IntentCategory\"}]}, \"service\": {\"description\": \"The service related to the user's intent.\", \"allOf\": [{\"$ref\": \"#/definitions/AmazonService\"}]}}, \"required\": [\"category\", \"service\"], \"definitions\": {\"IntentCategory\": {\"title\": \"IntentCategory\", \"description\": \"An enumeration.\", \"enum\": [\"RESOURCE_METADATA_DISCOVERY\", \"RESOURCE_USAGE_DISCOVERY\", \"RESOURCE_COST_DISCOVERY\", \"COST_DISCOVERY\", \"RECOMMENDATION\"], \"type\": \"string\"}, \"AmazonService\": {\"title\": \"AmazonService\", \"description\": \"An enumeration.\", \"enum\": [\"EC2\", \"RDS\", \"S3\", \"DYNAMODB\", \"EBS\", \"OPENSEARCH_SERVICE\", \"CLOUDWATCH\", \"ELASTICACHE\", \"VPC\", \"ELB\", \"CLOUDFRONT\"], \"type\": \"string\"}}}\n
\nUser's query: how many ec2 instances\n",
"llm.response.model": "gpt-35-turbo",
"llm.usage.total_tokens": 735,
"llm.usage.completion_tokens": 20,
"llm.usage.prompt_tokens": 715,
"llm.completions.0.finish_reason": "stop",
"llm.completions.0.role": "assistant",
"llm.completions.0.content": "{\n "category": "RESOURCE_METADATA_DISCOVERY",\n "service": "EC2"\n}"
},
"events": [],
"links": [],
"resource": {
"attributes": {
"service.name": "src/streamlit_main.py"
},
"schema_url": ""
}
}
{
"name": "PATCH",
"context": {
"trace_id": "0xd95ec01e44229f05b4844ca8325c94c2",
"span_id": "0x9dc9086284294043",
"trace_state": "[]"
},
"kind": "SpanKind.CLIENT",
"parent_id": null,
"start_time": "2024-01-29T11:06:16.068286Z",
"end_time": "2024-01-29T11:06:16.367807Z",
"status": {
"status_code": "UNSET"
},
"attributes": {
"http.method": "PATCH",
"http.url": "https://api.smith.langchain.com/runs/39ee576a-6e25-4260-9ba7-d6e080e65ec9",
"http.status_code": 200
},
"events": [],
"links": [],
"resource": {
"attributes": {
"service.name": "src/streamlit_main.py"
},
"schema_url": ""
}
}
{
"name": "POST",
"context": {
"trace_id": "0x6e2c49a93463ceb0965582c405be317e",
"span_id": "0x9f801f0e5058c0ba",
"trace_state": "[]"
},
"kind": "SpanKind.CLIENT",
"parent_id": null,
"start_time": "2024-01-29T11:06:16.068165Z",
"end_time": "2024-01-29T11:06:16.398113Z",
"status": {
"status_code": "UNSET"
},
"attributes": {
"http.method": "POST",
"http.url": "https://api.smith.langchain.com/runs",
"http.status_code": 200
},
"events": [],
"links": [],
"resource": {
"attributes": {
"service.name": "src/streamlit_main.py"
},
"schema_url": ""
}
}
{
"name": "PATCH",
"context": {
"trace_id": "0x66442059532c14185b5da4b6c0e3e2a5",
"span_id": "0xf8da2f9a81d0d8e1",
"trace_state": "[]"
},
"kind": "SpanKind.CLIENT",
"parent_id": null,
"start_time": "2024-01-29T11:06:16.067977Z",
"end_time": "2024-01-29T11:06:16.398613Z",
"status": {
"status_code": "UNSET"
},
"attributes": {
"http.method": "PATCH",
"http.url": "https://api.smith.langchain.com/runs/5260c5a9-68c0-4910-b206-5986e4244dac",
"http.status_code": 200
},
"events": [],
"links": [],
"resource": {
"attributes": {
"service.name": "src/streamlit_main.py"
},
"schema_url": ""
}
}
{
"name": "PATCH",
"context": {
"trace_id": "0xa4309c9db74617d83e689ac2fcaeae3d",
"span_id": "0x52d896629a51553c",
"trace_state": "[]"
},
"kind": "SpanKind.CLIENT",
"parent_id": null,
"start_time": "2024-01-29T11:06:16.067677Z",
"end_time": "2024-01-29T11:06:16.402780Z",
"status": {
"status_code": "UNSET"
},
"attributes": {
"http.method": "PATCH",
"http.url": "https://api.smith.langchain.com/runs/3047f648-72af-4c4e-b223-dbb38c97002e",
"http.status_code": 200
},
"events": [],
"links": [],
"resource": {
"attributes": {
"service.name": "src/streamlit_main.py"
},
"schema_url": ""
}
}
3.11.7
No response
None
Pinecone Instrumentation
Add a basic instrumentation for Pinecone requests - specifically the most important ones are query
requests.,
Enrich our instrumentation library
No response
None
Anthropic Instrumentation
Anthropic provides an async API as well as a sync API:
https://github.com/anthropics/anthropic-sdk-python?tab=readme-ov-file#async-usage
Both should be supported as part of the instrumentation
No response
This can be supported similarily to our OpenAI instrumentation
None
Traceloop SDK
Traceloop SDK was upgraded to use pydantic
v2, but this causes users with old pydantic
versions to not able to use our SDK. Some frameworks, like Haystack v1, also rely on pydantic v1 and so cannot be used together with our SDK. We need to support both pydantic
v1 and v2.
Use Traceloop SDK with pydantic
v1 -> version conflict
pydantic
v1 should also be supported
N/A
No response
This can be solved similarily to how Anthropic SDK does that.
None
All Packages
Instrument calls to Replicate, including adding attributes for input parameters, model, etc. - similarly to our Anthropic instrumentation. The instrumentation should support all types of calls - streaming, non streaming, async, etc.
Completeness of OpenLLMetry
Similar to other instrumentations we have in this repo.
None
Traceloop SDK
We're starting to hit some issues with dependency conflicts, which prevents us from upgrading key packages that we instrument in the library.
One of these conflicts, for instance, is how weaviate-client v4 depends on pydantic>=2 but haystack v1 depends on pydantic==1.
In theory, there's no need to always have all dependencies from all instrumentations to be always resolved simultaneously. We should instead make Traceloop SDK dependency light, and declare the instrumentations as optional dependencies.
One could then, install the specific instrumentations that he desires, like this:
pip install traceloop[pinecone, openai]
This would also enable us to simultaneously support many versions of the same libraries, as we can effectively just declare a new optional dependency for each version we're instrumenting.
โ๏ธ How do you aim to achieve this?
Update to pyproject.toml on traceloop-sdk, needs additional research on details.
๐๏ธ Additional Information
Many libraries in the wild offer this, we can find one and use their configuration as a starting point.
๐ Have you spent some time to check if this feature request has been raised before?
I checked and didn't find similar issue
Are you willing to submit PR?
Yes I am willing to submit a PR!
OpenAI Instrumentation
The OpenAI instrumentation is great, but is missing some span attributes compared with the httpx instrumentor that would be useful to capture. I'd like to know what the client attributes were for the OpenAI client like the host and the deployment name. Those are in SpanAttributes.HTTP_METHOD
and SpanAttributes.HTTP_URL
.
I'm using different OpenAI backends (Azure OpenAI) and want to capture the deployment name, the host name and other information that is in the client.
Add some additional span attributes to the trace that capture the properties of the open ai client object.
No response
Yes I am willing to submit a PR!
Watsonx Instrumentation
This is a follow up for #526, need check if watsonx support python 3.8
Check #526 for detail
Watsonx need to work with python 3.8
Check #526
No response
No response
None
OpenAI Instrumentation
I have an app running where I am streaming the tokens of my answers, streaming is working fine but when I initialize Traceloop using LlamaIndex the streaming features is disabled.
Streaming of tokens should still be working
Tokens are not streamed.
3.11.5
No response
None
All Packages
I can see https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation already have some python instrumentation code, any plan to upstream this to otel? Thanks
otel is the official repo for all instrumentations
move all repos to https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation
No response
Yes I am willing to submit a PR!
Traceloop SDK
Spans are shown with unknown_service
instead of the right service name as initialized in Traceloop.init()
Traceloop.init("some_service_name")
some_service_name
should be shown
No response
No response
Yes I am willing to submit a PR!
All Packages
Instrument calls to Weaviate, including adding attributes, similarly to our Chroma instrumentation. The instrumentation should support all types of calls - streaming, non streaming, async, etc.
Completeness of OpenLLMetry
Similar to other instrumentations, we have in this repo.
None
Traceloop SDK
on 3.8.13
, i get the error
class TracerWrapper(object):
File "/Users/jferge/code/ai/python-demo-project/venv/lib/python3.8/site-packages/traceloop/sdk/tracing/tracing.py", line 155, in TracerWrapper
headers: dict[str, str],
TypeError: 'type' object is not subscriptable
it appears that this is python 3.9 syntax, and needs from __future__ import annotations
for 3.8.
In the project's pyproject.toml, it appears 3.8 should be supported. are multiversion tests being ran?
https://github.com/traceloop/openllmetry/blob/main/packages/traceloop-sdk/pyproject.toml
use project in python 3.8
should not error upon import
errors upon import
3.8.13
No response
None
Chromadb Instrumentation
An error is logged when running a query with Chroma.
Use chromadb = "^0.4.22"
, and run the llama_index_chroma_app.py
from the sample_app.
No error should be thrown.
3.9.5
No response
None
Langchain Instrumentation
Right now, we monkey-patch classes and methods in LlamaIndex which requires endless work and constant maintenance. Langchain has a system for callbacks that can potentially be used to create/end spans without being too coupled with with the framework's inner structure.
Support Langchain entirely and be future-proof to internal API changes
Look into Langchain callbacks and how other frameworks are using it.
No response
None
Traceloop SDK
Support tracing of calls to the qdrant vector database
For users of qdrant
A new qdrant extension
No response
None
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.