Description

Testing plays a vital role in maintaining the reliability and stability of our project. While we have some test coverage, we need to expand and strengthen our testing suite to catch issues early in development.

We propose adding comprehensive unit and integration tests to our project. This effort will improve code quality, reduce the risk of regressions, and boost confidence in our project's reliability.

Proposed Tasks:

• Develop unit tests for critical components.
• Create integration tests to validate component interactions.
• Ensure high test coverage.
• Integrate testing into our CI/CD pipeline.
• Document testing procedures and guidelines.

Note : Those who are interested can engage with the project maintainers through our Discord channel for more details.

Allow tool_choice, tools, logprobs and top_logprobs which are valid params for OpenAI Chat completions API.
Also remove strict type for message content as it can be null in case of function calling response

Contact Details

[email protected]

What happened?

I'm using portkey-ai package along with langchain and chroma. chroma cannot work with typing-extensions<4.8.0, but portkey-ai package has strict dependency on typing-extensions==4.7.1

Is it possible to support a wider range of versions for typing-extensions?

Version

0.1.xx (Default)

Relevant log output

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
portkey-ai 1.1.1 requires typing-extensions==4.7.1, but you have typing-extensions 4.8.0 which is incompatible.

Code of Conduct

• I agree to follow this project's Code of Conduct

Create a new account for azure ai models for testing with portkey credentials.

Enable support for prompt/render route

Description

The images in the readme files are using the local paths and it doesn't work in the pypi org or anyother platform where this is deployed.
This needs to be updated with a CDN url from portkey.

Problem:
The latest changes done in LangChainLLM (from LLamaIndex) expects a streaming parameter from llm to determine if stream mode is available in the LLM or not. PortkeyLLM and ChatPortkey does not expose any streaming parameter and this causes a LLM must support streaming error.

Solution :
Expose streaming parameter from PortkeyLLM and ChatPortkey from langchain llm.

Reference for the LLamaIndex LangChainLLM
https://github.com/run-llama/llama_index/blob/fc290bb7425e4d28a4cb05bfe09879367ea6ea0a/llama_index/llms/langchain.py#L111

Contact Details

No response

What happened?

A bug happened!
Not able to generate completions from prompts as mentioned in the docs.

Version

0.1.xx (Default)

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

Contact Details

[email protected]

What happened?

When using the Python SDK to send chat completion requests to OpenAI, fields like cache, trace_id, and metadata fields are not respected although they appear to be sent with the request.
Code to reproduce:

import portkey
from portkey import Config, LLMOptions, CacheType, RetrySettings
PORTKEY_API_KEY = "PORTKEY_API_KEY"


def init_portkey():
    # Portkey Config
    client = portkey
    provider = "openai"
    virtual_key = "rashwan-openai-3fc"
    trace_id = "rashwan_bot"

    # Model Settings
    temperature = 0.1

    # Construct LLM
    gpt_4_llm = LLMOptions(
        provider=provider,
        virtual_key=virtual_key,
        trace_id=trace_id,
        model="gpt-4-0613",
        temperature=temperature,
        metadata={"_user": "rashwan_bot", "test_field": "test_value"},
        cache=True,
        cache_status=CacheType.SIMPLE,
        retry=RetrySettings(attempts=3, on_status_codes=[500, 502, 503, 504]),
    )
    gpt_35_llm = LLMOptions(
        provider=provider,
        virtual_key=virtual_key,
        trace_id=trace_id,
        model="gpt-3.5-turbo",
        temperature=temperature,
        metadata={"_user": "rashwan_bot", "test_field": "test_value"},
        cache=True,
        cache_status=CacheType.SIMPLE,
        retry=RetrySettings(attempts=3, on_status_codes=[500, 502, 503, 504]),
    )

    client.config = Config(
        api_key=PORTKEY_API_KEY ,
        mode="fallback",
        llms=[gpt_4_llm, gpt_35_llm],
    )

    response = client.ChatCompletions.create(
        messages=[
            {"role": "system", "content": "you are a helpful assistant"},
            {"role": "user", "content": "hello"},
        ]
    )
    print(response.choices[0].message["content"])

    return client

_ = init_portkey()

From the PortKey console here is the JSON request:

{
  "url": "https://api.openai.com/v1/chat/completions",
  "method": "POST",
  "headers": {
    "accept": "application/json",
    "accept-encoding": "gzip",
    "cf-connecting-ip": "130.211.80.105",
    "cf-ipcountry": "BE",
    "cf-ray": "817f9eefdcbc1541",
    "cf-visitor": "{\"scheme\":\"https\"}",
    "connection": "Keep-Alive",
    "content-length": "906",
    "content-type": "application/json",
    "host": "api.portkey.ai",
    "user-agent": "python-httpx/0.25.0",
    "x-forwarded-proto": "https",
    "x-real-ip": "130.211.80.105"
  },
  "body": {
    "config": {
      "mode": "fallback",
      "options": [
        {
          "provider": "openai",
          "virtual_key": "rashwan-openai-3fcd50",
          "cache": true,
          "cache_status": "simple",
          "trace_id": "rashwan_bot",
          "metadata": {
            "_user": "rashwan_bot",
            "test_field": "test_value"
          },
          "retry": {
            "attempts": 3,
            "on_status_codes": [
              500,
              502,
              503,
              504
            ]
          },
          "override_params": {
            "model": "gpt-4-0613",
            "temperature": 0.1
          },
          "api_key": "sk********QYq"
        },
        {
          "provider": "openai",
          "virtual_key": "rashwan-openai-3fcd50",
          "cache": true,
          "cache_status": "simple",
          "trace_id": "rashwan_bot",
          "metadata": {
            "_user": "rashwan_bot",
            "test_field": "test_value"
          },
          "retry": {
            "attempts": 3,
            "on_status_codes": [
              500,
              502,
              503,
              504
            ]
          },
          "override_params": {
            "model": "gpt-3.5-turbo",
            "temperature": 0.1
          },
          "api_key": "sk********QYq"
        }
      ]
    },
    "params": {
      "messages": [
        {
          "role": "system",
          "content": "you are a helpful assistant"
        },
        {
          "role": "user",
          "content": "hello"
        }
      ],
      "stream": false
    }
  },
  "portkeyHeaders": {
    "x-portkey-api-key": "do********ow=",
    "x-portkey-package-version": "portkey-0.1.53",
    "x-portkey-runtime": "CPython",
    "x-portkey-runtime-version": "3.9.18"
  }
}

No meta data is registered:

Cache is disabled:

Version

0.1.xx (Default)

Relevant log output

{
  "status": 200,
  "headers": {
    "access-control-allow-origin": "*",
    "cache-control": "no-cache, must-revalidate",
    "cf-cache-status": "DYNAMIC",
    "cf-ray": "817f9ef1a22a1541-CDG",
    "connection": "keep-alive",
    "content-length": "421",
    "content-type": "application/json",
    "date": "Wed, 18 Oct 2023 09:05:41 GMT",
    "openai-model": "gpt-4-0613",
    "openai-organization": "aim-technologies",
    "openai-processing-ms": "1050",
    "openai-version": "2020-10-01",
    "server": "cloudflare",
    "strict-transport-security": "max-age=15724800; includeSubDomains",
    "x-portkey-last-used-option-index": "0",
    "x-ratelimit-limit-requests": "10000",
    "x-ratelimit-limit-tokens": "150000",
    "x-ratelimit-remaining-requests": "9999",
    "x-ratelimit-remaining-tokens": "149973",
    "x-ratelimit-reset-requests": "6ms",
    "x-ratelimit-reset-tokens": "10ms",
    "x-request-id": "94e2dcbcba1c35aba26953342258404c"
  },
  "body": {
    "id": "chatcmpl-8AwmiB1oHP0vkq9ePKYUx2O2zED2K",
    "object": "chat.completion",
    "created": 1697619940,
    "model": "gpt-4-0613",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "Hello! How can I assist you today?"
        },
        "finish_reason": "stop"
      }
    ],
    "usage": {
      "prompt_tokens": 17,
      "completion_tokens": 9,
      "total_tokens": 26
    }
  },
  "responseTime": 1421
}

Code of Conduct

• I agree to follow this project's Code of Conduct

Test cases use type(key) is str at many places in test cases. This gives errors during build. Change this to isinstance(key, str) to fix build errors

Prompt method needs changes for the response the user gets

Contact Details

[email protected]

What happened?

I had this Module Not Found Error, while trying to work with Port-key-AI integrated with llamaindex.
If someone had the same issue, please make a sign.

I have the latest version of llamaindex (Version: 0.10.0) as well as Portkey-ai(Version: 1.1.7).

Here's the code

# Setup a custom service context by passing in the Portkey LLM
from llama_index.core import ServiceContext
from portkey_ai.llms.llama_index import PortkeyLLM

portkey = PortkeyLLM(api_key="PORTKEY_API_KEY", virtual_key="VIRTUAL_KEY")
service_context = ServiceContext.from_defaults(llm=portkey)

it's from the official documentation of Portkey

Version

0.1.xx (Default)

Relevant log output

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
File [c:\Users\saad\OneDrive\Bureau\LlamaChat-RAG-S2B\.conda\lib\site-packages\portkey_ai\llms\llama_index\utils.py:20](file:///C:/Users/saad/OneDrive/Bureau/LlamaChat-RAG-S2B/.conda/lib/site-packages/portkey_ai/llms/llama_index/utils.py:20), in all_available_models()
     [19](file:///C:/Users/saad/OneDrive/Bureau/LlamaChat-RAG-S2B/.conda/lib/site-packages/portkey_ai/llms/llama_index/utils.py:19) try:
---> [20](file:///C:/Users/saad/OneDrive/Bureau/LlamaChat-RAG-S2B/.conda/lib/site-packages/portkey_ai/llms/llama_index/utils.py:20)     from llama_index.llms.anthropic_utils import CLAUDE_MODELS
     [21](file:///C:/Users/saad/OneDrive/Bureau/LlamaChat-RAG-S2B/.conda/lib/site-packages/portkey_ai/llms/llama_index/utils.py:21)     from llama_index.llms.openai_utils import (
     [22](file:///C:/Users/saad/OneDrive/Bureau/LlamaChat-RAG-S2B/.conda/lib/site-packages/portkey_ai/llms/llama_index/utils.py:22)         AZURE_TURBO_MODELS,
     [23](file:///C:/Users/saad/OneDrive/Bureau/LlamaChat-RAG-S2B/.conda/lib/site-packages/portkey_ai/llms/llama_index/utils.py:23)         GPT3_5_MODELS,
   (...)
     [26](file:///C:/Users/saad/OneDrive/Bureau/LlamaChat-RAG-S2B/.conda/lib/site-packages/portkey_ai/llms/llama_index/utils.py:26)         TURBO_MODELS,
     [27](file:///C:/Users/saad/OneDrive/Bureau/LlamaChat-RAG-S2B/.conda/lib/site-packages/portkey_ai/llms/llama_index/utils.py:27)     )

ModuleNotFoundError: No module named 'llama_index.llms.anthropic_utils'

The above exception was the direct cause of the following exception:

Exception                                 Traceback (most recent call last)
Cell In[18], [line 3](vscode-notebook-cell:?execution_count=18&line=3)
      [1](vscode-notebook-cell:?execution_count=18&line=1) # Setup a custom service context by passing in the Portkey LLM
      [2](vscode-notebook-cell:?execution_count=18&line=2) from llama_index.core import ServiceContext
----> [3](vscode-notebook-cell:?execution_count=18&line=3) from portkey_ai.llms.llama_index import PortkeyLLM
      [5](vscode-notebook-cell:?execution_count=18&line=5) portkey = PortkeyLLM(api_key="PORTKEY_API_KEY", virtual_key="VIRTUAL_KEY")
      [6](vscode-notebook-cell:?execution_count=18&line=6) service_context = ServiceContext.from_defaults(llm=portkey)

File [c:\Users\saad\OneDrive\Bureau\LlamaChat-RAG-S2B\.conda\lib\site-packages\portkey_ai\llms\llama_index\__init__.py:1](file:///C:/Users/saad/OneDrive/Bureau/LlamaChat-RAG-S2B/.conda/lib/site-packages/portkey_ai/llms/llama_index/__init__.py:1)
...
     [36](file:///C:/Users/saad/OneDrive/Bureau/LlamaChat-RAG-S2B/.conda/lib/site-packages/portkey_ai/llms/llama_index/utils.py:36)     }
     [37](file:///C:/Users/saad/OneDrive/Bureau/LlamaChat-RAG-S2B/.conda/lib/site-packages/portkey_ai/llms/llama_index/utils.py:37) except ImportError as exc:
---> [38](file:///C:/Users/saad/OneDrive/Bureau/LlamaChat-RAG-S2B/.conda/lib/site-packages/portkey_ai/llms/llama_index/utils.py:38)     raise Exception(IMPORT_ERROR_MESSAGE) from exc

Exception: Llama-Index is not installed.Please install it with `pip install llama-index`.

Code of Conduct

• I agree to follow this project's Code of Conduct

Update typing dependencies version constraints from strict ==4.7.1 to >=4.7.1,<5.0 to resolve dependency conflict with package like chroma

Related bugs: #66

Currently, create method for embeddings does not accept extra params in it. This might cause temporary errors if the user is trying to use some new params released by provider but not updated in SDK.

Solution:

• Add all the known params to the create method: input, model, dimensions, encoding_format and user.
• Also allow **kwargs so that it does not break in future when user tries to use some new embeddings param.

Contact Details

[email protected]

What happened?

This SDK's dependency on an exact version of pydantic (pydantic==1.10.12) is causing dependency conflicts with some other packages, in my case django-ninja (depends on pydantic<3.0.0 and >=2.0)

If it's all possible to loosen the package's range of acceptable pydantic versions, I think it may be worth doing so to allow developers to use this SDK alongside a wider range of other packages.

Version

0.1.xx (Default)

Relevant log output

portkey-ai 0.1.27 depends on pydantic==1.10.12
django-ninja 1.0.1 depends on pydantic<3.0.0 and >=2.0

Code of Conduct

• I agree to follow this project's Code of Conduct

Hey,

I'm looking to updated our the portkey integration within LlamaIndex lib. Specifically looking to use v1 openai interface for function calling with portkey.

• When I try to use regular chat (which invokes portkey.ChatCompletions.create) I can pass kwargs tools and tool_choice but what I get back is a response that doesn't appear to use the tools at all
• When I try to use stream mode, I get thrown a ValidationError because of the added tools and tool_choice args.

Reference:
https://platform.openai.com/docs/api-reference/chat/create

Version upgrade - 0.1.53

Contact Details

No response

What happened?

I tried to prompt to the mistralai/Mistral-7B-Instruct-v0.2 model by Anyscale virtual key(Without adding the model in anyscale account). It shows the above error.

So how to use the opensource models directly from the huggingface or the fientuned models from my local folder?

Version

0.1.xx (Default)

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

Description:

As of now, our codebase lacks comprehensive documentation in the form of docstrings for functions. This hinders the understanding of how individual functions work, their purpose, and the expected inputs and outputs. Proper documentation is essential for maintainability and collaboration.

Tasks:

• Review the codebase to identify functions without docstrings.
• Add docstrings to each function, following a consistent format and style.
• Ensure that the docstrings are clear, concise, and informative.
• Update the project's documentation to emphasize the importance of maintaining docstrings for all new functions.

Save the Azure deployment details along with the virtual api keys

Summary

Asynchronous programming has become increasingly essential for handling real-time communication and streams efficiently. To meet the evolving needs of our users, we recognize the importance of extending the capabilities of our Python SDK by adding support for asynchronous chat and streams.

By adding support for asynchronous chat and streams we will enable users to perform real-time interactions and efficiently process streams while maintaining a non-blocking, responsive application.

Note: This feature's priority will depend on user demand. If you have a strong interest in seeing this feature implemented, we encourage you to voice your support by leaving a comment on this ticket. Your feedback will play a significant role in determining the feature's prioritization.

Hotfix: add openAI in install_requires

In the realm of LLM application development, especially when dealing with your data the ability to observe, troubleshoot, and assess your system is important. Whether you're analyzing the entire system or individual components, this functionality is essential.

With this feature, you can seamlessly fuse the Portkey library with robust observability. Set up a single configuration to setup observability and other features provided by portkey. This makes it easier to incorporate portkey into the existing systems without having to change the implementation.

We have introduced a significant enhancement to the Rubeus APIs by adding support for the use of config Slug in requests instead of the config body. This improvement allows users to effortlessly utilize a configId created via this method, simplifying the process of writing and reusing configurations.

To learn more about how to create a config using this new feature, please refer to our documentation: Creating Configurations with Config Slug.

Building our SDK in a way that it is a wrapper around OpenAI SDK.

Instead of calling our Gateway, making a call to OpenAI SDK with all the needed changes

Mentioned in discord:
https://discord.com/channels/1143393887742861333/1144592558132035655/1227709106832281772

To update Python SDK README

With the support of asynchronous calls in the SDK; we need to add a code snippet on how to use it as well.

The GenericResponse class has some properties which are Optional but they do not have a default value.

This works well for pydantic v1 as Optional make a property not-required. But this was changed in pydantic v2 where Optional now means that the property is required but it can have a None value. In pydantic v2, the code has to define a default value along with Optional to make it not-required.

Currently, whenever a project has pydantic v2 installed, the prompts.completions.create() fails because the response might not have the Optional value shown above. Error shown in the attached image

Reference for pydantic v2 which mentions the behaviour change of Optional (Also attached the image for the same): https://docs.pydantic.dev/latest/migration/#required-optional-and-nullable-fields

Solution: Add a default None value to all 3 Optional fields in GenericResponse class so that SDK does not fail even when the fields are not present in the response.

linked to #32

Contact Details

No response

What happened?

My application uses mypy = ">=1.2.0". The Portkey SDK setup requires mypy==0.991.

Version

0.1.xx (Default)

Relevant log output

Because portkey-ai (1.1.4) depends on mypy (0.991)
 and no versions of portkey-ai match >1.1.4,<2.0.0, portkey-ai (>=1.1.4,<2.0.0) requires mypy (0.991).
So, because <my-app> depends on both portkey-ai (^1.1.4) and mypy (>=1.2.0), version solving failed.

Code of Conduct

• I agree to follow this project's Code of Conduct

Problem statement:
When the SDK is used in a project which has pydantic v1 installed, chat completions create (and other methods as well) throws this error:

"ChatCompletions" object has no field "_headers"

Steps to reproduce:

• Install portkey SDK
• Install pydantic v1.10.12
• Make a chat completions call.

Solution:

• Use PrivateAttr from pydantic for all the _headers attribute in existing class

Cache Refresh was not working as it is supposed to

Summary

Currently the implementation of the portkey SDK is to create a config and pass the model params on the create methods. This has been useful in most cases but in somecases users want to be able to set the portkey params on the create method over-riding the config created globally.

Source: From the discord conversation with Ragas team.

Solution

Accepting the portkey params on the create methods of the SDK

To improve the developer experience, need to provide with the auto complete suggestions

Description

The local setup must be updated to the following code to be able to use it locally.

pip install -e .

Description

The current generate API doesn't accept any additional params that can be sent to portkey to over-ride the saved parameters like model, temperature.

Description:

Maintaining code quality and adherence to coding standards is essential for the long-term stability and maintainability of our project. While we have some development checks in place, enhancing our Continuous Integration (CI) pipeline with linting and code quality checks can help ensure that our codebase remains clean and consistent. These checks will analyze our codebase for potential issues, such as coding style violations and common errors, helping us catch and address them early in the development process.

Proposed Tasks:

• Integrate a linting tool (Pylint) into our CI pipeline.
• Configure linting rules to match our coding standards.
• Set up code quality checks (e.g., code complexity analysis) as needed.
• Ensure that the CI pipeline fails on linting errors.
• Document linting and code quality procedures for the development team.

Code quality is an ongoing effort; new code submissions should adhere to coding standards.
We welcome community contributions and input on linting rules.

Note: This issue tracks the addition of linting and code quality checks to our CI pipeline. Further discussions and contributions will happen in follow-up discussions and pull requests.

Your involvement in this issue, whether through code contributions, linting rule suggestions, or feedback, is vital as we work to improve code quality in our project.

Description

The current documentation for our SDK is comprehensive, but we've identified a need to improve the user experience and provide more practical guidance. One area we believe could benefit greatly is the addition of a cookbook section with a variety of examples.

The Cookbook section must include:

• Step-by-Step Tutorials: Detailed tutorials with step-by-step instructions for common tasks and workflows.
• Code Samples: Well-commented code snippets demonstrating specific functionalities.
• Best Practices: Guidance on best practices and recommended approaches.
• Integration Scenarios: Real-world integration examples with other tools or technologies.
• Use Cases: Examples showcasing how our project can solve specific problems or meet particular needs.

Note : Those who are interested in contributing can engage with the project maintainers through our Discord channel for more details.