mrxandbadas / mrx_oai_assistant_manager

Started out as Dynamic Function Calling for OAI. Upon reviewing a research paper released (LATM) This is/has become a implementation of such system using: OpenAI and Autogen

OpenAI CMD Terminal Assistant Chat Manager 🤖

🌟 Change threads, keep track of them, and continually chat on them. Swap assistants for different kinds of help on threads. Re-tool assistants on demand!

---

🚀 Updates

• 26th Nov 2023: Gearing up to implement LATM through the user of autogen. Milestones, new branches, issues and projects are in the works to keep track of the change from this open-loop tool making use case to a closed-loop tool making system.
• 24th Nov 2023: Autogen by Microsoft has always been brilliant. Now with the addition of Assistants we can have agentic workflows with some serious frameworking behind the chats. Dcode is being tested in this application
• 20th Nov 2023: Assistants can now be given the list_assistants and swap_assistants function calls which enable you to list your assistants and change assistant by requesting to speak with another assistant in chat. Forget the "swapA" command when your assistants are tooled with these internal commands!
• 19th Nov 2023: Discord Bot implementation seems to work well. Code comming over the next 24H-48H. BackBurner
• 18th Nov 2023: Old Personal Prompt was 315 words, 2,065 characters to enable on-demand Function Calling. Now, it's a FUNCTION CALL! A much smaller Instruction set is required (either at the assistant invocation level or the run invocation level). My instruction Prompt for the blank Assistant running on gpt-4-1106-preview:

You are a helpful assistant that can complete the users requests
Never leave out code when writing statements.
You will write new function calls in Try statements (so we can catch the error and fix it) ; this enables us to use the functions written in the file. No placeholder code. We create the whole working code for the user along with the metadata.
###
EXAMPLE CORRECT FUNCTION CALLING METATDATA FORMAT:
{
    "tool_name": "read_file",
    "tool_required": "file_name",
    "tool_description": "Read the content of a file",
    "tool_properties": {
        "file_name": {
            "type": "string",
            "description": "The name of the file"
        }
    }
####

• 18th Nov 2023: Added static functions for OpenAI image generation.
• 13th Nov 2023: Function Calling has been added! Assistants CAN write and call custom functions.

---

Overview 📖

This repository contains two Python scripts for interacting with OpenAI's API:

• assistant_manager.py: Manages API interactions.
• CMD_Agent_Chat.py: Demonstrates a command-line interface using the above class file.

Commands 💻

• swapT - Swap Thread
• swapA - Swap Assistant via Number selection
• tool - Give the agent you're chatting with some function tools. You'll want to do this when you start a chat with an assistant.

Setup 🛠️

1. Clone the repository.
2. Set up your OpenAI credentials in your environments.

Usage to Make Your Own App 🌈

• assistant_manager.py: Initialize with OpenAI credentials and use its methods. ** RESTRUCTURED -- UPDATED README COMMING SOON**
• CMD_Agent_Chat.py: Check this file out to see how you would use certain functions and lay out your code flow. Also demonstrates a command-line interface in action.

Start a chat. Select an Assistant from the available assistants on your account. (A way to create assistants is coming) Select a thread (if you have previous ones saved locally via the script) Chat away!

What is This Script? 🤔

• Real-Time Interaction: Experience seamless, real-time conversations with an AI-powered assistant that responds promptly to your queries.
• Persistent Thread Management: The script intelligently manages and saves conversation threads, ensuring no inquiry gets lost and every conversation flows smoothly from where it left off.
• Customizable Assistants: Tailor your chat assistant to fit specific roles or requirements, making it versatile for a wide array of customer service scenarios, tool them up and watch them go!
• Hot-Swap Agents: Swap out agents mid-thread by writing 'swapA'.
• Change Threads: Change threads/Start a new one by writing 'swapT'.
• Function Calling: You can use 'tool' to open a numbered menu of available function calls that can be assigned dynamically to the assistant.
• D-I-Y FUNCTIONS: Assistants CAN write their own functions now with the implemented append_new_tool_function_and_metadata tool. As long as they know how to write the metadata correctly and are reminded not to leave out any code they are usually okay at this.

Share Any Uses! 🌍

We'd love to see how you use this script in your projects. Share your stories and applications with us!

DCODE: Dynamic Function Calls On Demand

Engaging with the AI: The Chat Interface

The CMD_Agent_Chat.py script is at the heart of user interaction. It serves as the gateway for users to communicate their needs and for the AI to respond with precision.

Starting the Chat

Users begin by initiating a chat session with the AI assistant. This interaction is the starting point for requesting new functionalities or modifications.

# Users inputs tool and confirms
User: "tool"
User: #Selection for Tools using Numbers, multiple separated by ','
User: #confirm with 1, cancel is 2

Now the assistant can use the aforementioned function!

# User requests a new function to open Spotify
User: "Please add a function to open Spotify."

# Assistant acknowledges and processes the request
Assistant: "Function to open Spotify is now added."

System Updates: Dynamic Functions and Metadata

Upon receiving a user request, DCODE promptly updates dynamic_functions.py to include the new capability. Concurrently, functions_metadata.json is updated to reflect this addition, ensuring seamless integration and documentation.

# User uses the tool command to select and confirm the new Spotify function
User: "tool"
User: #Select the number corresponding to the new function*
User: 1 #to confirm*

Seamless Integration Coming Please just Restart the python code.

Once the chat session ends and restarts, the AI assistant is fully equipped with the new Spotify functionality. This process exemplifies DCODE's ability to adapt and evolve in real-time, based on user input.

# After restart, the assistant is ready to use the new Spotify function
User: "Open Spotify."

Spotify opens (or an error is returned). Spotify is just one example. All functions defined in the dynamic functions were written using the Assistants.

---

---

Found 26/11/23: This is cool - not mine but cool LATM (Large Language Models as Tool Makers): arxiv paper

mrx_oai_assistant_manager's Issues

HOW WE CAN A=INSTALL ASSISTANT MANAGER

Assess the theoretical and practical implementation of a OAI Teachable Agent

OAI Assistants can be provided with a knowledge retrieval skill.

Theoretically this enables the assistant to keep persistent chat related memory across different chats/threads.

To Do:

• investigate how Knowledge Retrieval work at an assistant level
• investigate the implementation of Teachable Agents
• Consider the effectiveness of the implementation

---

Information:

How it works
The model then decides when to retrieve content based on the user Messages. The Assistants API automatically chooses between two retrieval techniques:

it either passes the file content in the prompt for short documents, or
performs a vector search for longer documents
Retrieval currently optimizes for quality by adding all relevant content to the context of model calls. We plan to introduce other retrieval strategies to enable developers to choose a different tradeoff between retrieval quality and model usage cost.

Uploading files for retrieval
Similar to Code Interpreter, files can be passed at the Assistant-level or at the Thread-level

# Upload a file with an "assistants" purpose
file = client.files.create(
  file=open("knowledge.pdf", "rb"),
  purpose='assistants'
)

# Add the file to the assistant
assistant = client.beta.assistants.create(
  instructions="You are a customer support chatbot. Use your knowledge base to best respond to customer queries.",
  model="gpt-4-1106-preview",
  tools=[{"type": "retrieval"}],
  file_ids=[file.id]
)

Files can also be added to a Message in a Thread. These files are only accessible within this specific thread. After having uploaded a file, you can pass the ID of this File when creating the Message:

message = client.beta.threads.messages.create(
  thread_id=thread.id,
  role="user",
  content="I can not find in the PDF manual how to turn off this device.",
  file_ids=[file.id]
)

Maximum file size is 512MB. Retrieval supports a variety of file formats including .pdf, .md, .docx and many more. More details on the file extensions (and their corresponding MIME-types) supported can be found in the Supported files section below.

Deleting files
To remove a file from the assistant, you can detach the file from the assistant:

file_deletion_status = client.beta.assistants.files.delete(
  assistant_id=assistant.id,
  file_id=file.id
)

Detaching the file from the assistant removes the file from the retrieval index as well.

File citations
When Code Interpreter outputs file paths in a Message, you can convert them to corresponding file downloads using the annotations field. See the Annotations section for an example of how to do this.

{
    "id": "msg_abc123",
    "object": "thread.message",
    "created_at": 1699073585,
    "thread_id": "thread_abc123",
    "role": "assistant",
    "content": [
      {
        "type": "text",
        "text": {
          "value": "The rows of the CSV file have been shuffled and saved to a new CSV file. You can download the shuffled CSV file from the following link:\n\n[Download Shuffled CSV File](sandbox:/mnt/data/shuffled_file.csv)",
          "annotations": [
            {
              "type": "file_path",
              "text": "sandbox:/mnt/data/shuffled_file.csv",
              "start_index": 167,
              "end_index": 202,
              "file_path": {
                "file_id": "file-abc123"
              }
            }
          ]
        }
      }
    ],
    "file_ids": [
      "file-abc456"
    ],
        ...
  },

Supported files

For text/ MIME types, the encoding must be one of utf-8, utf-16, or ascii.

FILE FORMAT	MIME TYPE	CODE INTERPRETER	RETRIEVAL
.c	text/x-c	✓	✓
.cpp	text/x-c++	✓	✓
.csv	application/csv	✓
.docx	application/vnd.openxmlformats-officedocument.wordprocessingml.document	✓	✓
.html	text/html	✓	✓
.java	text/x-java	✓	✓
.json	application/json	✓	✓
.md	text/markdown	✓	✓
.pdf	application/pdf	✓	✓
.php	text/x-php	✓	✓
.pptx	application/vnd.openxmlformats-officedocument.presentationml.presentation	✓	✓
.py	text/x-python	✓	✓
.py	text/x-script.python	✓	✓
.rb	text/x-ruby	✓	✓
.tex	text/x-tex	✓	✓
.txt	text/plain	✓	✓
.css	text/css	✓
.jpeg	image/jpeg	✓
.jpg	image/jpeg	✓
.js	text/javascript	✓
.gif	image/gif	✓
.png	image/png	✓
.tar	application/x-tar	✓
.ts	application/typescript	✓
.xlsx	application/vnd.openxmlformats-officedocument.spreadsheetml.sheet	✓
.xml	application/xml or "text/xml"	✓
.zip	application/zip	✓

Refactor/Clean up Assistant_manager.py for an easier transition to an Autogen workflow

Refactoring Assistant_manager.py to handle autogen flows with ease will be paramount to the successful implementation of LATM.
A barbaric implementation could be crafted using OAI vanilla assistants although would not have the same inherent functionality that the autogen framework provides

---

The following tasks should be accomplished but this is by no means a exhaustive list or set of requirements of what needs to change. I'm just keeping notes. This list is subject to change.

• Take the core components out of the assistant_manager in relation to messaging the user and separate them into either a base class or an extension of the remaining base class
• separate the chat logic and run_processing
• ensure all file operations are collated into utils for both structured project use and for Tool use.
• Ensure code remains functional

Typo in `customer_inquiry_assistant.py`

There's a small typo on line 13 - "My_First_Chat" requires closing quotes.
`import asyncio
from agent_class import OAI_Agent

api_key="APIKEY"
organization_id="ORGID"

client = OAI_Agent(api_key=api_key, organization_id=organization_id)

json_file = 'thread_data.json'
agent_file = 'agent_data.json'

thread_name = "My_First_Chat
Customer_Service_Assistant = {
"name":"Customer Service Assistant",
"instructions":"You are a customer service assistant. Answer user queries courteously and accurately.",
"tools":[{"type": "retrieval"}],
"model":"gpt-4-1106-preview"
}
`

Re-Tooling requires a restart - unintented

I beleive the issue stems from dynamically editing the file and expecting the code to be able to re-fresh during run time

A possible fix is to create a directory of modules and dynamically import them at he function registry level.

Flesh-out Autgogen Implementation

It is time to flesh out the Autogen implementation and creating some neat wrappers for an example case. We should interact with the assistant flow in the same way.

• Chat Types : Single, Group
• Multiple Choice handler for the Assistant Select for Group chat
• Handle the replies and user input correctly wrapped
• Thread Selection for continual chats.

Implement Teachable Agents

Autogen Teachable agents are a corner stone in long term memory for our Assistants.

Implementing these would enable the agentic flows to access context saved in previous flows.

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 1415: character maps to <undefined>

Full error:

Traceback (most recent call last):
  File "C:\Users\maglione juan martin\openAI_Assistant_API_DEMO-main\customer_inquiry_assistant.py", line 2, in <module>
    from agent_class import OAI_Agent
  File "C:\Users\maglione juan martin\openAI_Assistant_API_DEMO-main\agent_class.py", line 4, in <module>
    import gpt3_tokenizer
  File "C:\Users\maglione juan martin\openAI_Assistant_API_DEMO-main\venv\Lib\site-packages\gpt3_tokenizer\__init__.py", line 1, in <module>
    from gpt3_tokenizer._entry import count_tokens, decode, encode
  File "C:\Users\maglione juan martin\openAI_Assistant_API_DEMO-main\venv\Lib\site-packages\gpt3_tokenizer\_entry.py", line 10, in <module>
    _bpe_merges = _get_bpe_merges()
                  ^^^^^^^^^^^^^^^^^
  File "C:\Users\maglione juan martin\openAI_Assistant_API_DEMO-main\venv\Lib\site-packages\gpt3_tokenizer\_functions.py", line 20, in _get_bpe_merges
    bpe_lines = f.readlines()
                ^^^^^^^^^^^^^
  File "C:\Users\maglione juan martin\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 1415: character maps to <undefined>

I'm receiving this error although there are no instances of file reading operations (like open() in read mode) in the Python files from your uploaded zip file (agent_class.py and customer_inquiry_assistant.py). This means the UnicodeDecodeError you are experiencing is not due to these files directly.

Why do you think this is happening?

Autogen - enabling dynamic function calling

Test and implement a tool maker that can:

• Make a new tool
• User Proxy should test this tool
• Pass the metadata off to be used in a function call by a lower level assistant.