agenta-ai / agenta Goto Github PK

Describe the bug
Error when adding a column to a csv file uploded in Test Sets, after modifying the rows values.

To Reproduce
Steps to reproduce the behavior:

1. Go to 'Test Sets'
2. Click on 'Upload a test set'
3. Upload a test set
4. Click on the test set to modify it
5. Modify rows values
6. Add a column and rename it
7. Click on 'Update Column names'

Describe the bug
The documentation page for contributing contains a link to local installation guide, which contains a link back to contributing .

Expected behavior
The documentation should contain a guide to install the repo in an editable fashion, most likely:

pip install -e .

Screenshots

Description
Currently you can create test sets in the UI with empty names. They will be saved but cannot be used

Solution
Do not allow the creation of a test set with an empty name. If the name of the set is empty, show a modal letting the user know that they first need to give the test set a name

We have [soon] elements in many parts of the UI. @aakrem I propose we remove them, since we don't plan to implement them very soon.

How to do it?

Just find the components in the agenta-web codebase (Second screenshot should be in sidebar.tsx), and remove the components.

Is your feature request related to a problem? Please describe.
When you run an auto-evaluation (exact match) and then go to results, you cannot see which variant resulted in that evaluation

Describe the solution you'd like
The UI should be updated to show the variant that has been evaluated

Currently, our SDK for the LLM app only supports parameters of type TextParam (text) and FloatParam (float). We propose to expand this functionality by introducing a new parameter type, MultipleChoiceParam.

The MultipleChoiceParam will allow users to specify a discrete set of choices. For example, users could choose between different versions of GPT models to use. To utilize MultipleChoiceParam, the user would use the SDK as follows:

@post
def query(
    transcript: str,
    model: MultipleChoiceParam = MultipleChoiceParam(["gpt-3.5-turbo", "gpt-4"]),
    prompt: TextParam = DEFAULT_PROMPT,
) -> str:

In this context, the first value of the MultipleChoiceParam list would serve as the default selection.

Inclusion of the MultipleChoiceParam will result in the openapi.json file including an x-parameter, which would contain the list of potential choices. It's important to note that the frontend interprets the parameters from the LLM app via these x-parameters.

The relevant code snippet that parses these parameters is as follows:

export const parseOpenApiSchema = (schema: any): Parameter[] => {
    const parameters: Parameter[] = []

    // check if requestBody exists
    const requestBody = schema?.paths?.["/generate"]?.post?.requestBody
    if (requestBody) {
        const bodySchemaName = requestBody.content["application/json"].schema["$ref"]
            .split("/")
            .pop()

        // get the actual schema for the body parameters
        const bodySchema = schema.components.schemas[bodySchemaName].properties

        Object.entries(bodySchema).forEach(([name, param]: [string, any]) => {
            parameters.push({
                name: name,
                input: param["x-parameter"] ? false : true,
                type: param["x-parameter"] ? determineType(param["x-parameter"]) : "string",
                required: schema.components.schemas[bodySchemaName].required.includes(name),
                default: param.default,
            })
        })
    }

    return parameters
}

Introducing this parameter will include a number of tasks:

• Adding the parameter to the SDK
• Adding the parameter to the Frontend (playground)
• Writing examples (and updating examples) with this parameter
• Updating the documentation to add this parameter

Describe the bug
If you try to create an app name with _ in the CLI you get an error.

Describe the bug
Go to test set, click create test set with an API. You will not find the code snippets that are expected to be there. See the screenshot

Describe the bug
Endpoints view does not work for variants created using the UI

To Reproduce
Steps to reproduce the behavior:

1. Fork a new variant using the UI
2. Go to endpoints
3. Select the variant
4. You get an error "failed to load variants"

Create a transcript analysis application with one variant. The goal of the application is to ask questions to a transcript, the model would provide an answer together with the quote from the transcript.

For instance, take this transcript:

Salesperson: Good morning! My name is Alex, and I'm from XYZ Solutions. How can I assist you today?
Prospect: Hi Alex, I'm interested in upgrading our current software system. We need a solution that can handle higher volumes and provide better reporting capabilities.

Salesperson: Absolutely! We have the perfect software suite that meets your requirements. It's our latest version, specifically designed for businesses like yours. It can handle large volumes of data and offers advanced reporting features. Would you like me to walk you through the key features?
Prospect: Yes, please. I want to understand how it can benefit our team and streamline our operations.

Salesperson: Great! With our software, you'll experience improved data processing, reducing the time it takes to generate reports by up to 50%. Additionally, it offers customizable dashboards, so you can monitor key metrics in real-time. The system also integrates seamlessly with your existing tools, eliminating any disruption during implementation.
Prospect: That sounds impressive. How about the cost of the upgrade? We have a budget to consider.

Salesperson: Of course. The pricing for the software depends on the number of user licenses and the level of customization required. I can provide you with a detailed quote tailored to your specific needs. Additionally, we offer flexible payment options to accommodate your budget.
Prospect: That's good to know. Can you also share some success stories or case studies from other clients who upgraded to this version?

Here is how the app work

Question: Did the the prospect ask about the cost of the upgrade?
LLM app answer: Yes they did, here is the quote where they do that "That sounds impressive. How about the cost of the upgrade? We have a budget to consider."

App specifications

The application takes as input:

• The transcript as a string
  The parameters will be:
• temperature
• chunk size (in char)
  The application returns:
• A string containing the LLM answer as shown above

Algorithm

Similarly to the previous example, the transcript will be chunked with the size. Each chunk will be embedded and saved in the db. The question will also be saved in an embedding. The embedding will be used to search for the relevant chunks and the answer returned. See the app sales_call_analyzer for a very similar example using llama_index

Right now the sidebar synchronously scrolls with the page content, leading to a less-than-ideal user experience.

We aim to rectify this by decoupling the scroll functionality of the sidebar from the page content. With this fix, users can enjoy a seamless and uninterruptive navigation experience, where the sidebar remains static while they scroll through the page content.

Description
We would like to load the test set data with one button click in the playground. This would allow users to quickly get started with the playground.

Describe the solution you'd like
Add a button to the right of 2. Preview and test (in the playground) that allows to load a test set. Clicking on the button would show a modal where the user can select the test set between multiple choices. After confirming, the data set will be appended to the already existing rows.

Additional context
Add any other context or screenshots about the feature request here.

We've noticed some discrepancies in the terminologies used across our frontend and backend codebases.

These inconsistencies could potentially lead to confusion and miscommunication.

As such, we aim to standardise these terms to maintain consistency, enhance code readability, and streamline collaboration across teams.

We would like to start with renaming:

• ComparisonTable, comparisonTable, comparison_table .. To evaluation
• AppEvaluation, app_evaluation appEvaluation .. To evaluation

We also need a migration script to the new schema.

Is your feature request related to a problem? Please describe.
Currently if you run agenta variant serve and the server is down, you get a long obfuscated error

To reproduce
Shut down the docker-compose then run agenta serve. You will get this error

(agenta-py3.9) (base) mahmoudmabrouk@MacBook-Pro-2 baby_name_generator % agenta variant serve
Traceback (most recent call last):
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/urllib3/connection.py", line 200, in _new_conn
    sock = connection.create_connection(
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/urllib3/connection.py", line 388, in request
    self.endheaders()
  File "/Users/mahmoudmabrouk/opt/anaconda3/lib/python3.9/http/client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/mahmoudmabrouk/opt/anaconda3/lib/python3.9/http/client.py", line 1034, in _send_output
    self.send(msg)
  File "/Users/mahmoudmabrouk/opt/anaconda3/lib/python3.9/http/client.py", line 974, in send
    self.connect()
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/urllib3/connection.py", line 236, in connect
    self.sock = self._new_conn()
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/urllib3/connection.py", line 215, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fed2071d250>: Failed to establish a new connection: [Errno 61] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /api/app_variant/list_variants/?app_name=babyname (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fed2071d250>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/bin/agenta", line 6, in <module>
    sys.exit(cli())
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/mahmoudmabrouk/agenta/code/agenta-core/agenta-cli/agenta/cli/variant_commands.py", line 242, in serve_cli
    config_check(app_folder)
  File "/Users/mahmoudmabrouk/agenta/code/agenta-core/agenta-cli/agenta/cli/variant_commands.py", line 212, in config_check
    helper.update_config_from_backend(config_file, host=host)
  File "/Users/mahmoudmabrouk/agenta/code/agenta-core/agenta-cli/agenta/cli/helper.py", line 37, in update_config_from_backend
    config = update_variants_from_backend(app_name, config, host)
  File "/Users/mahmoudmabrouk/agenta/code/agenta-core/agenta-cli/agenta/cli/helper.py", line 21, in update_variants_from_backend
    variants: List[AppVariant] = client.list_variants(app_name, host)
  File "/Users/mahmoudmabrouk/agenta/code/agenta-core/agenta-cli/agenta/client/client.py", line 63, in list_variants
    response = requests.get(f"{host}/{BACKEND_URL_SUFFIX}/app_variant/list_variants/?app_name={app_name}", timeout=600)
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /api/app_variant/list_variants/?app_name=babyname (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fed2071d250>: Failed to establish a new connection: [Errno 61] Connection refused'))

Describe the solution you'd like
We need to provide better exception handling, so that when this happens the user gets a message explaining the issue (failed to connect to agenta backend) and some pointers for the solution (check that the backend is working, restart docker compose)

Right now when we right click on an item from the sidebar's menu we can't get "open in a new tab" or "open in a new window".

Describe the bug
To complete

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

We've noticed some discrepancies in the terminologies used across our frontend and backend codebases.

These inconsistencies could potentially lead to confusion and miscommunication.

As such, we aim to standardise these terms to maintain consistency, enhance code readability, and streamline collaboration across teams.

We would like to rename:

dataset .. To testset

We also need a migration script to the new schema.

Describe the bug
If you try agenta variant serve and it fails because the docker fails to start. The variant is nevertheless added to the UI.

To Reproduce
Steps to reproduce the behavior:

1. Create a buggy variant (i.e. missing requirements.txt)
2. Serve the variant
3. Go to the UI

Expected behavior
CLI fails and the variant are not added

We aim to ensure that a user doesn't lose the data they've entered into a new or existing testset.

To facilitate this, we intend to display a modal prompting the user to confirm whether they want to save the entered data or not.

If the user chooses to save, we'll store the data and redirect them to their intended destination

Describe the bug
There is a number of issues when running agenta variant serve and overwriting an existing variants:

• The image for the old image is not removed. As a result, the disk gets full with old images
• There might be issues when overwriting a variant with the same name as created from the playground

Additional context
The issue need more research to determine the exact problems when overwriting

Description
The usability of the playground is not very good. Currently, it is very hard for a user to test the same examples with different variants. To do that, the only way to copy paste the same inputs for each variant/tab in the playground. See an example in this video

Describe the solution you'd like
All the tabs/variants need to share the same context for the inputs, going from one tab to another would only change the outputs but not the inputs

For instance, in the screenshot above, [sdf,sdf], and [df,sdf] will be shared between all the tabs/variants in the playground view, while each variant would of course have different results for each input.

Additional context
The code to be changed is in components Playground

The hierarchy of the components is:
Playground -> holder component. redundant, can be removed
VersionTabs -> holder of the different tabs and the logic of adding and removing tabs/variants. The inputs props/context should be at this level
ViewNavigation -> holds the data for the variant (what are its parameters, where to call it) and the two parts of the view
-ParametersView -> Allows to modify the parameters of the variant. Needed when calling the variants in the second view

• TestView -> This is where the inputs/outputs currently lie.

In order to enhance user comfort, save power on screens, cater to aesthetic preferences, improve accessibility, and accommodate nighttime usage we would like to support dark mode!

Describe the bug

In the evaluation page we have the following warnings

Warning: Each child in a list should have a unique "key" prop.

Check the render method of `Cell`. See https://reactjs.org/link/warning-keys for more information.
span
Cell@webpack-internal:///./node_modules/rc-table/es/Cell/index.js:48:70
tr
BodyRow@webpack-internal:///./node_modules/rc-table/es/Body/BodyRow.js:28:70
ImmutableComponent@webpack-internal:///./node_modules/@rc-component/context/es/Immutable.js:72:5
tbody
Body@webpack-internal:///./node_modules/rc-table/es/Body/index.js:25:70
ImmutableComponent@webpack-internal:///./node_modules/@rc-component/context/es/Immutable.js:72:5
table
div
div
div
Provider@webpack-internal:///./node_modules/@rc-component/context/es/context.js:24:17
Table@webpack-internal:///./node_modules/rc-table/es/Table.js:154:58
ImmutableComponent@webpack-internal:///./node_modules/@rc-component/context/es/Immutable.js:40:61
div
div
Spin@webpack-internal:///./node_modules/antd/es/spin/index.js:78:9
SpinFC@webpack-internal:///./node_modules/antd/es/spin/index.js:136:7
div
InternalTable@webpack-internal:///./node_modules/antd/es/table/InternalTable.js:80:7
Table@webpack-internal:///./node_modules/antd/es/table/Table.js:16:61
div
EvaluationsList@webpack-internal:///./src/components/Evaluations/EvaluationsList.tsx:28:74
div
Evaluations@webpack-internal:///./src/components/Evaluations/Evaluations.tsx:32:74
Evaluation
div
main
Basic<@webpack-internal:///./node_modules/antd/es/layout/layout.js:60:9
Content
section
BasicLayout<@webpack-internal:///./node_modules/antd/es/layout/layout.js:76:42
Layout
section
BasicLayout<@webpack-internal:///./node_modules/antd/es/layout/layout.js:76:42
Layout
App@webpack-internal:///./src/components/Layout/Layout.tsx:25:25
App@webpack-internal:///./src/pages/_app.tsx:14:38
PathnameContextProviderAdapter@webpack-internal:///./node_modules/next/dist/shared/lib/router/adapters.js:74:44
ErrorBoundary@webpack-internal:///./node_modules/next/dist/compiled/@next/react-dev-overlay/dist/client.js:303:63
ReactDevOverlay@webpack-internal:///./node_modules/next/dist/compiled/@next/react-dev-overlay/dist/client.js:852:908
Container@webpack-internal:///./node_modules/next/dist/client/index.js:77:1
AppContainer@webpack-internal:///./node_modules/next/dist/client/index.js:181:25
Root@webpack-internal:///./node_modules/next/dist/client/index.js:360:37

Is your feature request related to a problem? Please describe.
Users have local and remote environment. We need to enable users using the cli to deploy their code both to the local env and remote env

Describe the solution you'd like
agenta init
=> Asks the user for the url/ip of the remote repo

agenta variant deploy
Deploys the variant remotely

We aim to enhance the visibility of the activities taking place within the playground. This includes tracking the cost of each run, identifying the prompts being used, and observing the sub-results of each LLM call that is executed.

Our long-term goal is to monitor this data every time the code is executed.

This issue relates to the design of the feature.

Describe the bug
The way agenta serve works is that the whole folder is compressed and sent to the backend, which decompresses the folder and then create a container out of it.
The problem is that our tutorials ask the user to create a virutal environement in the folder they work in to test the app locally before serving it.
This means the whole myenv folder containg the python files (200M) is being compressed and sent each time to the backend which takes an eternity.

Expected behavior
We should find a way to solve this. The problem is the name of the folder can change. Some possible solutions are:

• Only copy .py, .yaml, .toml and .env files, for the rest the user needs to have a config file to explicitly allow other file format
• Only copy the content of the main folder. If the user wants to include subfolder, they need to explicitly allow it

We should research how other platforms do this (steamship, railway.app...)

In the long run, if we connect with git directly, then we make selecting the files a problem for the user.

This connect to #60

Describe the bug
Agenta variant serve with a container that dies just after starting do not provide enough information to determine the source of the error

To Reproduce
Steps to reproduce the behavior:

• Create a new app using agenta init and choosing some template
• Remove requirements.txt
• Serve the variant

You will get the following uninformative error:

(base) mahmoudmabrouk@MacBook-Pro-2 error % agenta variant serve
? Please enter the variant name 83
Error while building image: 500 Server Error: Internal Server Error for url: http://localhost/api/containers/build_image/?app_name=84&variant_name=83

Similarly if you try using a requirements.txt with the wrong requirements, you get an informative error:

(agenta-py3.9) (base) mahmoudmabrouk@MacBook-Pro ingestion_v1 % agenta variant serve
? Please enter the variant name test2
Variant test2 for App ingestion2 added successfully to Agenta!
Traceback (most recent call last):
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/bin/agenta", line 6, in <module>
    sys.exit(cli())
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/mahmoudmabrouk/agenta/code/agenta-core/agenta-cli/agenta/cli/variant_commands.py", line 246, in serve_cli
    start_variant(variant_name=variant_name, app_folder=app_folder, host=host)
  File "/Users/mahmoudmabrouk/agenta/code/agenta-core/agenta-cli/agenta/cli/variant_commands.py", line 126, in start_variant
    endpoint = client.start_variant(app_name, variant_name, host=host)
  File "/Users/mahmoudmabrouk/agenta/code/agenta-core/agenta-cli/agenta/client/client.py", line 49, in start_variant
    raise APIRequestError(
agenta.client.client.APIRequestError: Request to start variant endpoint failed with status code 500 and error message: {"detail":"Failed to start Docker container for app variant ingestion2/test2"}.

Expected behavior
A clear and concise description of what happened with the error log. In the secon case

2023-07-09 19:22:01 Traceback (most recent call last):
2023-07-09 19:22:01   File "/app/main.py", line 3, in <module>
2023-07-09 19:22:01     import _app  # This will register the routes with the FastAPI application
2023-07-09 19:22:01   File "/app/_app.py", line 3, in <module>
2023-07-09 19:22:01     from llama_index import VectorStoreIndex, SimpleDirectoryReader, load_index_from_storage, StorageContext, Prompt
2023-07-09 19:22:01 ModuleNotFoundError: No module named 'llama_index'

• Basically, the error needs to tell the user how to fix it *

How to solve it

A possible solution is to return the output from the docker log in the error message, to give the option to explain the error and provide a solution there
A better solution is to run the output from the docker log into an LLM to extract the insights :)

Describe the bug
We have the following warning in antd for playground page

Warning: [antd: Tabs] Tabs.TabPane is deprecated. Please use `items` directly. [client.js:1:909](webpack://_N_E/node_modules/next/dist/compiled/@next/react-dev-overlay/dist/client.js?4d30)
    error client.js:1
    error next-dev.js:27
    warning warning.js:19
    call warning.js:39
    warningOnce warning.js:44
    warning warning.js:8
    useLegacyItems useLegacyItems.js:19
    Tabs index.js:66
    React 8
    workLoop scheduler.development.js:266
    flushWork scheduler.development.js:239
    performWorkUntilDeadline scheduler.development.js:533

I ran in an issue after serving agenta in aws, the space got very quickly full and I failed to serve new variants.
Many problems here:

1. There was no error informing me that the issue was a space issue, I needed to read the docker logs
2. No where was I informed that i should get a minimum space
3. Why was it taking so much space? Arent you guys pruning the images after ????

I ran image prune to solve the issue

We are making lots of changes to agenta. Some of our users are finding bugs just because they are using an older version of the agenta CLI. We would like to notify users when their agenta version is different than the last version available in pip.

Describe the solution you'd like
When running agenta we quickly check whether a new version is available, in that case we print a message

A new release of gh is available: $their_version → $new_version`
To upgrade, run: pip install --upgrade agenta

Describe the bug
Currently, we are showing incomplete results for A/B evaluation in the results view. This does not make sense and make the results unclear. we would either need to add filter, to select whether to show or hide incomplete evaluations (per default hidden incomplete evaluation), or not show at all incomplete evaluations.

Additional context
Implementing this might be challenging, since we need first to determine or save somewhere in the backend whether a evaluation is incomplete or complete!

Describe the bug
Variants that are not running (container died) cannot be removed from the UI

To Reproduce
Steps to reproduce the behavior:

1. Stop a container of a variant (or kill docker)
2. Go to playground

You cannot remove the variant, you get instead a message

Error connecting to the variant test. Network Error

To debug this issue, please follow the steps below:

    Run "agenta variant start"
    Verify whether the API is up by checking if http://localhost/ingestion2/test/openapi.json is accessible.
    Check if the Docker container for the variant test is running. The image should be called agenta-server/ingestion2_test.

In case the docker container is not running. Please simply start it (using cli or docker desktop)

If the issue persists please file an issue in github or directly contact us under [email protected]

Expected behavior
A button to delete the variant

See milestone for description

Describe the bug
When I serve the variant (after setting up the aws instance and initializing agenta and providing the URL of my remote host ), I get a wrong link to the app. It says localhost when it should be the URL I provided.

Issue

A number of test users forgot to save a test set after creating it in the UI. The reason is that the Save Test Set position in the view is not very obvious, it's easy to miss

I propose moving the Save button from bottom to near the textinput with the Test Set Name and making it larger to make it unmissable.
Depending on the difficulty of implementation, a warning when leaving the page while having unsaved changes would be nice.

We've noticed some discrepancies in the terminologies used across our frontend and backend codebases.

These inconsistencies could potentially lead to confusion and miscommunication.

As such, we aim to standardise these terms to maintain consistency, enhance code readability, and streamline collaboration across teams.

We would like to rename:

EvaluationRow, evaluationRow, evaluation_row .. To evaluation_scenario

We also need a migration script to the new schema.

Path: /installation/local-installation/local-installation

Add troubleshooting:

• What happens if you have the port used
• How do you clean your docker

Is your feature request related to a problem? Please describe.
Agenta variant serve with a container that dies just after starting do not provide enough information to determine the source of the error

To Reproduce
Steps to reproduce the behavior:

1. Create a new app using agenta init and choosing some template
2. Remove requirements.txt
3. Serve the variant

You will get the following uninformative error:

(base) mahmoudmabrouk@MacBook-Pro-2 error % agenta variant serve
? Please enter the variant name 83
Error while building image: 500 Server Error: Internal Server Error for url: http://localhost/api/containers/build_image/?app_name=84&variant_name=83

Similarly if you try using a requirements.txt with the wrong requirements, you get an informative error:

(agenta-py3.9) (base) mahmoudmabrouk@MacBook-Pro ingestion_v1 % agenta variant serve
? Please enter the variant name test2
Variant test2 for App ingestion2 added successfully to Agenta!
Traceback (most recent call last):
File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/bin/agenta", line 6, in
sys.exit(cli())
File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/mahmoudmabrouk/Library/Caches/pypoetry/virtualenvs/agenta-UfaD-TV3-py3.9/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/Users/mahmoudmabrouk/agenta/code/agenta-core/agenta-cli/agenta/cli/variant_commands.py", line 246, in serve_cli
start_variant(variant_name=variant_name, app_folder=app_folder, host=host)
File "/Users/mahmoudmabrouk/agenta/code/agenta-core/agenta-cli/agenta/cli/variant_commands.py", line 126, in start_variant
endpoint = client.start_variant(app_name, variant_name, host=host)
File "/Users/mahmoudmabrouk/agenta/code/agenta-core/agenta-cli/agenta/client/client.py", line 49, in start_variant
raise APIRequestError(
agenta.client.client.APIRequestError: Request to start variant endpoint failed with status code 500 and error message: {"detail":"Failed to start Docker container for app variant ingestion2/test2"}.

Solution

1. When running agenta variant serve, check whether requirements.txt exists in the app folder. If not, warn the user (same logic we do for .env)
2. When we get a status code 500, give the users pointer to solve the problem: (e.g. "Serving the variant failed. Here is how you can solve the issue. First make sure that the requirements.txt file has all the dependencies that you need. Second check the docker logs for the backend image to see the error when running the docker)

line 13, in generate
llm = OpenAI(temperature=temperature)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.init
pydantic.error_wrappers.ValidationError: 1 validation error for OpenAI
root
Did not find openai_api_key, please add an environment variable OPENAI_API_KEY which contains it, or pass openai_api_key as a named parameter. (type=value_error)

I need to add load_dotenv() in the generate function to make it work

Describe the bug
The evaluation view requires the columns in the test set to have the same names of the inputs in the variant that is to be run. When this does not happen we get an error instead (instead of a clear message explaining what happened).

To Reproduce
Steps to reproduce the behavior:

1. Create a test set with wrong column names
2. Do evaluation

Expected behavior
A clear and concise message explaining what is the issue and how to solve it

Screenshots

Describe the bug
We have multiple warnings because of deprecation from antd in the main page (apps)

Warning: [antd: Menu] `children` will be removed in next major version. Please use `items` instead. [client.js:1:909](webpack://_N_E/node_modules/next/dist/compiled/@next/react-dev-overlay/dist/client.js?4d30)
    error client.js:1
    error next-dev.js:27
    warning warning.js:19
    call warning.js:39
    warningOnce warning.js:44
    warning warning.js:8
    InternalMenu menu.js:57
    React 8
    workLoop scheduler.development.js:266
    flushWork scheduler.development.js:239
    performWorkUntilDeadline scheduler.development.js:533

Warning: [antd: Modal] `visible` will be removed in next major version, please use `open` instead. [client.js:1:909](webpack://_N_E/node_modules/next/dist/compiled/@next/react-dev-overlay/dist/client.js?4d30)
    error client.js:1
    error next-dev.js:27
    warning warning.js:19
    call warning.js:39
    warningOnce warning.js:44
    warning warning.js:8
    Modal Modal.js:57

Describe the bug
When we refresh the Evaluate view we get an error that is hard to get rid of.

The error is:

SyntaxError: Cannot use import statement outside a module
...

To Reproduce
Steps to reproduce the behavior:

1. Go to 'Evaluate' view
2. Refresh the page

Expected behavior
No errors on a page refresh.

Screenshots

Describe the bug
When using the cli, if you create a variant name with a space, you get an error.

Expected behavior
The CLI should not allow the usage of variant names with spaces. The same logic we use to the app name should be used

Describe the bug

Saving changes to a new variant in the playground clears all test data in the view

To Reproduce
Steps to reproduce the behavior:

1. Go to playground
2. Add some test data (add data, click run, add a new row and do the same
3. Click Save changes

The test data is now lost!

Expected behavior
The test data should stay the same after the save

Describe the bug
You currently cannot create an app called ss-xxx or ss_xxx when running agenta init.

To Reproduce
Steps to reproduce the behavior:

1. agenta init
2. Choose name xxx-xxx

Expected behavior
You should be able to create such app name. You cannot create one that has spaces or such

Description:
Running evaluations with lots of data using an app that uses openai does not work due to rate limitation error.

The issue is that this is neither solved in the frontend through exponential backoff, nor is the error shown solmewhere

Describe the bug
Currently if you have variants with a different inputs. Evaluation does not work, even if the test set fits the variant.

To reproduce

1. Create two variants from CLI, one with one input, the second with two inputs and serve them.
2. Create a test set in the UI with two inputs (fitting the second variant)
3. Go the evaluation and start an evalution
   You will get an error

Context
Current solution is comparing the inputs of the test set to the ones from the first variant.

Solution

1. Compare the inputs between the test set and the variant
2. Accepts subinputs, meaning that if the test sets have two columns, while the variant take one input described in the column, then that's alright.

We would like to add a new evaluation type (AI Critic).

You can see the implementation in langchain

https://github.com/langchain-ai/langchain/blob/490ad93b3cf7d24b30f8993f860b654ff107e638/libs/langchain/langchain/evaluation/qa/eval_prompt.py

An example of the UI can be seen here