langchain-ai / auto-evaluator Goto Github PK

View Code? Open in Web Editor NEW

730.0 11.0 96.0 193.48 MB

Home Page: https://autoevaluator.langchain.com/

License: Other

JavaScript 0.17% TypeScript 52.65% CSS 1.01% Python 35.04% Jupyter Notebook 11.13%

auto-evaluator's Issues

Add sidebar scrolling

Bottom if cut-off without ability to scroll:

refactor Body.tsx into multiple sub react components

Add 50MB file limit and single file

add logrocket for error tracking and product analytics

Remove the Insight panel (not sure it is correct and uses subjective weighting)

Example:

[backlog] App re-generating index when it does not need to

I ran an expt.

I updated to use Descriptive.

It re-generated the index.

Chart w/ answer score vs retrieval score and latency at size

add a file to the demo envioronment

File transfer and reading time is slow in prod

See results:

Transfer time ~30 sec for ~30MB of files: consider stripping out images from pdf prior to sending.
Loading time per file ~5-8 sec vs ~0 sec on local.

Possible to hang during generate_eval( )

If the text produces consistent JSONdecode error during QA generation, may get stuck in while loop.

API continues to run after connection to client is closed / refreshed

I kick off a job.

Then refresh client.

API will proceed:

But:
1/ Does not happen if API and front-end are deployed locally.
2/ Does not happen if only the front-end is deployed locally.

It appears to be a problem Vercel (remote deployed front-end).

Create "default" page w/ pre-populated data (from Karpathy pod) and "playground"

Anthropic model appears to be deprecated

/opt/venv/lib/python3.8/site-packages/langchain/llms/anthropic.py:130: UserWarning: This Anthropic LLM is deprecated. Please use `from langchain.chat_models import ChatAnthropic` instead

Fix SVM in prod

It is timing out; it may require a Langchain upgrade.

Add instructions to the demo landing page

Top (Text box, grey background)

Welcome to the auto-evaluator! This is an app to evaluate the performance of question-answering LLM chains. This demo has pre-loaded two things: (1) a document (the Lex Fridman podcast with Andrej Karpathy) and (2) a "test set" of question-answer pairs for this episode. The aim to evaluate the performance of various question-answering LLM chain configuration against test set. You can build any QA chain using on the components and score its performance.

Button (Text box, green background)

Choose the question-answering chain configuration (left) and launch an experiment using the button below. For more detail on each setting, see full the documentation here.

Color: green
Title: run experiment

Summary

Re-name initial row as baseline

Experiment Results (Text box, grey background)

This table shows the each question-answer pair from the test set along with the model's answer to the question. The app will score two things: (1) the relevance of the retrieved documents relative to the question and (2) the similarity of the LLM generated answer relative to ground truth answer. The prompts for both can be seen here and can be chosen by the user in the drop-down list Grading prompt style. The FAST prompt will only have the LLM grader output the score. The other prompts will also produce an explanation.

Write-up on learnings / opportunities

[Nit] Change link in repo to new app (https://autoevaluator.langchain.com/)

Apparently Harrison needs to do this; I don't have edit permission.

Fails to run w/ Llama-Ix

And gives the incorrect Insight
The experiment that performed the best was Experiment #2 due to combination of accuracy and latency.

Remove Doppler dep

Still need Doppler for local testing.

Should we remove?

And for local tested we should create a .env template for folks for follow (e.g., with all required API keys).

Add OpenAI closedQA grader prompt

See here: https://github.com/openai/evals/blob/main/evals/registry/modelgraded/closedqa.yaml

Expt numbering is wrong after we start w/ 3 examples

Experiment numbers are not refreshing correctly

Started w/ expts 1-3 filled out.

Then updated eval set and ran an expt.

This expt was expt 4.

Then it rotated back to expts 2, 3, ... n.

[mega request] Caching on back-end of index so it does not re-generate each expt

disable adding test data by default in the playground environment

Add Mantine table (collapsible cols)

PR is #14

horizontal scroll is needed on Summary table

Prod (likely) hitting OOM

TL;DR we see OOM (or server returning

These are 15 large pdfs fail (503 server error) on prod, but work on local:

curl -X POST \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "num_eval_questions=1" \
-F "chunk_chars=1000" \
-F "overlap=100" \
-F "split_method=RecursiveTextSplitter" \
-F "retriever_type=similarity-search" \
-F "embeddings=OpenAI" \
-F "model_version=gpt-3.5-turbo" \
-F "grade_prompt=Fast" \
-F "num_neighbors=3" \
https://auto-evaluator-production.up.railway.app/evaluator-stream

The first 7 work on prod w/ similarity-search:

curl -X POST \                                                                                        
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "num_eval_questions=1" \
-F "chunk_chars=1000" \
-F "overlap=100" \
-F "split_method=RecursiveTextSplitter" \
-F "retriever_type=similarity-search" \
-F "embeddings=OpenAI" \
-F "model_version=gpt-3.5-turbo" \
-F "grade_prompt=Fast" \
-F "num_neighbors=3" \
https://auto-evaluator-production.up.railway.app/evaluator-stream

They are ~17 MB.

Adding any additional files fail (e.g., below fails):

curl -X POST \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]" \
-F "num_eval_questions=1" \
-F "chunk_chars=1000" \
-F "overlap=100" \
-F "split_method=RecursiveTextSplitter" \
-F "retriever_type=similarity-search" \
-F "embeddings=OpenAI" \
-F "model_version=gpt-3.5-turbo" \
-F "grade_prompt=Fast" \
-F "num_neighbors=3" \
https://auto-evaluator-production.up.railway.app/evaluator-stream

Make all side panels drop-down lists

Center / enlarge the spinner (maybe show processing stages, as before)

current menu item should be underlines

if you in Demo / then dont who Demo menu item

OpenAI invalid API key error

https://auto-evaluator.sentry.io/issues/4147078946/?query=is%3Aunresolved&referrer=issue-stream&stream_index=1

bug: incorrect # of output rows

Sometimes the output will have 1 less than the # of eval pairs requested

Add support for modifying input prompt

Allow user to specify the prompt (both for the QA task and for the grading) in the UI.

Back-end crashing

Reports crashed -

Server crashed -

Logging is not obvious -

INFO:     Uvicorn running on http://0.0.0.0:7106/ (Press CTRL+C to quit)
INFO:uvicorn.error:Uvicorn running on http://0.0.0.0:7106/ (Press CTRL+C to quit)
INFO:     192.168.0.2:33390 - "POST /evaluator-stream HTTP/1.1" 200 OK
2023-05-05 09:04:42,541 loglevel=INFO   logger=evaluator_app run_evaluator() L334  Reading file: PosteVivereProtetti_CGA.pdf
2023-05-05 09:04:42,544 loglevel=INFO   logger=evaluator_app run_evaluator() L338  File PosteVivereProtetti_CGA.pdf is a PDF
2023-05-05 09:05:01,346 loglevel=INFO   logger=evaluator_app run_evaluator() L355  Splitting texts
2023-05-05 09:05:01,347 loglevel=INFO   logger=evaluator_app split_texts() L80   `Splitting doc ...`
2023-05-05 09:05:01,375 loglevel=INFO   logger=evaluator_app run_evaluator() L358  Make LLM
model!
gpt-4
2023-05-05 09:05:01,376 loglevel=INFO   logger=evaluator_app run_evaluator() L361  Make retriever
2023-05-05 09:05:01,376 loglevel=INFO   logger=evaluator_app make_retriever() L120  `Making retriever ...`
2023-05-05 09:05:07,734 loglevel=INFO   logger=evaluator_app run_evaluator() L365  Make chain
2023-05-05 09:05:07,735 loglevel=INFO   logger=evaluator_app generate_eval() L46   `Generating eval QA pair ...`
2023-05-05 09:05:32,805 loglevel=INFO   logger=evaluator_app run_eval() L236  `Running eval ...`
2023-05-05 09:05:50,387 loglevel=INFO   logger=evaluator_app grade_model_answer() L177  `Grading model answer ...`
2023-05-05 09:05:55,938 loglevel=INFO   logger=evaluator_app grade_model_retrieval() L205  `Grading relevance of retrieved docs ...`
INFO:     192.168.0.4:36928 - "POST /evaluator-stream HTTP/1.1" 200 OK
2023-05-05 09:06:33,392 loglevel=INFO   logger=evaluator_app run_evaluator() L334  Reading file: karpathy-pod.txt
2023-05-05 09:06:33,392 loglevel=INFO   logger=evaluator_app run_evaluator() L347  File karpathy-pod.txt is a TXT
2023-05-05 09:06:33,392 loglevel=INFO   logger=evaluator_app run_evaluator() L355  Splitting texts
2023-05-05 09:06:33,392 loglevel=INFO   logger=evaluator_app split_texts() L80   `Splitting doc ...`
2023-05-05 09:06:33,506 loglevel=INFO   logger=evaluator_app run_evaluator() L358  Make LLM
model!
gpt-3.5-turbo
2023-05-05 09:06:33,506 loglevel=INFO   logger=evaluator_app run_evaluator() L361  Make retriever
2023-05-05 09:06:33,506 loglevel=INFO   logger=evaluator_app make_retriever() L120  `Making retriever ...`
2023-05-05 09:06:35,081 loglevel=INFO   logger=evaluator_app run_evaluator() L365  Make chain
2023-05-05 09:06:35,082 loglevel=INFO   logger=evaluator_app run_eval() L236  `Running eval ...`
2023-05-05 09:06:38,116 loglevel=INFO   logger=evaluator_app generate_eval() L46   `Generating eval QA pair ...`
2023-05-05 09:06:48,428 loglevel=INFO   logger=evaluator_app run_eval() L236  `Running eval ...`
2023-05-05 09:06:57,001 loglevel=INFO   logger=evaluator_app grade_model_answer() L177  `Grading model answer ...`
2023-05-05 09:07:00,491 loglevel=INFO   logger=evaluator_app grade_model_answer() L177  `Grading model answer ...`
2023-05-05 09:07:02,402 loglevel=INFO   logger=evaluator_app grade_model_retrieval() L205  `Grading relevance of retrieved docs ...`
2023-05-05 09:07:04,823 loglevel=INFO   logger=evaluator_app run_eval() L236  `Running eval ...`
2023-05-05 09:07:15,961 loglevel=INFO   logger=evaluator_app grade_model_retrieval() L205  `Grading relevance of retrieved docs ...`
2023-05-05 09:07:22,977 loglevel=INFO   logger=evaluator_app generate_eval() L46   `Generating eval QA pair ...`
2023-05-05 09:07:26,002 loglevel=INFO   logger=evaluator_app grade_model_answer() L177  `Grading model answer ...`
2023-05-05 09:07:38,376 loglevel=INFO   logger=evaluator_app run_eval() L236  `Running eval ...`
2023-05-05 09:07:39,099 loglevel=INFO   logger=evaluator_app grade_model_retrieval() L205  `Grading relevance of retrieved docs ...`
2023-05-05 09:07:47,511 loglevel=INFO   logger=evaluator_app grade_model_answer() L177  `Grading model answer ...`
INFO:     192.168.0.4:56978 - "POST /evaluator-stream HTTP/1.1" 200 OK
2023-05-05 09:07:50,268 loglevel=INFO   logger=evaluator_app run_evaluator() L334  Reading file: karpathy-pod.txt
2023-05-05 09:07:50,269 loglevel=INFO   logger=evaluator_app run_evaluator() L347  File karpathy-pod.txt is a TXT
2023-05-05 09:07:50,269 loglevel=INFO   logger=evaluator_app run_evaluator() L355  Splitting texts
2023-05-05 09:07:50,269 loglevel=INFO   logger=evaluator_app split_texts() L80   `Splitting doc ...`
2023-05-05 09:07:50,386 loglevel=INFO   logger=evaluator_app run_evaluator() L358  Make LLM
model!
gpt-4
2023-05-05 09:07:50,386 loglevel=INFO   logger=evaluator_app run_evaluator() L361  Make retriever
2023-05-05 09:07:50,386 loglevel=INFO   logger=evaluator_app make_retriever() L120  `Making retriever ...`
2023-05-05 09:07:50,743 loglevel=INFO   logger=evaluator_app grade_model_retrieval() L205  `Grading relevance of retrieved docs ...`
2023-05-05 09:07:51,940 loglevel=INFO   logger=evaluator_app run_evaluator() L365  Make chain
2023-05-05 09:07:51,940 loglevel=INFO   logger=evaluator_app run_eval() L236  `Running eval ...`
2023-05-05 09:08:01,489 loglevel=INFO   logger=evaluator_app grade_model_answer() L177  `Grading model answer ...`
2023-05-05 09:08:09,404 loglevel=INFO   logger=evaluator_app grade_model_retrieval() L205  `Grading relevance of retrieved docs ...`
2023-05-05 09:08:14,427 loglevel=INFO   logger=evaluator_app run_eval() L236  `Running eval ...`
2023-05-05 09:08:24,486 loglevel=INFO   logger=evaluator_app grade_model_answer() L177  `Grading model answer ...`
2023-05-05 09:08:28,668 loglevel=INFO   logger=evaluator_app grade_model_retrieval() L205  `Grading relevance of retrieved docs ...`
2023-05-05 09:08:35,293 loglevel=INFO   logger=evaluator_app run_eval() L236  `Running eval ...`
INFO:     192.168.0.4:60550 - "POST /evaluator-stream HTTP/1.1" 200 OK
2023-05-05 09:08:45,730 loglevel=INFO   logger=evaluator_app run_evaluator() L334  Reading file: PosteVivereProtetti_CGA.pdf
2023-05-05 09:08:45,731 loglevel=INFO   logger=evaluator_app run_evaluator() L338  File PosteVivereProtetti_CGA.pdf is a PDF
2023-05-05 09:08:49,638 loglevel=INFO   logger=evaluator_app grade_model_answer() L177  `Grading model answer ...`
2023-05-05 09:08:55,964 loglevel=INFO   logger=evaluator_app grade_model_retrieval() L205  `Grading relevance of retrieved docs ...`
2023-05-05 09:08:59,938 loglevel=INFO   logger=evaluator_app run_eval() L236  `Running eval ...`
2023-05-05 09:09:04,879 loglevel=INFO   logger=evaluator_app run_evaluator() L355  Splitting texts
2023-05-05 09:09:04,879 loglevel=INFO   logger=evaluator_app split_texts() L80   `Splitting doc ...`
2023-05-05 09:09:04,906 loglevel=INFO   logger=evaluator_app run_evaluator() L358  Make LLM
model!
gpt-4
2023-05-05 09:09:04,906 loglevel=INFO   logger=evaluator_app run_evaluator() L361  Make retriever
2023-05-05 09:09:04,906 loglevel=INFO   logger=evaluator_app make_retriever() L120  `Making retriever ...`
2023-05-05 09:09:16,493 loglevel=INFO   logger=evaluator_app grade_model_answer() L177  `Grading model answer ...`
2023-05-05 09:09:20,289 loglevel=INFO   logger=evaluator_app grade_model_retrieval() L205  `Grading relevance of retrieved docs ...`
2023-05-05 09:09:24,466 loglevel=INFO   logger=evaluator_app run_eval() L236  `Running eval ...`
2023-05-05 09:09:30,584 loglevel=INFO   logger=evaluator_app run_evaluator() L365  Make chain
2023-05-05 09:09:30,585 loglevel=INFO   logger=evaluator_app run_eval() L236  `Running eval ...`

Change app name: "Evaluator AI - evaluate your QA chains." to "Auto Evaluator"

Anthropic model is failing on prod

We are probably missing the API key?

Bug in the FAST scoring

Bad answer scored as PASS:

But using DETAILED prompt, it is corrected marked as INCORRECT:

Failure on upload of some PDFs

GPT-3
https://arxiv.org/pdf/2005.14165.pdf

Galatica
https://arxiv.org/abs/2211.09085

Chinchilla
https://arxiv.org/abs/2203.15556

Removal of doppler means that some env variables were missing.

Both local (main) and prod are broken.

Adding these env variables to .env.local fixes local -

NEXT_PUBLIC_EVALUATOR_API_URL="http://127.0.0.1:8000"
NEXT_PUBLIC_API_URL="http://127.0.0.1:8000"

What is required for prod to work?

Last logging:

2023-05-02 17:31:03,008 loglevel=INFO   logger=evaluator_app grade_model_retrieval() L206  `Grading relevance of retrieved docs ...`
INFO:     Shutting down
2023-05-02 17:38:48,691 loglevel=INFO   logger=uvicorn.error shutdown() L253  Shutting down
INFO:     Waiting for application shutdown.
2023-05-02 17:38:48,793 loglevel=INFO   logger=uvicorn.error shutdown() L66   Waiting for application shutdown.
INFO:     Application shutdown complete.
2023-05-02 17:38:48,793 loglevel=INFO   logger=uvicorn.error shutdown() L77   Application shutdown complete.
INFO:     Finished server process [1]
2023-05-02 17:38:48,793 loglevel=INFO   logger=uvicorn.error serve() L85   Finished server process [1]

Not quite obvious why it crashed based on this logging.

Catch server errors and return alert to the user

Test w/ GPT-4 paper (example of invalid token):

ValueError: Encountered text corresponding to disallowed special token '<|endofprompt|>'.
If you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|endofprompt|>', ...}`.
If you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|endofprompt|>'})`.
To disable this check for all special tokens, pass `disallowed_special=()`.

langchain-ai / auto-evaluator Goto Github PK

auto-evaluator's Issues

Recommend Projects

Recommend Topics

Recommend Org