jakobjbauer / llm-question-parsing

LLM-Question-Parsing

This repository is part of my bachelor thesis and provides the code to test different LLMs with precompiled prompts.

The prompts are static and can be generated using prompt_builder.py

abstract_model.py traverses the prompts generated by the prompt builder and logs the result in a local csv file located at ../logs. In order to test a specific Model a specific implementation needs to be written, which inherits from AbstractModel and implements the send_prompt function.

Implementations for a few models are already provided in model_implementations using various APIs.

The specific models can be run using run_<model_name>.py using python 3.9 or greater

Reproduce the Experiments

1. Set up an Ubuntu based Operating System
2. Install Python 3.9
3. Clone GitHub (https://github.com/JakobJBauer/LLM-Question-Parsing)
4. Install the required pip modules.
5. GPT 3.5

• Create an account or sign in on Open AI (https://openai.com).
• Add a payment method to your account.
• Install openai using pip.
• Create a new API key and insert the token into run_gpt35.py.
• Run run_gpt35.py using Python 3.9.

6. GPT4

• Create an account or sign in on Open AI (https://openai.com).
• Subscribe to the ChatGPT Plus plan.
• Follow the instructions listen on GitHub (https://github.com/Erol444/gpt4- openai-api).
• Insert the token into run_gpt4.py.
• Run run_gpt4.py using Python 3.9.

7. Vicuna 13b

• Create an account or sign in on Replicate (https://replicate.com/).
• Add a payment method to your account.
• Install replicate using pip.
• Create a new API key and add it to the environment variable "REPLI- CATE_API_TOKEN".
• Run run_vicuna13b.py.

8. GPT4All 32 6.2. Results and Discussion

• Install GPT4All model and weights by following the instructions on GitHub (https://github.com/nomic-ai/gpt4all).
• Install gpt4all using pip.
• Run run_gpt4all.py.

9. Google Bard

• Create an account or sign in on Google (https://www.google.com/).
• If you are in a country unsupported by Google Bard, reroute all your internet traffic through a supported country using a VPN service.
• Follow the instructions on GitHub (https://github.com/dsdanielpark/Bard- API).
• Insert your API token into run_bard.py
• Run run_bard.py.

10. Run create_statistics.py.
11. The results are now available in ..\stats\collection