zhaochenyang20 / zeno-build

Build, evaluate, understand, and fix LLM-based apps

Zeno Build

Zeno Build is a tool for developers who want to quickly build, compare, and iterate on applications using large language models.

It provides:

• Simple examples of code to build LLM-based apps. The examples are architecture agnostic, we don't care if you are using OpenAI, LangChain, or Hugging Face.
• Experiment management and hyperparameter optimization code, so you can quickly kick off experiments using a bunch of different settings and compare the results.
• Evaluation of LLM outputs, so you can check if your outputs are correct, fluent, factual, interesting, or "good" by whatever definition of good you prefer! Use these insights to compare models and iteratively improve your application with model, data, or prompt engineering.

Sound interesting? Read on!

Getting Started

To get started with zeno-build, install the package from PyPI:

pip install zeno-build

Next, start building! Browse to the docs directory to get a primer or to the examples/ directory, where we have a bunch of examples of how you can use zeno-build for different tasks, such as chatbots, text summarization, or text classification.

Each of the examples include code for running experiments and evaluating the results. zeno-build will produce a comprehensive report with the Zeno AI evaluation platform.

Interactive Demos/Reports

Using Zeno Build, we have generated reports and online browsing demos of state-of-the-art systems for different popular generative AI tasks. Check out our pre-made reports below:

• Chatbots (Report, Browser): A report comparing different methods for creating chatbots, including API-based models such as ChatGPT and Cohere, with open-source models such as Vicuna, Alpaca, and MPT.
• Translation (Report, Browser): A report comparing GPT-based methods, Microsoft Translator, and the best system from the Conference on Machine Translation.

Building Your Own Apps (and Contributing Back)

Each of the examples in the examples/ directory is specifically designed to be self-contained and easy to modify. To get started building your own apps, we suggest that you first click into the directory and read the general README, find the closest example to what you're trying to do, copy the example to the new directory, and start hacking!

If you build something cool, we'd love for you to contribute it back. We welcome pull requests of both new examples, new reports for existing examples, and new functionality for the core zeno_build library. If this is of interest to you, please click through to our contributing doc doc to learn more.

Get in Touch

If you have any questions, feature requests, bug reports, etc., we recommend getting in touch via the github issues page or discord, where the community can discuss and/or implement your suggestions!