sjehan / gbnfgen

TypeScript generator for llama.cpp Grammar directly from TypeScript interfaces

What is it?

gbnfgen is a library for generating grammars based on your typed JSON objects, described through normal TypeScript interfaces and enums.

Generate API calls (and responses) parse free text into structured formats, and build your own llama.cpp-powered agents with ease!

Check out the Live Demo.

Installation

npm i --save @intrinsicai/gbnfgen

Quickstart

import { compile, serializeGrammar } from "@intrinsicai/gbnfgen";

// Supporting Enum for multiple choices (cannot be numbers)
const grammar = compile(
    `enum Mood {
      Happy = "happy",
      Sad = "sad",
      Grateful = "grateful",
      Excited = "excited", 
      Angry = "angry, 
      Peaceful = "peaceful"
    }
    
    interface Person {
         name: string;
         occupation: string;
         age: number;
         mood: Mood,
     }`, "Person");

Why?

Language models allow for open-ended generation of text via autoregressive execution, whereby they generate one token, feed it through a decoder to get a probability distribution of follow-on tokens, and sample from that distribution in an iterative process to generate text.

This is great for activities like generating marketing prose or writing stories, but some of the most exciting usecases involve plugging autonomous LLM agents into existing systems. Interacting with databases and REST APIs requires the model's output to fit a pre-existing schema, usually serialized as JSON.

llama.cpp recently incorporated grammar-based sampling as part of an effort to make it easier to constrain LLM output. Users do this by authoring GBNF files, which are a constrained flavor of Backus-Naur notation for defining a context-free language.

gbnfgen takes the difficulty out of building grammars to let your LLM apps interact with external systems.

What TypeScript types are supported?

Currently the library is narrowly focused, we only provide support for the following types

• string and string[]
• number and number[]
• boolean
• Interface types and single-dimensional arrays of interface types. These must be interface types that you define within a single call to compile
• String enums, as defined in this section of the TypeScript handbook.

Inspiration

Microsoft's TypeChat is a similar solution but targeted at OpenAI and other cloud-hosted models. They effectively take an interface definition from the user code, then generate text with GPT4. They use the TypeScript Compiler API to type-check the output of the code to see if it's valid JSON that conforms to the typing.

Most users of llama.cpp are either using the C++ code directly or using it via the llama-cpp-python bindings to Python. TypeScript interfaces provide both humand and machine-friendly representations of typed objects that millions of users are already familiar with, so we decided that it served as a great description format that users of llama.cpp could use to bridge between the other

---

Up Next

• Improved type-checking of code passed to compile. Currently we just extract the AST without doing any explicit type-checking, so things like duplicate property declarations and other simple mistakes will not be caught.
• Support for more type declarations
  • Literals
  • Union types
  • Type aliases