intrinsiclabsai / gbnfgen

TypeScript generator for llama.cpp Grammar directly from TypeScript interfaces

What is it?

gbnfgen is a library for generating grammars based on your typed JSON objects, described through normal TypeScript interfaces and enums.

Generate API calls (and responses) parse free text into structured formats, and build your own llama.cpp-powered agents with ease!

Check out the Live Demo.

Installation

npm i --save @intrinsicai/gbnfgen

Quickstart

import { compile, serializeGrammar } from "@intrinsicai/gbnfgen";

// Supporting Enum for multiple choices (cannot be numbers)
const grammar = compile(
    `enum Mood {
      Happy = "happy",
      Sad = "sad",
      Grateful = "grateful",
      Excited = "excited", 
      Angry = "angry, 
      Peaceful = "peaceful"
    }
    
    interface Person {
         name: string;
         occupation: string;
         age: number;
         mood: Mood,
     }`, "Person");

Why?

Language models allow for open-ended generation of text via autoregressive execution, whereby they generate one token, feed it through a decoder to get a probability distribution of follow-on tokens, and sample from that distribution in an iterative process to generate text.

This is great for activities like generating marketing prose or writing stories, but some of the most exciting usecases involve plugging autonomous LLM agents into existing systems. Interacting with databases and REST APIs requires the model's output to fit a pre-existing schema, usually serialized as JSON.

llama.cpp recently incorporated grammar-based sampling as part of an effort to make it easier to constrain LLM output. Users do this by authoring GBNF files, which are a constrained flavor of Backus-Naur notation for defining a context-free language.

gbnfgen takes the difficulty out of building grammars to let your LLM apps interact with external systems.

What TypeScript types are supported?

Currently the library is narrowly focused, we only provide support for the following types

• string and string[]
• number and number[]
• boolean
• Interface types and single-dimensional arrays of interface types. These must be interface types that you define within a single call to compile
• String enums, as defined in this section of the TypeScript handbook.

Inspiration

Microsoft's TypeChat is a similar solution but targeted at OpenAI and other cloud-hosted models. They effectively take an interface definition from the user code, then generate text with GPT4. They use the TypeScript Compiler API to type-check the output of the code to see if it's valid JSON that conforms to the typing.

Most users of llama.cpp are either using the C++ code directly or using it via the llama-cpp-python bindings to Python. TypeScript interfaces provide both humand and machine-friendly representations of typed objects that millions of users are already familiar with, so we decided that it served as a great description format that users of llama.cpp could use to bridge between the other

---

Up Next

• Improved type-checking of code passed to compile. Currently we just extract the AST without doing any explicit type-checking, so things like duplicate property declarations and other simple mistakes will not be caught.
• Support for more type declarations
  • Literals
  • Union types
  • Type aliases

gbnfgen's Issues

Exclude unicode characters from string?

Hi, I can't find any documentation but would like to exclude unicode characters from strings.

I.e. 0x00-0x08 and 0x0B-0x1F so that I don't get crap. Is there a way in GBNF to eliminate these? It would be useful to have this in the string set.

An enum member name must be followed by a ',', '=', or '}

Running the quick start, node 20.x

Error: Compilation failed: Unterminated string literal.
An enum member name must be followed by a ',', '=', or '}'.
    at compileSync (file:///path/to/ai/experiments/grammer-gen/node_modules/@intrinsicai/gbnfgen/dist/compiler.js:169:15)
    at file:///path/to/ai/experiments/grammer-gen/node_modules/@intrinsicai/gbnfgen/dist/compiler.js:137:29
    at new Promise (<anonymous>)
    at compile (file:///path/to/ai/experiments/grammer-gen/node_modules/@intrinsicai/gbnfgen/dist/compiler.js:135:12)
    at file:///path/to/ai/experiments/grammer-gen/gen.js:4:17
    at ModuleJob.run (node:internal/modules/esm/module_job:218:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:329:24)
    at async loadESM (node:internal/process/esm_loader:28:7)
    at async handleMainPromise (node:internal/modules/run_main:113:12)

Request: support for Python classes and/or `dataclass`

It seems like TypeScript and Python are two of the main languages for the LLM community.

It would be cool to support Python with this tool:

• Native classes
• dataclasses.dataclass

Allow for mixed types lists like (string|number)[]

Title

Could it be possible to support lists with multiple types ?

Thanks.

String should not contain control characters.

Right now, strings can for example contain a new line, not an escaped one.

Which will lead to incorrect json generation

Here is my suggestion :

string ::= "\"" (char | escapeSequence)* "\""
char ::= [^"\\\x7F\x00-\x1F]
escapeSequence ::= "\\\"" | "\\\\" | "\\n" | "\\r" | "\\t" | "\\b" | "\\f" 

I could open a PR but I could be missing something so I open an issue to discuss this

top level array

how to declare top level object array ?
compile(interface Character { name: string; }, Character[]); #=> Error: Root type Character[] is not one of the declared types
compile(interface Character { name: string; } type Root = Character[];, Root); #=> Error: Invalid top-level declaration of kind TypeAliasDeclaration