bellthomas / gpt.local

A work-in-progress, from-scratch implementation of a generative pre-trained transformer (GPT) in vanilla PyTorch. Intended as a personal sandbox + learning environment for both training and inference.

gpt.local

A work-in-progress, from-scratch implementation of a generative pre-trained transformer (GPT) in vanilla Pytorch. The purpose of this project is to be a personal sandbox + learning environment, looking at both training and inference, with a long-term aim of becoming a self-hosted language assistant running in a homelab environment.

Inspired by OpenAI's GPT 2, Andrej Karpathy's nanoGPT, and Hugging Face's GPT 2 implementation.

---

Getting Started

git clone [email protected]:bellthomas/gpt.local.git
cd gpt.local

# Download training data.
python -m data
> Downloading collection: bellthomas/herodotus
> ...


# Train a GPT.
python -m train --collection "openwebtext" --experiment "openwebtext-1" --device cpu
> *Experiment: openwebtext-1
> Data: ./gpt.local/data/openwebtext/{validation,training}
> Training... (parameters: 124.11M, device: cpu)
>     (0) loss 10.9385 (9715.26ms, ~0.51 tflops)
>     ...