davidgonmar / my-makemore

Implementing simple character level language models

My makemore

Inspired by Andrej Karpathy's makemore, this is a collection of different models that generate names given a dataset of names. It can be generalized to any dataset of strings.

Models

• BigramByCounting: Simple model trained by counting bigrams in the dataset, and sampling from the distribution of bigrams.
• BigramNN: Same as BigramByCounting, but using a SGD trained neural network to predict the next character given the previous one.
• MLP: A model, following (slightly modified) A Neural Probabilistic Language Model, Bengio et al..
• RNN, LSTM, GRU: Recurrent neural network models, following the well-known architectures.

Usage

To run any of the scripts, just run:

python <model>.py

Will train the model and generate names, or ask the user for an initial sequence to generate names from.

Installation

Requirements

Python 3.11.5 was used to develop this project. The requirements can be installed with:

pip install -r requirements.txt

Probably, any modern version of Python + Torch will work just fine.

References

• Andrej Karpathy's makemore
• Andrej Karpathy's YouTube series
• A Neural Probabilistic Language Model, Bengio et al.
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Recurrent neural network based language model
• Generating Sequences With Recurrent Neural Networks
• Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks

License

This project is licensed under the MIT License - see the LICENSE file for details.