This file contains the implementation of a simple lexer for a programming language. The lexer is responsible for converting source code into meaningful tokens.
To use this lexer, simply call the tokenize
function with your source code in example.txt
. The function will return a vector of Token
matching any of the predefined TokenType
.
This is my own C++ implementation inspired by Tyler Laceby's "How To Build A Programming Language From Scratch" tutorial originally written for TypeScript. You can find the tutorial here.