zetah11 / lexi Goto Github PK
View Code? Open in Web Editor NEWa streaming lexer
a streaming lexer
lexi lexi is a simple proof-of-concept for a fast, streaming lexer. this is designed to lex with only a partial view of the source code available at any moment, and without needing to take up more memory than necessary. it is fed some portion of the source text, which it stores in a buffer. that buffer is then tokenized as usual, until the end of the source buffer is reached. at any point, more source text can be fed into the lexer. when this happens, the buffer is shifted so that only the non-processed source is kept, with the new source text being appended onto that. this way, the memory footprint can be kept low, even for large amounts of source text. the demo itself works with pipes - it reads input text from stdin and outputs each token onto its own line with the token type, lexeme length, and lexeme. it currently lexes any string of non-(ascii-whitespace) characters as one token. $ wc -w readme 281 readme $ cat readme | ./lexi | wc -l 281 -- performance ----------------------------------------------------------------- lexi consistently uses less than 10KiB for large amounts of random data. this has been tested on 1M, 100M, 1G, and 5G piped from /dev/urandom, as well as on `bible.txt` and `world192.txt` from the canterbury corpus and the large corpus from https://corpus.canterbury.ac.nz/descriptions. $ cat bible.txt | /usr/bin/time -f "%M" ./lexi > /dev/null 2536 $ cat world192.txt | /usr/bin/time -f "%M" ./lexi > /dev/null 2592 $ cat /dev/urandom | head -c 5G | /usr/bin/time -f "%M" ./lexi > /dev/null 2496 the results can vary depending on what is and how it is measured but in general, lexi never seems to cross 1MiB. the point is, lexi needs *far less* than 5G of memory to lex 5G of data.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.