acalejos / mighty

Natural Language Processing in Elixir

Recent Blog Posts

1. The Best Way to Use FFmpeg - 🏷️ Tags: Elixir, Productivity, Machine Learning

FFmpeg is one of the most powerful tools out there, but it can be daunting to use — until now.

2. Resist Vendor Lock-In With Supabase - 🏷️ Tags: Elixir

Not only is Supabase the open-source Firebase alternative, it just might be safer too.

3. Launch Your AI App in 48 Hours - 🏷️ Tags: Elixir, Machine Learning, Productivity

Don't spend 6 months building your app just to find out nobody wants it. Do this instead.

4. Livebook: Elixir's Swiss Army Knife - 🏷️ Tags: Elixir

From learning, to prototyping, to production - Livebook can really do it all.

5. Plotting XGBoost Trees in Elixir - 🏷️ Tags: Elixir, Machine Learning

XGBoost is one of the most popular Machine Learning tools. Here's how to make beautiful plots for it with Elixir.

---

mighty's Issues

Tokenization

The first step in any NLP application is tokenization, so that should be the first issue tackled.

A nice explanation of Spacy's tokenization algorithm can be found at https://spacy.io/usage/linguistic-features#how-tokenizer-works

This is the tokenization algorithm I will be proceeding with for now unless there are compelling reasons to go another direction. This issue will also involve creating all basic container data structures needed to compose the Tokenization pipeline, such as Tokenizer, Vocab, Doc, etc.

For now, I will try to do a fairly faithful reimplementation of Spacy's tokenizer and we can diverge from it as we need to

Mighty Goals and Design Decisions

Goal / End-state: Mighty NLP is a production-grade Natural Language Processing library for Elixir. Mighty is customizable and extensible and provides tools for all aspects of NLP. Bumblebee already provides pre-trained transformers that can perform some of the main tasks typical of an NLP library, so Mighty aims to expand on those models and integrate them as part of a comprehensive pipeline.

The comparable library is Python's Spacy (https://spacy.io/)

To gain a better understanding of Spacy's architecture and pipeline operations, you can refer to their excellent documentation: https://spacy.io/api and https://spacy.io/usage/spacy-101

Some points for discussion:

• How opinionated should the lib be?
  • Spacy comes w/ premade pipeline that you can tune if you choose
• What are the major data structures / structs going to be?
  • Spacy uses Language, Vocab, and Doc as its main containers
• I like the pipeline construct they use. Some parts of the pipeline are rules based, some are statistical (i.e. trained network), some are language dependent, etc.
• Do we want interoperability w/ Bumblebee?
  • I think bumblebee could "power" certain components of the pipeline
• Where do we get language data for all languages we want to support?
  • We can just use much of Spacy's open source rules for languages from https://github.com/explosion/spaCy/tree/master/spacy/lang