fredwu / simple_bayes

A Naive Bayes machine learning implementation in Elixir.

Simple Bayes

A Naive Bayes machine learning implementation in Elixir.

In machine learning, naive Bayes classifiers are a family of simple probabilistic classifiers based on applying Bayes' theorem with strong (naive) independence assumptions between the features.

Naive Bayes has been studied extensively since the 1950s. It was introduced under a different name into the text retrieval community in the early 1960s, and remains a popular (baseline) method for text categorization, the problem of judging documents as belonging to one category or the other (such as spam or legitimate, sports or politics, etc.) with word frequencies as the features. With appropriate preprocessing, it is competitive in this domain with more advanced methods including support vector machines. It also finds application in automatic medical diagnosis.

Naive Bayes classifiers are highly scalable, requiring a number of parameters linear in the number of variables (features/predictors) in a learning problem. Maximum-likelihood training can be done by evaluating a closed-form expression, which takes linear time, rather than by expensive iterative approximation as used for many other types of classifiers. - Wikipedia

Features

• Naive Bayes algorithm with different models
  • Multinomial
  • Binarized (boolean) multinomial
  • Bernoulli
• Multiple storage options
  • In-memory (default)
  • File system
  • Dets (Disk-based Erlang Term Storage)
• Ignores stop words
• Additive smoothing
• TF-IDF
• Optional keywords weighting
• Optional word stemming via Stemmer

Feature Matrix

	Multinomial	Binarized multinomial	Bernoulli
Stop words	✅	✅	✅
Additive smoothing	✅	✅
TF-IDF	✅
Keywords weighting	✅
Stemming	✅	✅	✅

Usage

Install by adding :simple_bayes and optionally :stemmer (for the default stemming functionality) to deps in your mix.exs:

defp deps do
  [
    {:simple_bayes, "~> 0.12"},
    {:stemmer,      "~> 1.0"}
  ]
end

If you're on Elixir 1.3 or below, ensure :simple_bayes and optionally :stemmer are started before your application:

def application do
  [applications: [:logger, :simple_bayes, :stemmer]]
end

bayes = SimpleBayes.init()
        |> SimpleBayes.train(:apple, "red sweet")
        |> SimpleBayes.train(:apple, "green", weight: 0.5)
        |> SimpleBayes.train(:apple, "round", weight: 2)
        |> SimpleBayes.train(:banana, "sweet")
        |> SimpleBayes.train(:banana, "green", weight: 0.5)
        |> SimpleBayes.train(:banana, "yellow long", weight: 2)
        |> SimpleBayes.train(:orange, "red")
        |> SimpleBayes.train(:orange, "yellow sweet", weight: 0.5)
        |> SimpleBayes.train(:orange, "round", weight: 2)

bayes |> SimpleBayes.classify_one("Maybe green maybe red but definitely round and sweet.")
# => :apple

bayes |> SimpleBayes.classify("Maybe green maybe red but definitely round and sweet.")
# => [
#   apple:  0.18519202529366116,
#   orange: 0.14447781772131096,
#   banana: 0.10123406763124557
# ]

bayes |> SimpleBayes.classify("Maybe green maybe red but definitely round and sweet.", top: 2)
# => [
#   apple:  0.18519202529366116,
#   orange: 0.14447781772131096,
# ]

With and without word stemming (requires a stem function, we recommend Stemmer):

SimpleBayes.init()
|> SimpleBayes.train(:apple, "buying apple")
|> SimpleBayes.train(:banana, "buy banana")
|> SimpleBayes.classify("buy apple")
# => [
#   banana: 0.05719389206673358,
#   apple: 0.05719389206673358
# ]

SimpleBayes.init(stem: &Stemmer.stem/1) # or any other stemming function
|> SimpleBayes.train(:apple, "buying apple")
|> SimpleBayes.train(:banana, "buy banana")
|> SimpleBayes.classify("buy apple")
# => [
#   apple: 0.18096114003107086,
#   banana: 0.15054767928902865
# ]

Configuration (Optional)

For application wide configuration, in your application's config/config.exs:

config :simple_bayes, model: :multinomial
config :simple_bayes, storage: :memory
config :simple_bayes, default_weight: 1
config :simple_bayes, smoothing: 0
config :simple_bayes, stem: false
config :simple_bayes, top: nil
config :simple_bayes, stop_words: ~w(
  a about above after again against all am an and any are aren't as at be
  because been before being below between both but by can't cannot could
  couldn't did didn't do does doesn't doing don't down during each few for from
  further had hadn't has hasn't have haven't having he he'd he'll he's her here
  here's hers herself him himself his how how's i i'd i'll i'm i've if in into
  is isn't it it's its itself let's me more most mustn't my myself no nor not of
  off on once only or other ought our ours ourselves out over own same shan't
  she she'd she'll she's should shouldn't so some such than that that's the
  their theirs them themselves then there there's these they they'd they'll
  they're they've this those through to too under until up very was wasn't we
  we'd we'll we're we've were weren't what what's when when's where where's
  which while who who's whom why why's with won't would wouldn't you you'd
  you'll you're you've your yours yourself yourselves
)

Alternatively, you may pass in the configuration options when you initialise:

SimpleBayes.init(
  model:          :multinomial,
  storage:        :memory,
  default_weight: 1,
  smoothing:      0,
  stem:           false,
  top:            nil,
  stop_words:     []
)

Available options for :model are:

• :multinomial (default)
• :binarized_multinomial
• :bernoulli

Available options for :storage are:

• :memory (default, can also be used by any database, see below for more details)
• :file_system
• :dets

Storage options have extra configurations:

Memory

• :namespace - optional, it's only useful when you want to load by the namespace

File System

• :file_path

Dets

• :file_path

File System vs Dets

File system encodes and decodes data using base64, whereas Dets is a native Erlang library. Performance wise file system with base64 tends to be faster with less data, and Dets faster with more data. YMMV, please do your own comparison.

Configuration Examples

# application-wide configuration:
config :simple_bayes, storage: :file_system
config :simple_bayes, file_path: "path/to/the/file.txt"

# per-initialization configuration:
SimpleBayes.init(
  storage: :file_system,
  file_path: "path/to/the/file.txt"
)

Storage Usage

opts = [
  storage:   :file_system,
  file_path: "test/temp/file_sysmte_test.txt"
]

SimpleBayes.init(opts)
|> SimpleBayes.train(:apple, "red sweet")
|> SimpleBayes.train(:apple, "green", weight: 0.5)
|> SimpleBayes.train(:apple, "round", weight: 2)
|> SimpleBayes.train(:banana, "sweet")
|> SimpleBayes.save()

SimpleBayes.load(opts)
|> SimpleBayes.train(:banana, "green", weight: 0.5)
|> SimpleBayes.train(:banana, "yellow long", weight: 2)
|> SimpleBayes.train(:orange, "red")
|> SimpleBayes.train(:orange, "yellow sweet", weight: 0.5)
|> SimpleBayes.train(:orange, "round", weight: 2)
|> SimpleBayes.save()

SimpleBayes.load(opts)
|> SimpleBayes.classify("Maybe green maybe red but definitely round and sweet")

In-memory save/2, load/1 and the encoded_data option

Calling SimpleBayes.save/2 is unnecessary for :memory storage. However, when using the in-memory storage, you are able to get the encoded data - this is useful if you would like to store the encoded data in your persistence of choice. For example:

{:ok, _pid, encoded_data} = SimpleBayes.init()
|> SimpleBayes.train(:apple, "red sweet")
|> SimpleBayes.train(:apple, "green", weight: 0.5)
|> SimpleBayes.train(:apple, "round", weight: 2)
|> SimpleBayes.train(:banana, "sweet")
|> SimpleBayes.save()

# now store `encoded_data` in your database of choice
# once `encoded_data` is fetched again from the database, you are then able to:

SimpleBayes.load(encoded_data: encoded_data)
|> SimpleBayes.train(:banana, "green", weight: 0.5)
|> SimpleBayes.train(:banana, "yellow long", weight: 2)
|> SimpleBayes.train(:orange, "red")
|> SimpleBayes.train(:orange, "yellow sweet", weight: 0.5)
|> SimpleBayes.train(:orange, "round", weight: 2)
|> SimpleBayes.classify("Maybe green maybe red but definitely round and sweet")

Changelog

Please see CHANGELOG.md.

License

Licensed under MIT.

simple_bayes's Issues

Remove classify_one in favour of an option

This is minor but it seems as though allowing classify_one or classify is unnecessary. There could just be an option of :top (or something better) to specify how many to classify:

bayes |> SimpleBayes.classify("Maybe green maybe red but definitely round and sweet.", top: 3) # classify 3

bayes |> SimpleBayes.classify("Maybe green maybe red but definitely round and sweet.") # classify all

bayes |> SimpleBayes.classify("Maybe green maybe red but definitely round and sweet.", top: 1) # classify 1, rather than a separate function

Seems that being generic to the number to return would be handy, and trim down any special casing.

Additional Bayes algorithms

First, many thanks for your great Bayes library!

As an enhancement and to handle different use cases, it would be great to have the ability to select the Bayes algorithm to use.

I would suggest these two additional algorithms:

• Binarized Multinomial Naive Bayes (The Binarized Multinomial Naive Bayes can be used when the frequencies of the words don’t play a key role in a classification, e.g. sentiment analysis)
• Bernoulli Naive Bayes (The Bernoulli Naive Bayes can be used when the absence of a particular word matters, e.g. spam or adult content filtering)

What do you think?

Universal save/load function

In order to save the bayes filter e.g. in a database/key-value store I think an universal save/load function would be great.

My suggestion is to adapt the storage behavior so that the save function returns a {:ok, pid, data} tuple. In case of the filesystem storage, which doesn't need to return data, the returned tuple is {:ok, pid, nil}.

The load function than needs to accepts the encoded data.

What do you think?

Improved encoding/decoding?

Heya!

Loving this library, except I've encountered an unfortunate pathological case.

I have trained a classifier on roughly 288,000 labeled texts. Cardinality of labels is 5, and length of the text ~5 words.

Here's how it's configured:

model:          :bernoulli,
storage:        :file_system,
file_path:      @storage,
default_weight: 1.0,
smoothing:      0.0,
stem:           false,
stop_words:     []

I then persisted this trained model to disk. It's roughly 11Mb when stored. Then I attempted to load it back into memory...

After 10 (!) minutes I received this error:

** (ArithmeticError) bad argument in arithmetic expression
    (stdlib) :math.pow(7318, 24158)
             lib/simple_bayes/classifier/models/bernoulli.ex:7: SimpleBayes.Classifier.Model.Bernoulli.probability_of/3
             lib/simple_bayes/classifier/probability.ex:37: anonymous fn/4 in SimpleBayes.Classifier.Probability.for_collection/3
    (elixir) lib/map.ex:114: Map.do_new_transform/3
             lib/simple_bayes/classifier.ex:13: SimpleBayes.Classifier.classify/3

Ouch.

So, there's two things here:

1. There's an issue prevent me from reloading the trained classifier.
2. Loading is painfully slow.

When I have the time, I'll explore the reasons behind (1). For now, I'll see how far dets can get me.