mmourafiq / philo2vec

An implementation of word2vec applied to [stanford philosophy encyclopedia](http://plato.stanford.edu/)

philo2vec

A Tensorflow implementation of word2vec applied to stanford philosophy encyclopedia, the implementation supports both cbow and skip gram

for more reference, please have a look at this papers:

• Distributed Representations of Words and Phrases and their Compositionality
• word2vec Parameter Learning Explained
• Explained: Deriving Mikolov et al.’s Negative-Sampling Word-Embedding Method

After training the model returns some interesting results, see interesting results part

Evaluating hume - empiricist + rationalist:

descartes
malebranche
spinoza
hobbes
herder

Some interesting results

Similarities

Similar words to death:

untimely
ravages
grief
torment

Similar words to god:

divine
De Providentia
christ
Hesiod

Similar words to love:

friendship
affection
christ
reverence

Similar words to life:

career
live
lifetime
community
society

Similar words to brain:

neurological
senile
nerve
nervous

operations

Evaluating hume - empiricist + rationalist:

descartes
malebranche
spinoza
hobbes
herder

Evaluating ethics - rational:

hiroshima

Evaluating ethic - reason:

inegalitarian
anti-naturalist
austere

Evaluating moral - rational:

commonsense

Evaluating life - death + love:

self-positing
friendship
care
harmony

Evaluating death + choice:

regret
agony
misfortune
impending

Evaluating god + human:

divine
inviolable
yahweh
god-like
man

Evaluating god + religion:

amida
torah
scripture
buddha
sokushinbutsu

Evaluating politic + moral:

rights-oriented
normative
ethics
integrity

The repo contains:

• an object to crawl data from the philosophy encyclopedia; PlatoData
• a object to build the vocabulary based on the crawled data; VocabBuilder
• the model that computes the continuous distributed representations of words; Philo2Vec

Installation

The dependencies used for this module can be easily installed with pip:

> pip install -r requirements.txt

The params for the VocabBuilder:

• min_frequency: the minimum frequency of the words to be used in the model.
• size: the size of the data, the model then use the top size most frequenct words.

The hyperparams of the model:

• optimizer: an instance of tensorflow Optimizer, such as GradientDescentOptimizer, AdagradOptimizer, or MomentumOptimizer.
• model: the model to use to create the vectorized representation, possible values: CBOW, SKIP_GRAM.
• loss_fct: the loss function used to calculate the error, possible values: SOFTMAX, NCE.
• embedding_size: dimensionality of word embeddings.
• neg_sample_size: number of negative samples for each positive sample
• num_skips: numer of skips for a SKIP_GRAM model.
• context_window: window size, this window is used to create the context for calculating the vector representations [ window target window ].

Quick usage:

params = {
    'model': Philo2Vec.CBOW,
    'loss_fct': Philo2Vec.NCE,
    'context_window': 5,
}
x_train = get_data()
validation_words = ['kant', 'descartes', 'human', 'natural']
x_validation = [StemmingLookup.stem(w) for w in validation_words]
vb = VocabBuilder(x_train, min_frequency=5)
pv = Philo2Vec(vb, **params)
pv.fit(epochs=30, validation_data=x_validation)

params = {
    'model': Philo2Vec.SKIP_GRAM,
    'loss_fct': Philo2Vec.SOFTMAX,
    'context_window': 2,
    'num_skips': 4,
    'neg_sample_size': 2,
}
x_train = get_data()
validation_words = ['kant', 'descartes', 'human', 'natural']
x_validation = [StemmingLookup.stem(w) for w in validation_words]
vb = VocabBuilder(x_train, min_frequency=5)
pv = Philo2Vec(vb, **params)
pv.fit(epochs=30, validation_data=x_validation)

about stemming

Since the words are stemmed as part of the preprocessing, some operation are sometimes necessary

StemmingLookup.stem('religious')  # returns "religi"

StemmingLookup.original_form('religi')  # returns "religion"

Getting similarities

pv.get_similar_words(['rationalist', 'empirist'])

Evaluating operations

pv.evaluate_operation('moral - rational')

plotting vectorized words

pv.plot(['hume', 'empiricist', 'descart', 'rationalist'])

Training details

skip_gram:

cbow:

philo2vec's Issues

Any plans for an update?

Hi
So i was trying to update your repository to be compatible with python 3.7. I got the crawler and vocab builder working but i have no idea of the tensorflow implementation of word2vec. Specifically it seems that it's the requirement for tensorflow version 0.9.0 that causes trouble, as this version isn't available any longer and current versions aren't compatible. Any chance you're planning to update the repository? Or if somehow there is a way to get it working?

Sorry if my questions seems weirdly formulated, still learning to program.