cmry / omesa

Experiment, Storage and Visualization Framework for Machine Learning research.

👋 Hi

I’m an Assistant Professor at the Department of Cognitive Science & Artificial Intelligence at Tilburg University.

⚗️ Research

I'm interested in the effect of intelligent systems on our lives. Systems that uncover our personal information, monitor and change our behavior, subtly restrict our exposure to information, and treat us unfairly. My dissertation focused on the dual-use of computational stylometry; a field that aims to infer information from writing for good, proving harmfully invasive at the same time. I develop(ed) open-source tools to better understand, and defend against such techniques invading one's privacy.

📫 Contact

See info in the dooblidoo ⬅️.

omesa's Issues

JSON Serialization

Store models in .json rather than .pickle. This is a more transparent format, less prone to be corrupted, and it will allow for a pure JSON backend such as blitzdb.

Class labels not retrievable in Vectorizer (for two-class problem)

I wanted to retrieve the feature names and class labels of a Vectorizer object in order to do some feature analysis. After searching a while, I found that the feature names are availabel in the 'hasher' attribute. However, I was unable to retrieve the class labels. I think they're supposed to be in the 'encoder' attribute, but this seems to be by-passed for binary (but also two-class) problems. Yet, in a two-class problem (e.g. male vs. female) you still need to know which of these is 0 and which is 1.

Classifier Stack + Best Pick through Grid

Currently, the only way to select a classifier in Experiment is through:

Experiment({
    ...
    "classifier": LinearSVC(),
    "params": {"C": np.logspace(...)},
    ...
})

This should be more generic, and allow the ability to add multiple classifiers in the grid search mix, where from the best will be chosen. Something like:

    ...
    "classifiers": {
        LinearSVC(): {"C": ...}
    }
    ...

Major Overhaul

Some big things that are in the works:

• Remove Environment.
• Add Experiment that is controlled with a dict.
• Generalize Dataloader.
• Make generic CSV reader.
• Write model wrapper for storage.

ScikitClassifier in lime_eval.py

ScikitClassifier in latest version of lime.text is no longer available.

You can import make_pipeline instead

from sklearn.pipeline import make_pipeline

and swap the arguments on line 68

self.c = make_pipeline(vectorizer, classifier)

https://github.com/marcotcr/lime/issues/8

Documentation of containers.py - parameter name

In documentation of containers.py, on line 215 a parameter 'text' is described which should be named 'data' cf. line 247 and further

visualizations and storage for experiments

This is a big one, which requires the set-up of a database back-end (done already) and a (web) front-end:

• Dump serialized config files to JSON database.
• Represent the performance in a way that general metrics can be shown.
  • project
  • name
  • training_set (i.a.)
  • testing_set (i.a) -- this and above probably need to be abstracted from loaders
  • string representation of the used features
  • string NAME of the classifier used
  • POS / NEG f1-scores (could be put in a graph) micro f-1
• Able to overview and compare experiments visually.
  • Flat Performance bar.
  • Plotting performance on data proportions.
  • Summary of experiment configurations.
  • Confusion matrices.
  • Aggregate performances in one report.
  • t-SNE?
• Insight into feature importances.
  • LIME evaluation.
  • Coof. representations.

integrate word embeddings as features

Should be doable, something from https://github.com/3Top/word2vec-api and maybe skip thought vectors for the Featurizer.

Breaks through refactor

See differences between old and current. The classify function was broken through old attribute names being used.

Sparse Feature Hashing

Implement feature hashing or at least feature-value mappings through the DictVectorizer.

cross-platform operations

especially in the db and app interactions there are still many directory pointers that will only be compatible with unix systems, these can be changed via os, relevant links here and here.

Parameter max_feats in documentation for NGrams, not implemented

As title says.
Either remove parameter max_feats from documentation of NGrams (in featurizer.py), or implement.

add multi-processing in feature extraction

Should be doable with some rewrites to the data stream to allow for using multiple cores to extract features, like so.

full documentation update

Lot of changes have been made since porting the framework to GitHub. All components have to be rechecked for documentation validity, as this describes the old classes in many cases.