mitjanikolaus / machine-analysis

library to do analysis on machine

machine-analysis

This project was conducted as a Project AI for the Artificial Intelligence Master's programme of the Universiteit van Amsterdam in 2018. The goal was to analyze the activations of the Attentive Guidance model proposed by Hupkes et al. (2018) and recover differences with respect to the baseline model, which could help understand the differences and bias future models in a way that produces the same results, but renders the training procedure from the above paper unnecessary.

Models were trained on the table lookup task and stored in a model zoo. The original models, coming from here, were followingly augmented in order to allow the extraction of activation values easily.

Results

(Summary)

Our experiments highlighted crucial differences in the way neurons in the AG model respond to inputs for both the encoder and decoder in comparison to the baseline. We showed that neurons in the AG model display a very disparate distribution in activation values. Moreover, these values changed more between time steps, indicating a higher sensitivity to inputs that the baseline does not seem to possess. The fact that the gates of the AG model were more often saturated than the baseline ones is in accordance with this hypothesis. Further results pointed into the same direction, as we were able to identify groups of neurons that respond to specific input tokens that are smaller than for the baseline, possibly indicating a higher degree of specialization. That this behaviour does indeed lead the AG model to construct compositional solutions. We did not manage to say with certainty whether the AG model is better at encoding more general information about the input, like the length of the sequence.

Requirements

All required python packages can be installed by running

pip3 install -r requirements.txt

For PyTorch, a manual installation might be necessary. See the official site for more information.

Modules

This repository contains the following modules:

Module name	Content	Used for report sections
activations.py	Store model activations in a special data set class	All
baseline_guided_classification.py	Train a classifier to preditct model type (baseline vs. guided) from encoder outputs	4
count_model_inspection.py	Ablation study for predicting timesteps using perceptron	4.3
count_prediction.py	Ablation study for predicting timesteps using perceptron	4.3
distributions.py	Quantify the distributions of activation values and the change between time steps	4.1, Appendix C
functional_groups.py	Learn a diagnostic classifier in order to learn functional groups	4.5
get_hidden_activations.py	Extract the activations produced by an model and store them in a special data set	All
inspect_gate_activations.py	Produce gate saturation plots	4.2
plots.py	Generate plots for ablation study with perceptron	4.3
similar_activations.py	Show how similar activations for similar / dissimilar samples are	4.4, Appendix A
visualization.py	Create a variety of plots.	4.1, Appendix C
models.analysable_cells.py	Create GRU and LSTM that allow the retrieval of their activations	All
models.analysable_decoder.py	Create an open machine-task decoder	All
models.analysable_encoder.py	Create an open machine-task encoder	All
models.analysable_seq2seq.py	Create a machine-task model that stores all activations	All