SentEval is a library for evaluating the quality of sentence embeddings. We assess their generalization power by using them as features on a broad and diverse set of "transfer" tasks (more details here). Our goal is to ease the study and the development of general-purpose fixed-size sentence representations.
news: Added STS2012, STS2013 (-SMT), STS2014, STS2015 and STS2016.
This code is written in python. The dependencies are:
- Python 2.7 (with recent versions of NumPy/SciPy)
- Pytorch >= 0.12
- scikit-learn>=0.18.0
SentEval allows you to evaluate your sentence embeddings as features for the following tasks:
- Binary classification: MR (movie review), CR (product review), SUBJ (subjectivity status), MPQA (opinion-polarity), SST (Stanford sentiment analysis)
- Multi-class classification: TREC (question-type classification), SST (fine-grained Stanford sentiment analysis)
- Entailment (NLI): SNLI (caption-based NLI), MultiNLI (Multi-genre NLI), SICK (Sentences Involving Compositional Knowledge, entailment)
- Semantic Textual Similarity: STS12, STS13 (-SMT), STS14, STS15, STS16
- Semantic Relatedness: STSBenchmark, SICK
- Paraphrase detection: MRPC (Microsoft Research Paraphrase Corpus)
- Caption-Image retrieval: COCO dataset (with ResNet-101 2048d image embeddings)
To get all the transfer tasks datasets, run (in data/):
./get_transfer_data.bash
This will automatically download and preprocess the datasets, and put them in data/senteval_data.
WARNING: Extracting the MRPC MSI file requires the "cabextract" command line (i.e apt-get/yum install cabextract).
In examples/bow.py, we evaluate the quality of the average(GloVe) embeddings.
To get GloVe embeddings [2GB], run (in examples/):
./get_glove.bash
To reproduce the results for avg(GloVe) vectors, run (in examples/):
python bow.py
As required by SentEval, this script implements two functions: prepare (optional) and batcher (required) that turn text sentences into sentence embeddings. Then SentEval takes care of the evaluation on the transfer tasks using the embeddings as features.
To get the InferSent model and reproduce our results, download our best models and run infersent.py (in examples/):
curl -Lo examples/infersent.allnli.pickle https://s3.amazonaws.com/senteval/infersent/infersent.allnli.pickle
curl -Lo examples/infersent.snli.pickle https://s3.amazonaws.com/senteval/infersent/infersent.snli.pickle
To evaluate your own sentence embedding method, you will need to implement two functions:
- prepare (sees the whole dataset of each task and can thus construct the word vocabulary, the dictionary of word vectors etc)
- batcher (transforms a batch of text sentences into sentence embeddings)
batcher only sees one batch at a time while the samples argument of prepare contains all the sentences of a task.
prepare(params, samples)
- batch: numpy array of text sentences
- params: senteval parameters (note that "prepare" outputs are stored in params).
- output: None. Any "output" computed in this function is stored in "params" and can be further used by batcher.
Example: in bow.py, prepare is is used to build the vocabulary of words and construct the "params.word_vect* dictionary of word vectors.
batcher(batch, params)
- batch: numpy array of text sentences (of size params.batch_size)
- params: senteval parameters (note that "prepare" outputs are stored in params).
- output: numpy array of sentence embeddings (of size params.batch_size)
Example: in bow.py, batcher is used to compute the mean of the word vectors for each sentence in the batch using params.word_vec. Use your own encoder in that function to encode sentences.
After having implemented the batch and prepare function for your own sentence encoder,
- to perform the actual evaluation, first import senteval and define a SentEval object:
import senteval
se = senteval.SentEval(batcher, prepare, params_senteval)
(to import senteval, you can either add senteval path to your pythonpath or use sys.path.insert)
- define the set of transfer tasks on which you want SentEval to perform evaluation and run the evaluation:
transfer_tasks = ['MR', 'SICKEntailment', 'STS14', 'STSBenchmark']
results = se.eval(transfer_tasks)
The current list of available tasks is:
['CR', 'MR', 'MPQA', 'SUBJ', 'SST', 'TREC', 'MRPC', 'SNLI',
'SICKEntailment', 'SICKRelatedness', 'STSBenchmark', 'ImageCaptionRetrieval',
'STS12', 'STS13', 'STS14', 'STS15', 'STS16']
SentEval has several parameters (only task_path is required):
- task_path (str): path to data, generated by data/get_transfer_data.py
- seed (int): random seed for reproducability (default: 1111)
- usepytorch (bool): use pytorch or scikit learn (when possible) for logistic regression (default: True). Note that sklearn is quite fast for small dimensions. Use pytorch for SNLI.
- classifier (str): if usepytorch, choose between 'LogReg' and 'MLP' (tanh) (default: 'LogReg')
- nhid (int): if usepytorch and classifier=='MLP' choose nb hidden units (default: 0)
- batch_size (int): size of minibatch of text sentences provided to "batcher" (sentences are sorted by length). Note that this is not the batch_size used by pytorch logistic regression, which is fixed.
- kfold (int): k in the kfold-validation. Set to 10 to be comparable to published results (default: 5)
- ... and any parameter you want to have access to in "batcher" or "prepare" functions.
Please cite 1 if using this code for evaluating sentence embedding methods.
[1] A. Conneau, D. Kiela, H. Schwenk, L. Barrault, A. Bordes, Supervised Learning of Universal Sentence Representations from Natural Language Inference Data
@article{conneau2017supervised,
title={Supervised Learning of Universal Sentence Representations from Natural Language Inference Data},
author={Conneau, Alexis and Kiela, Douwe and Schwenk, Holger and Barrault, Loic and Bordes, Antoine},
journal={arXiv preprint arXiv:1705.02364},
year={2017}
}
Contact: [email protected], [email protected]