kimjeyoung / sngp-bert-pytorch

This is reimplementation of "Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness" in Pytorch.

🔥 SNGP-BERT (Unofficial) 🔥

This is reimplementation of "Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness" in Pytorch.

The codes are based on official repo (Tensorflow) and huggingface.

Original Paper : Link

Installation ☕

Training environment : Ubuntu 18.04, python 3.6

pip3 install torch torchvision torchaudio
pip install scikit-learn

Download bert-base-uncased checkpoint from hugginface-ckpt
Download bert-base-uncased vocab file from hugginface-vocab
Download CLINC OOS intent detection benchmark dataset from tensorflow-dataset

The downloaded files' directory should be:

SNGP-BERT
ㄴckpt
  ㄴbert-base-uncased-pytorch_model.bin
ㄴdataset
  ㄴclinc_oos
    ㄴtrain.csv
    ㄴval.csv
    ㄴtest.csv
    ㄴtest_ood.csv
  ㄴvocab
    ㄴbert-base-uncased-vocab.txt
ㄴmodels
...

Dataset Info 📖

In their paper, the authors conducted OOD experiment for NLP using CLINC OOS intent detection benchmark dataset, the OOS dataset contains data for 150 in-domain services with 150 training sentences in each domain, and also 1500 natural out-of-domain utterances. You can download the dataset at Link.

Original dataset paper, and Github : Paper Link, Git Link

Run 🌟

Train

python main.py --train_or_test train --method sngp --device gpu --gpu 0

Test

python main.py --train_or_test test --method sngp --device gpu --gpu 0

Results ✨

Results for SNGP-BERT on CLINC OOS.
NOTE : Depending on the random seed, the result may be slightly different.

Version	ACC	AUROC	AUPRC
Paper (Tensorflow)	96.6	0.969	0.880
Pytorch (batch size = 256)	96.1	0.974	0.900
Pytorch (batch size = 64)	95.9	0.972	0.894

References

[1] https://github.com/google/uncertainty-baselines/blob/main/baselines/clinc_intent/sngp.py
[2] https://huggingface.co/
[3] https://github.com/google/edward2

sngp-bert-pytorch's Issues

Non-symmetrical gp_cov_matrix returned

Hi,

Thank you for all the work on porting this to PyTorch. I'm running into a bit of an issue however. When using this for BERT for span-extractive question answering I am getting a covariance matrix that is not symmetrical, and thus not a valid covariance matrix.

Any ideas what is happening or if I am forgetting something?

Thanks!

License of this repository

Hi, I have a simple question. What is the license of this repository?
I can't find any information about that.