Coder Social home page Coder Social logo

wjn1996 / awesomemrc Goto Github PK

View Code? Open in Web Editor NEW

This project forked from cooelf/awesomemrc

0.0 1.0 0.0 2.39 MB

This repo is our research summary and playground for MRC. More features are coming.

Home Page: https://arxiv.org/abs/2005.06249

Python 99.93% Shell 0.07%

awesomemrc's Introduction

AwesomeMRC

*working in progress

This repo is our research summary and playground for MRC. More features are coming.

Summary

Looking for a comprehensive and comparative review of MRC? check out our new survey paper: Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond (preprint, 2020).

In this work, MRC model is regarded as a two-stage Encoder-Decoder architecture. Our empirical analysis is shared in this repo.

Encoder:

  1. Language Units

    Subword-augmented Embedding for Cloze Reading Comprehension (COLING 2018)

    Effective Subword Segmentation for Text Comprehension (TASLP)

  2. Linguistic Knowledge

    Semantics-aware BERT for language understanding (AAAI 2020)

    SG-Net: Syntax-Guided Machine Reading Comprehension (AAAI 2020)

    LIMIT-BERT: Linguistic Informed Multi-Task BERT (preprint)

  3. Commonsense Injection

    Multi-choice Dialogue-Based Reading Comprehension with Knowledge and Key Turns (preprint)

  4. Contextualized language models (CLMs) for MRC:

Decoder:

The implementation is based on Transformers v2.3.0.

As part of the techniques in our Retro-Reader paper:

Retrospective Reader for Machine Reading Comprehension (preprint)

Answer Verification

1) Multitask-style verification

We evaluate different loss functions

cross-entropy (run_squad_av.py)

binary cross-entropy (run_squad_av_bce.py)

mse regression (run_squad_avreg.py)

2) External verification

Train an external verifier (run_cls.py)

Matching Network

Cross Attention (run_squad_seq_trm.py)

Matching Attention (run_squad_seq_sc.py)

Related Work:

Modeling Multi-turn Conversation with Deep Utterance Aggregation (COLING 2018)

DCMN+: Dual Co-Matching Network for Multi-choice Reading Comprehension (AAAI 2020)

Answer Dependency

Model answer dependency (start + seq -> end) (run_squad_dep.py)

Example: Retrospective Reader

  1. train a sketchy reader (sh_albert_cls.sh)

  2. train an intensive reader (sh_albert_av.sh)

  3. rear verification: merge the prediction for final answer (run_verifier.py)

SQuAD 2.0 Dev Results:	

  ```
  {
  "exact": 87.75372694348522, 
  "f1": 90.91630165754992, 
  "total": 11873, 
  "HasAns_exact": 83.1140350877193, 
  "HasAns_f1": 89.4482539777485, 
  "HasAns_total": 5928, 
  "NoAns_exact": 92.38015138772077, 
  "NoAns_f1": 92.38015138772077, 
  "NoAns_total": 5945
  }
  ```

Question Classification

One-shot Learning for Question-Answering in Gaokao History Challenge (COLING 2018)

Citation

@article{zhang2020retrospective,
  title={Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond},
  author={Zhang, Zhuosheng and Zhao, Hai and Wang, Rui},
  journal={arXiv preprint arXiv:2005.06249},
  year={2020}
}

@article{zhang2020retrospective,
  title={Retrospective reader for machine reading comprehension},
  author={Zhang, Zhuosheng and Yang, Junjie and Zhao, Hai},
  journal={arXiv preprint arXiv:2001.09694},
  year={2020}
}

Related Records (best)

CMRC 2017: The best single model (2017).

SQuAD 2.0: The best among all submissions (both single and ensemble settings); The first to surpass human benchmark on both EM and F1 scores with a single model (2019).

SNLI: The best among all submissions (2019-2020).

RACE: The best among all submissions (2019).

GLUE: The 3rd best among all submissions (early 2019).

Contact

Feel free to email zhangzs [at] sjtu.edu.cn if you have any questions.

awesomemrc's People

Contributors

cooelf avatar dependabot[bot] avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.