chaklam-silpasuwanchai / python-fo-natural-language-processing

This is the repository for the course Natural Language Processing at Asian Institute of Technology. Covers word vectors, spaCy, PyTorch, HuggingFace.

Natural Language Processing

This is the repository for the Natural Language Processing course at the Asian Institute of Technology.

Google slide lectures can be found: https://drive.google.com/drive/folders/14x9_-Y_aWysPIZFLaVrZE2ngy2h_beJj?usp=sharing

YT playlist: https://www.youtube.com/playlist?list=PLqL-7eLmqd9V3faivSAST76YQClS44dSz

Credits

I would also like to give huge credits to several GitHub / web resources that I have revised to create this:

• https://web.stanford.edu/class/cs224n/
• http://spacy.pythonhumanities.com/
• http://courses.spacy.com
• https://github.com/kushalj001/pytorch-question-answering
• https://github.com/DSKSD/DeepNLP-models-Pytorch
• https://github.com/bentrevett
• https://github.com/graykode/nlp-tutorial
• https://huggingface.co/course
• https://kikaben.com
• https://github.com/kipgparker/soft-prompt-tuning
• https://github.com/moein-shariatnia/OpenAI-CLIP

Useful GitHub

• https://github.com/keon/awesome-nlp (collection of all NLP learning resources)
• https://github.com/sebastianruder/NLP-progress (omg...this is like a mini Wikipedia for NLP!)
• https://github.com/mhagiwara/100-nlp-papers (listed first 100 influential NLP papers)
• https://captum.ai/tutorials/ (good for working on explainable AI)

I would also like to thank students who have contributed:

• Amanda Raj Shrestha; Email: [email protected]
• Pranisaa Charnparttarvanit; Email: [email protected]
• Chanapa Pananookooln; Email: [email protected]
• Todsavad Tangtortan; Email: [email protected]

Outline

1. Fundamentals

   • Word Vectors - Word2Vec (Naive)
   • Word Vectors - Word2Vec (Negative sampling)
   • Word Vectors - GloVe
   • Window-Based Name Entity Recognition
   • Dependency Parsing
   • Information Retrieval / Salient Spans

2. DL

   • Classification
   • Sequence-to-Sequence Models
     • LSTM, biGRU, CNN, Transformer
   • Langauge Models
     • LSTM, Transformer
   • Masked Language Models
     • BERT
     • Much more...
   • Case studies
     • QA
     • Summarization
     • Pruning
     • distilBERT
     • SentenceBERT
     • SimCSE
     • Much more...

3. SpaCy

4. Huggingface

   • Case studies

5. Retrieval Augmented Generation

   • Prompt, Chain, Tools, Agent

6. Multimodal Language Model

   • ViT, BEIT, CLIP, SimVLM, Flamingo, BLIP-2, CoCa

7. Reinforcement Learning with Human Feedback

   • SFT, PPO, DPO, RRHF

python-fo-natural-language-processing's Issues

Possible typo and error on a2 word2vec/Word2Vec.ipynb

1. The first term should be
   -u_o (1 - sigmoid(u_o.T v_c)) or u_o (sigmoid(u_o.T v_c) - 1)
   instead of
   u_o (1 - sigmoid(u_o.T v_c))

(i'm not very confident, but replacing that allow me to pass the test function)

2. {U} on line 35 and 59 on test_word2vec() function cause an error KeyError: "U" (since {U} is intepreted as variable when using .format() )
   replacing {U} with {{U}} fix the problem for me