zenmoore / noahkit

Code notebook for python package learning.

NoahKit

Code notebook for python package learning.

Checkbar

emoji	connotation	tag
🌕 or ✅	experienced	:full_moon:
🌔 or ✅	advanced	:moon:
🌓 or ✅	well-grounded	:first_quarter_moon:
🌒 or ⬜	understanding	:waxing_crescent_moon:
🌑 or ⬜	strange	:new_moon:

Checklist

ML Framework

integrated machine/deep/reinforcement learning toolkits.

Deep Learning and Neural Network

• 🌓 pytorch : open source deep learning framework commonly used in academia.
• 🌓 tensorflow : open source deep learning framework commonly used in industry.
• 🌑 paddle : baidu open source deep learning platform derived from industry practices.
• 🌑 ncnn : tencent high performance neural network forward computing framework optimized for mobiles.

Reinforcement Learning

• 🌒 gym : a toolkit for developing and comparing reinforcement learning algorithms.
• 🌑 ACME : a research framework for reinforcement learning by DeepMind.

Machine Learning

• 🌓 sklearn : machine learning library.

Methodology

• 🌑 autokeras : automl system based on keras.
• 🌑 pytorch_geometric : geometric deep learning extension library for pytorch.
• 🌓 textbrewer: nlp knowledge distillation toolkit.
• 🌓 OpenPrompt : prompt tuning toolkit developed by Tsinghua.
• 🌑 OpenAttack : an open-source package for texual adversarial attack.
• 🌑 PyCID: a python library for causal influence diagrams.

Applied AI

NLP/CV/etc. packages.

Vision

• 🌔 torchvision : vision processing toolkit in pytorch.
• 🌓 ​cv2 : opencv in python.
• 🌓 ​pillow : python imaging library.

Text

• 🌓 torchtext : natural language data processing for pytorch.

• 🌓 nltk : english natural language toolkit by UPenn.

  for chinese, we should import Stanford NLP Chinese package.

• 🌓 ltp : chinese language technology platform by HIT.

• 🌓 gensim : topic modelling for humans (semantic vector).

• 🌓 glove : toy python implementation of glove.

• 🌓 ​hugging face : sota pre-trained language model toolkit.

  mainly for pre-trained and transformer, etc.

• 🌓 ​fairseq : a seq2seq learning toolkit for pytorch by fair.

• 🌑 EDA : easy data augmentation techniques for text.

• 🌑 pattern : simpler to get started than nltk.

• 🌑 pyenchant : easy access to dictionaries

• 🌒 allennlp : natural language processing toolkit based on pytorch by Allen.

  mainly for ELMo, KnowBERT, etc.

• 🌑 seqeval : evaluation for sequence labelling.

• 🌑 sentence transformers : compute dense vector representations for sentences, paragraphs, and images.

• 🌒 ​(corenlp, spacy, jieba) : chinese processing, language processing, etc.

we put gensim and glove together in this code notebook.

Audio

• 🌑 ​torchaudio : audio processing toolkit in pytorch.

Data Science

data processing and computing.

Processing

• 🌓 pandas : high performance and user-friendly data structure and analysis tool.
• 🌔 matplotlib : comprehensive library for creating static, animated and interactive visualizations in python.

Computing

• 🌓 numpy : scientific computing.
• 🌓 scipy : open-source software for mathematics, science, and engineering.

DevTool

programming aids.

Running Auxiliary

• 🌒 ​tqdm : progress bar.
• 🌑 labml : training visualization on the mobile phone.
• 🌑 wandb: experiment results management.
• 🌑 TensorSensor：visualization of errors about tensor.
• 🌑 optuna: automatic hyperparameter searching.
• 🌑 alfred: visualization of annotations.

Optimization

• 🌓 threading : multithreading.
• 🌓 multiprocessing : multiprocessing.
• 🌓 deepspeed : microsoft training acceleration toolkit.
• 🌑 bminf : a low-resource inference package for large-scale pretrained language models (PLMs).

We put threading and multiprocessing together in this code notebook.

File System

• 🌑 ​glob : file path manager.
• 🌑 ​shutil : advanced file operation module.

Information Retrieval

• 🌑 ​re : regex.
• 🌑 ​elasticsearch : search and analysis engine.
• 🌓 scrapy : web crawler.

Towards User

• 🌑 ​pyqt : GUI.
• 🌑 ​pyinstaller : application packaging.

Operating System

• 🌒 os : access os-related functions.
• 🌑 ​sys : access interpreter-related variables and functions.

Encoding Management

• 🌑 unidecode : because ascii is much easier to deal with
• 🌑 chardet : character encoding detection

Note

• .py file is not runnable : we view it as a codebook composed by numerous fragments, but not an executable file (there is no strong relation between every two fragments).