lukanovak93 / python-data-science

List of useful Python packages for data science

python-data-science

List of useful Python packages for data science:

PyData stack

• numpy
• scipy
• pandas
• jupyter
• statsmodels

Profiling

• pandas-profiling
• memory_profiler
• py-spy
• pyflame ❗ (Does not support Windows)

Forecasting

• pyramid-arima
• fbprophet - time series forecasting (additive model) which performs best with high frequency data
• pyflux - time series library

Niche stats libraries

• lifelines - survival analysis
• convoys

"Large Data" libraries

• dask
• pyarrow - Python bindings for Apache Arrow
• fastparquet
• vaex
• modin - pandas on Ray execution framework
• dampr
• datatable - pandas equivalent for performing big data (up to 100GB) operations on a single-node machine, at the maximum speed possible

visualization libraries

• PyViz stack (part of PyData stack)
• Panel - dashboarding from PyData Stack
• Plotly
• Dash - dashboard library from plotly
• matplotlib
• seaborn
• altair
• pdvega
• chartify
• dataspyre - dashboard framework with flask backend
• folium
• geoplot
• plotnine - clone of R's ggplot2
• joypy
• bqplot
• jmpy
• pyqtgraph
• probscale - probability scaled axis
• adjustText - add non-overlapping annotated text

Jupyter Notebook Related

• ipysheet
• ipypivot
• papermill + scrapbook - parameterized notebooks
• nteract-scapbook - for passing data between notebooks
• jupytext - edit notebooks as text files

database related

• pyodbc
• turbodbc
• ipython-sql
• sqlalchemy
• sqlalchemy-turbodbc

ETL or data engineering related sorted from lightest to heaviest framework

• papermill + scrapbook - parameterized notebooks
• dequindre
• petl
• bonobo
• pypeln
• botflow
• mara data integration
• dbt
• Luigi from Spotify - works with Windows
• Apache Airflow - Windows not supported

Data validation and cleaning frameworks

• cerberus
• great_expectations
• pandera
• pyjanitor
• schema
• PandasSchema
• engarde

Machine Learning Related - mostly for tabular data or non-NN

• scikit-learn
• sklearn-pandas
• imbalanced-learn
• hyperopt-sklearn - not pip installable yet
• tpot
• xgboost
• lightgbm
• DMTK - Microsoft Distributed Machine Learning Toolkit
• fastText

Deep learning

• Netron - visualize NNs
• BlingFire -Microsoft Bing team, a tokenizer designed for fast-speed and quality tokenization of Natural Language text

Webscraping

• beautifulsoup4
• mechanicalsoup
• selenium - headless-browser framework for extracting javascript data
• scrapy - OOP framework
• newspaper3k - easily extract text from articles
• requests-html

Utilities

• tldr-python-client - replacement for man pages
• bropages](http://bropages.org/) - sudo apt-get install ruby-dev, sudo gem install bropages
• howdoi
• inspect
• prettypandas
• fuzzywuzzy
• fuzzymatcher
• pytest
• requests
• psutil

CLI

• click - for making CLI
• fire - for making CLI
• questionary

Progress Bars

• tqdm
• fastprogress

Misc.

• glances - CPU/memory monitoring
• pendulum - a better datetime library, better than arrow
• visidata - free, open-source tool that lets you quickly open, explore, summarize, and analyze datasets in your computer’s terminal
• schedule - job scheduling for humans
• pyautogui
• ptpython - better REPL
• xlwings - Excel VBA, but with Python instead
• openpyxl
• scriptedforms
• black - source code formatter

debugging

• ipdb
• pudb - Windows not supported, need cygwin

logging

• loguru
• PySnooper