DeepPavlov is an open-source conversational AI library built on TensorFlow, Keras and PyTorch.
DeepPavlov is designed for
- development of production ready chat-bots and complex conversational systems,
- research in the area of NLP and, particularly, of dialog systems.
- Demo demo.deeppavlov.ai
- Documentation docs.deeppavlov.ai
- Model List docs:features/
- Contribution Guide docs:contribution_guide/
- Issues github/issues/
- Forum forum.deeppavlov.ai
- Blogs medium.com/deeppavlov
- Tutorials examples/ and extended colab tutorials
- Docker Hub hub.docker.com/u/deeppavlov/
- Docker Images Documentation docs:docker-images/
Please leave us your feedback on how we can improve the DeepPavlov framework.
Models
Intent/Sentence Classification | Question Answering over Text (SQuAD)
Knowledge Base Question Answering
Sentence Similarity/Ranking | TF-IDF Ranking
Morphological tagging | Syntactic parsing
Entity Linking | Multitask BERT
Skills
Goal(Task)-oriented Bot | Open Domain Questions Answering
Frequently Asked Questions Answering
Embeddings
BERT embeddings for the Russian, Polish, Bulgarian, Czech, and informal English
ELMo embeddings for the Russian language
FastText embeddings for the Russian language
Auto ML
Integrations
-
We support
Linux
platform,Python 3.6
,3.7
,3.8
and3.9
Python 3.5
is not supported!- installation for
Windows
requiresGit
(for example, git) andVisual Studio 2015/2017
withC++
build tools installed!
-
Create and activate a virtual environment:
Linux
python -m venv env source ./env/bin/activate
Windows
python -m venv env .\env\Scripts\activate.bat
-
Install the package inside the environment:
pip install deeppavlov
There is a bunch of great pre-trained NLP models in DeepPavlov. Each model is determined by its config file.
List of models is available on
the doc page in
the deeppavlov.configs
(Python):
from deeppavlov import configs
When you're decided on the model (+ config file), there are two ways to train, evaluate and infer it:
- via Command line interface (CLI) and
- via Python.
To run supported DeepPavlov models on GPU you should have CUDA 10.0
installed on your host machine and TensorFlow with GPU support (tensorflow-gpu
)
installed in your python environment. Current supported TensorFlow version is 1.15.2.
Run
pip install tensorflow-gpu==1.15.2
before installing model's package requirements to install supported tensorflow-gpu
version.
Before making choice of an interface, install model's package requirements (CLI):
python -m deeppavlov install <config_path>
- where
<config_path>
is path to the chosen model's config file (e.g.deeppavlov/configs/classifiers/insults_kaggle_bert.json
) or just name without .json extension (e.g.insults_kaggle_bert
)
To get predictions from a model interactively through CLI, run
python -m deeppavlov interact <config_path> [-d]
-d
downloads required data -- pretrained model files and embeddings (optional).
You can train it in the same simple way:
python -m deeppavlov train <config_path> [-d]
Dataset will be downloaded regardless of whether there was -d
flag or not.
To train on your own data you need to modify dataset reader path in the train config doc. The data format is specified in the corresponding model doc page.
There are even more actions you can perform with configs:
python -m deeppavlov <action> <config_path> [-d]
<action>
can bedownload
to download model's data (same as-d
),train
to train the model on the data specified in the config file,evaluate
to calculate metrics on the same dataset,interact
to interact via CLI,riseapi
to run a REST API server (see doc),predict
to get prediction for samples from stdin or from <file_path> if-f <file_path>
is specified.
<config_path>
specifies path (or name) of model's config file-d
downloads required data
To get predictions from a model interactively through Python, run
from deeppavlov import build_model
model = build_model(<config_path>, download=True)
# get predictions for 'input_text1', 'input_text2'
model(['input_text1', 'input_text2'])
- where
download=True
downloads required data from web -- pretrained model files and embeddings (optional), <config_path>
is path to the chosen model's config file (e.g."deeppavlov/configs/ner/ner_ontonotes_bert_mult.json"
) ordeeppavlov.configs
attribute (e.g.deeppavlov.configs.ner.ner_ontonotes_bert_mult
without quotation marks).
You can train it in the same simple way:
from deeppavlov import train_model
model = train_model(<config_path>, download=True)
download=True
downloads pretrained model, therefore the pretrained model will be, first, loaded and then train (optional).
Dataset will be downloaded regardless of whether there was -d
flag or
not.
To train on your own data you need to modify dataset reader path in the train config doc. The data format is specified in the corresponding model doc page.
You can also calculate metrics on the dataset specified in your config file:
from deeppavlov import evaluate_model
model = evaluate_model(<config_path>, download=True)
DeepPavlov is Apache 2.0 - licensed.