Coder Social home page Coder Social logo

deeppavlov / deeppavlov Goto Github PK

View Code? Open in Web Editor NEW
6.5K 209.0 1.1K 31.85 MB

An open source library for deep learning end-to-end dialog systems and chatbots.

Home Page: https://deeppavlov.ai

License: Apache License 2.0

Python 99.53% HTML 0.31% Dockerfile 0.13% Shell 0.03%
bot nlp chatbot dialogue-systems question-answering chitchat slot-filling intent-classification entity-extraction named-entity-recognition

deeppavlov's Introduction

License Apache 2.0 Python 3.6, 3.7, 3.8, 3.9, 3.10, 3.11 Downloads

DeepPavlov is an open-source conversational AI library built on PyTorch.

DeepPavlov is designed for

  • development of production ready chat-bots and complex conversational systems,
  • research in the area of NLP and, particularly, of dialog systems.

Quick Links

Please leave us your feedback on how we can improve the DeepPavlov framework.

Models

Named Entity Recognition | Intent/Sentence Classification |

Question Answering over Text (SQuAD) | Knowledge Base Question Answering

Sentence Similarity/Ranking | TF-IDF Ranking

Syntactic Parsing | Morphological Tagging

Automatic Spelling Correction | Entity Extraction

Open Domain Questions Answering | Russian SuperGLUE

Relation Extraction

Embeddings

BERT embeddings for the Russian, Polish, Bulgarian, Czech, and informal English

ELMo embeddings for the Russian language

FastText embeddings for the Russian language

Auto ML

Tuning Models

Integrations

REST API | Socket API

Amazon AWS

Installation

  1. DeepPavlov supports Linux, Windows 10+ (through WSL/WSL2), MacOS (Big Sur+) platforms, Python 3.6, 3.7, 3.8, 3.9 and 3.10. Depending on the model used, you may need from 4 to 16 GB RAM.

  2. Create and activate a virtual environment:

    • Linux
    python -m venv env
    source ./env/bin/activate
    
  3. Install the package inside the environment:

    pip install deeppavlov
    

QuickStart

There is a bunch of great pre-trained NLP models in DeepPavlov. Each model is determined by its config file.

List of models is available on the doc page in the deeppavlov.configs (Python):

from deeppavlov import configs

When you're decided on the model (+ config file), there are two ways to train, evaluate and infer it:

GPU requirements

By default, DeepPavlov installs models requirements from PyPI. PyTorch from PyPI could not support your device CUDA capability. To run supported DeepPavlov models on GPU you should have CUDA compatible with used GPU and PyTorch version required by DeepPavlov models. See docs for details. GPU with Pascal or newer architecture and 4+ GB VRAM is recommended.

Command line interface (CLI)

To get predictions from a model interactively through CLI, run

python -m deeppavlov interact <config_path> [-d] [-i]
  • -d downloads required data - pretrained model files and embeddings (optional).
  • -i installs model requirements (optional).

You can train it in the same simple way:

python -m deeppavlov train <config_path> [-d] [-i]

Dataset will be downloaded regardless of whether there was -d flag or not.

To train on your own data you need to modify dataset reader path in the train config doc. The data format is specified in the corresponding model doc page.

There are even more actions you can perform with configs:

python -m deeppavlov <action> <config_path> [-d] [-i]
  • <action> can be
    • install to install model requirements (same as -i),
    • download to download model's data (same as -d),
    • train to train the model on the data specified in the config file,
    • evaluate to calculate metrics on the same dataset,
    • interact to interact via CLI,
    • riseapi to run a REST API server (see doc),
    • predict to get prediction for samples from stdin or from <file_path> if -f <file_path> is specified.
  • <config_path> specifies path (or name) of model's config file
  • -d downloads required data
  • -i installs model requirements

Python

To get predictions from a model interactively through Python, run

from deeppavlov import build_model

model = build_model(<config_path>, install=True, download=True)

# get predictions for 'input_text1', 'input_text2'
model(['input_text1', 'input_text2'])

where

  • install=True installs model requirements (optional),
  • download=True downloads required data from web - pretrained model files and embeddings (optional),
  • <config_path> is model name (e.g. 'ner_ontonotes_bert_mult'), path to the chosen model's config file (e.g. "deeppavlov/configs/ner/ner_ontonotes_bert_mult.json"), or deeppavlov.configs attribute (e.g. deeppavlov.configs.ner.ner_ontonotes_bert_mult without quotation marks).

You can train it in the same simple way:

from deeppavlov import train_model 

model = train_model(<config_path>, install=True, download=True)

To train on your own data you need to modify dataset reader path in the train config doc. The data format is specified in the corresponding model doc page.

You can also calculate metrics on the dataset specified in your config file:

from deeppavlov import evaluate_model 

model = evaluate_model(<config_path>, install=True, download=True)

DeepPavlov also allows to build a model from components for inference using Python.

License

DeepPavlov is Apache 2.0 - licensed.

deeppavlov's People

Contributors

acriptis avatar alexeysorokin avatar avkamenev avatar deeppavlovadmin avatar dilyararimovna avatar dimakarp1996 avatar dmitrijeuseew avatar eugen2525 avatar guitaricet avatar ignatovfedor avatar kolpnick avatar kudep avatar litinsky avatar madrugado avatar mu-arkhipov avatar my-master avatar nastyachizhikova avatar nikolay-bushkov avatar oserikov avatar pagul avatar peganovanton avatar puleon avatar seliverstov avatar tsundokum avatar varvara-l avatar vaskonov avatar velik97 avatar vikmary avatar yoptar avatar yurakuratov avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

deeppavlov's Issues

Show loss in training logs

train_on_batch method can return loss for this batch and we can average it out when logging, for example. This feature would be nice by itself but it's becoming even more important for models that should not be infered too often, so we should not calculate validation metrics when training

'Usage of cuDNN RNN layers require GPU along with cuDNN library

Good day, I can`t start DeepPavlov on my notebook..

python -m deeppavlov.deep interact ./deeppavlov/configs/ner/ner_rus.json
/home/mpoder/Projects/DeepPavlov/venv/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
return f(*args, **kwds)
/home/mpoder/Projects/DeepPavlov/venv/lib/python3.6/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Using TensorFlow backend.
[nltk_data] Downloading package punkt to /home/mpoder/nltk_data...
[nltk_data] Package punkt is already up-to-date!
[nltk_data] Downloading package stopwords to /home/mpoder/nltk_data...
[nltk_data] Package stopwords is already up-to-date!
2018-06-04 15:42:32.709 DEBUG in 'gensim.models.doc2vec'['doc2vec'] at line 73: Fast version of gensim.models.doc2vec is being used
2018-06-04 15:42:32.715 INFO in 'summa.preprocessing.cleaner'['textcleaner'] at line 20: 'pattern' package not found; tag filters are not available for English
2018-06-04 15:42:34.101 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 92: [loading vocabulary from /home/mpoder/Projects/DeepPavlov/DeepPavlov/download/ner_rus/word.dict]
2018-06-04 15:42:34.154 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 92: [loading vocabulary from /home/mpoder/Projects/DeepPavlov/DeepPavlov/download/ner_rus/tag.dict]
2018-06-04 15:42:34.155 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 92: [loading vocabulary from /home/mpoder/Projects/DeepPavlov/DeepPavlov/download/ner_rus/char.dict]
2018-06-04 15:42:34.155 INFO in 'deeppavlov.models.embedders.fasttext_embedder'['fasttext_embedder'] at line 51: [loading embeddings from /home/mpoder/Projects/DeepPavlov/DeepPavlov/download/embeddings/lenta_lower_100.bin]
2018-06-04 15:42:36.597520: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX
2018-06-04 15:42:36.598 ERROR in 'deeppavlov.core.common.params'['params'] at line 112: Exception in <class 'deeppavlov.models.ner.network.NerNetwork'>
Traceback (most recent call last):
File "/home/mpoder/Projects/DeepPavlov/DeepPavlov/deeppavlov/core/common/params.py", line 106, in from_params
component = cls(**dict(config_params, **kwargs))
File "/home/mpoder/Projects/DeepPavlov/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 47, in call
obj.init(*args, **kwargs)
File "/home/mpoder/Projects/DeepPavlov/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 28, in _wrapped
return func(*args, **kwargs)
File "/home/mpoder/Projects/DeepPavlov/DeepPavlov/deeppavlov/models/ner/network.py", line 102, in init
units = self._build_cudnn_rnn(features, n_hidden_list, cell_type, intra_layer_dropout, self.mask_ph)
File "/home/mpoder/Projects/DeepPavlov/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 28, in _wrapped
return func(*args, **kwargs)
File "/home/mpoder/Projects/DeepPavlov/DeepPavlov/deeppavlov/models/ner/network.py", line 168, in _build_cudnn_rnn
raise RuntimeError('Usage of cuDNN RNN layers require GPU along with cuDNN library')
RuntimeError: Usage of cuDNN RNN layers require GPU along with cuDNN library
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/mpoder/Projects/DeepPavlov/DeepPavlov/deeppavlov/deep.py", line 77, in
main()
File "/home/mpoder/Projects/DeepPavlov/DeepPavlov/deeppavlov/deep.py", line 64, in main
interact_model(pipeline_config_path)
File "/home/mpoder/Projects/DeepPavlov/DeepPavlov/deeppavlov/core/commands/infer.py", line 104, in interact_model
model = build_model_from_config(config)
File "/home/mpoder/Projects/DeepPavlov/DeepPavlov/deeppavlov/core/commands/infer.py", line 43, in build_model_from_config
component = from_params(component_config, vocabs=[], mode=mode)
File "/home/mpoder/Projects/DeepPavlov/DeepPavlov/deeppavlov/core/common/params.py", line 106, in from_params
component = cls(**dict(config_params, **kwargs))
File "/home/mpoder/Projects/DeepPavlov/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 47, in call
obj.init(*args, **kwargs)
File "/home/mpoder/Projects/DeepPavlov/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 28, in _wrapped
return func(*args, **kwargs)
File "/home/mpoder/Projects/DeepPavlov/DeepPavlov/deeppavlov/models/ner/network.py", line 102, in init
units = self._build_cudnn_rnn(features, n_hidden_list, cell_type, intra_layer_dropout, self.mask_ph)
File "/home/mpoder/Projects/DeepPavlov/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 28, in _wrapped
return func(*args, **kwargs)
File "/home/mpoder/Projects/DeepPavlov/DeepPavlov/deeppavlov/models/ner/network.py", line 168, in _build_cudnn_rnn
raise RuntimeError('Usage of cuDNN RNN layers require GPU along with cuDNN library')
RuntimeError: Usage of cuDNN RNN layers require GPU along with cuDNN library

No super class method found for "load"?

When i call python download.py with python3.5

Traceback (most recent call last):
File "download.py", line 24, in
from deeppavlov.core.data.utils import download, download_decompress
File "/home/inmove/code/deeplearning/lib/python3.5/site-packages/deeppavlov-0.0.1-py3.5.egg/deeppavlov/init.py", line 1, in
from deeppavlov.core.models.keras_model import KerasModel
File "/home/inmove/code/deeplearning/lib/python3.5/site-packages/deeppavlov-0.0.1-py3.5.egg/deeppavlov/core/models/keras_model.py", line 38, in
class KerasModel(Trainable, Inferable, metaclass=TfModelMeta):
File "/home/inmove/code/deeplearning/lib/python3.5/site-packages/deeppavlov-0.0.1-py3.5.egg/deeppavlov/core/models/keras_model.py", line 145, in KerasModel
sample_weight_mode=None, weighted_metrics=None, target_tensors=None):
File "/home/inmove/code/deeplearning/lib/python3.5/site-packages/overrides/overrides.py", line 70, in overrides
method.name)
AssertionError: No super class method found for "load"

Plz help me thx.

GO-bot fails with special character input

After launching
python deep.py interact skills/go_bot/config.json
and inputs containing `~*, i got:

Traceback (most recent call last):
File "deep.py", line 63, in
main()
File "deep.py", line 49, in main
interact_model(pipeline_config_path)
File "/home/nab/work/DeepPavlov/deeppavlov/core/commands/infer.py", line 87, in interact_model
raise e
File "/home/nab/work/DeepPavlov/deeppavlov/core/commands/infer.py", line 84, in interact_model
pred = model.infer(context)
File "/home/nab/work/DeepPavlov/deeppavlov/skills/go_bot/go_bot.py", line 251, in infer
self._encode_context(context, db_result),
File "/home/nab/work/DeepPavlov/deeppavlov/skills/go_bot/go_bot.py", line 123, in _encode_context
self.tracker.update_state(self.slot_filler.infer(tokenized))
File "/home/nab/work/DeepPavlov/deeppavlov/models/ner/slotfill.py", line 74, in infer
return self.predict_slots(instance.lower())
File "/home/nab/work/DeepPavlov/deeppavlov/models/ner/slotfill.py", line 84, in predict_slots
tags = self._ner_network.predict_for_token_batch([tokens])[0]
File "/home/nab/work/DeepPavlov/deeppavlov/core/models/tf_backend.py", line 28, in _wrapped
return func(*args, **kwargs)
File "/home/nab/work/DeepPavlov/deeppavlov/models/ner/ner_network.py", line 408, in predict_for_token_batch
max_token_len = max([len(token) for utt in tokens_batch for token in utt])
ValueError: max() arg is an empty sequence

Downloading requirements

I was trying to install deeppavlov and had a problem following the installation steps.

  1. There is no download.py file in root folder, it is in deeppavlov/download.py
python download.py [-all] 
  1. Even if I use that file it outputs the error:
(env) root@mysexyhost:~/work/ipavlov/DeepPavlov# python3 deeppavlov/download.py
/home/ubuntu/work/ipavlov/env/local/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
2018-03-12 07:34:11.490 ERROR in 'deeppavlov.core.models.serializable'['log'] at line 54: LOGGER ERROR: Can not initialise deeppavlov.core.models.serializable logger, logging to the stderr. Error traceback:
Traceback (most recent call last):
  File "/home/ubuntu/work/ipavlov/DeepPavlov/deeppavlov/core/common/log.py", line 32, in get_logger
    with open(log_config_path) as log_config_json:
TypeError: invalid file: PosixPath('/home/ubuntu/work/ipavlov/DeepPavlov/deeppavlov/log_config.json')
2018-03-12 07:34:11.491 ERROR in 'deeppavlov.core.models.keras_model'['log'] at line 54: LOGGER ERROR: Can not initialise deeppavlov.core.models.keras_model logger, logging to the stderr. Error traceback:
Traceback (most recent call last):
  File "/home/ubuntu/work/ipavlov/DeepPavlov/deeppavlov/core/common/log.py", line 32, in get_logger
    with open(log_config_path) as log_config_json:
TypeError: invalid file: PosixPath('/home/ubuntu/work/ipavlov/DeepPavlov/deeppavlov/log_config.json')
Traceback (most recent call last):
  File "deeppavlov/download.py", line 24, in <module>
    from deeppavlov.core.data.utils import download, download_decompress
  File "/home/ubuntu/work/ipavlov/DeepPavlov/deeppavlov/__init__.py", line 1, in <module>
    import deeppavlov.core.models.keras_model
  File "/home/ubuntu/work/ipavlov/DeepPavlov/deeppavlov/core/models/keras_model.py", line 39, in <module>
    class KerasModel(NNModel, metaclass=TfModelMeta):
  File "/home/ubuntu/work/ipavlov/DeepPavlov/deeppavlov/core/models/keras_model.py", line 143, in KerasModel
    sample_weight_mode=None, weighted_metrics=None, target_tensors=None):
  File "/home/ubuntu/work/ipavlov/env/local/lib/python3.5/site-packages/overrides/overrides.py", line 70, in overrides
    method.__name__)
AssertionError: No super class method found for "load"

Unexpected loss shown during `deep.py train`

python deep.py train config.json
first trains model, then loads it from disk and calculates valid and test scores.
In case when load_path and save_path differ, final test score will be calculated for model from load_path, which creates quite a confusion.

Error while trying to "interact" with intent classifier

dima@HAL9000:~/dev/DeepPavlov/deeppavlov$ python deep.py interact ./configs/intents/intents_dstc2.json
/home/dima/anaconda3/lib/python3.6/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Using TensorFlow backend.
[nltk_data] Downloading package punkt to /home/dima/nltk_data...
[nltk_data] Package punkt is already up-to-date!
[nltk_data] Downloading package stopwords to /home/dima/nltk_data...
[nltk_data] Package stopwords is already up-to-date!
2018-05-30 13:18:09.959 INFO in 'summarizer.preprocessing.cleaner'['textcleaner'] at line 37: 'pattern' package not found; tag filters are not available for English
/home/dima/anaconda3/lib/python3.6/site-packages/fuzzywuzzy/fuzz.py:35: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
2018-05-30 13:18:12.732 INFO in 'deeppavlov.models.embedders.fasttext_embedder'['fasttext_embedder'] at line 51: [loading embeddings from /home/dima/dev/DeepPavlov/download/embeddings/dstc2_fastText_model.bin]
2018-05-30 13:18:12.776 ERROR in 'deeppavlov.core.common.params'['params'] at line 112: Exception in <class 'deeppavlov.models.embedders.fasttext_embedder.FasttextEmbedder'>
Traceback (most recent call last):
File "fasttext/fasttext.pyx", line 152, in fasttext.fasttext.load_model
RuntimeError: vector::_M_default_append

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/dima/dev/DeepPavlov/deeppavlov/core/common/params.py", line 106, in from_params
component = cls(**dict(config_params, **kwargs))
File "/home/dima/dev/DeepPavlov/deeppavlov/models/embedders/fasttext_embedder.py", line 40, in init
self.model = self.load()
File "/home/dima/dev/DeepPavlov/deeppavlov/models/embedders/fasttext_embedder.py", line 53, in load
model = Fasttext.load_model(model_file)
File "fasttext/fasttext.pyx", line 154, in fasttext.fasttext.load_model
Exception: fastText: Cannot load /home/dima/dev/DeepPavlov/download/embeddings/dstc2_fastText_model.bin due to C++ extension failed to allocate the memory
Traceback (most recent call last):
File "fasttext/fasttext.pyx", line 152, in fasttext.fasttext.load_model
RuntimeError: vector::_M_default_append

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "deep.py", line 77, in
main()
File "deep.py", line 64, in main
interact_model(pipeline_config_path)
File "/home/dima/dev/DeepPavlov/deeppavlov/core/commands/infer.py", line 104, in interact_model
model = build_model_from_config(config)
File "/home/dima/dev/DeepPavlov/deeppavlov/core/commands/infer.py", line 43, in build_model_from_config
component = from_params(component_config, vocabs=[], mode=mode)
File "/home/dima/dev/DeepPavlov/deeppavlov/core/common/params.py", line 106, in from_params
component = cls(**dict(config_params, **kwargs))
File "/home/dima/dev/DeepPavlov/deeppavlov/models/embedders/fasttext_embedder.py", line 40, in init
self.model = self.load()
File "/home/dima/dev/DeepPavlov/deeppavlov/models/embedders/fasttext_embedder.py", line 53, in load
model = Fasttext.load_model(model_file)
File "fasttext/fasttext.pyx", line 154, in fasttext.fasttext.load_model
Exception: fastText: Cannot load /home/dima/dev/DeepPavlov/download/embeddings/dstc2_fastText_model.bin due to C++ extension failed to allocate the memory

32 Gb RAM on the machine

Question: How to use the model for my own task?

I want to use the model go_bot for my own task. In particular, the model is similar to configs/go_bot/gobot_dstc2.json. But I want to use my own dialogue data and slots. My question is what files do I need to provide? I understand I need to provide my own configuration in deeppavlov/configs/go_bot/, my own dataset_reader, my own tracker. But how to provide the data for the slot_filler? It would be great if you could specify what files are needed in which directory. Could anyone help me? Thanks.

Reuse of repeated parameters in config

Sometimes duplicated parameters with equal values may appear in config file.

Look at the example below:

"pipe": [
{
"name": "squad_preprocessor",
"context_limit": 400,
"question_limit": 150,
"char_limit": 16,
},
{
"name": "squad_vocab_embedder",
"context_limit": "#squad_prepr.context_limit",
"question_limit": "#squad_prepr.question_limit",
"char_limit": "#squad_prepr.context_limit",
{
"name": "squad_vocab_embedder",
"level": "char",
"context_limit": "#squad_prepr.context_limit",
"question_limit": "#squad_prepr.question_limit",
"char_limit": "#squad_prepr.char_limit",
},
{
"name": "squad_model",
"id": "squad",
"word_emb": "#vocab_embedder.emb_mat",
"char_emb": "#char_vocab_embedder.emb_mat",
"context_limit": "#squad_prepr.context_limit",
"question_limit": "#squad_prepr.question_limit",
"char_limit": "#squad_prepr.char_limit",
}
]

It could be more convenient if there was no need to copy context_limit, question_limit, char_limit and to fill them from something like global parameters scope.

Files not found while running telegram bot

I run telegram bot interface (copy-paste from readme)

python -m deeppavlov.deep interactbot deeppavlov/configs/go_bot/gobot_dstc2.json -t TELEGRAM_TOKEN

But the error is given:

File "/home/ubuntu/work/ipavlov/DeepPavlov/deeppavlov/core/common/file.py", line 22, in read_json
    with open(fpath) as fin:
FileNotFoundError: [Errno 2] No such file or directory: '../telegram_utils/models_info.json' 

All files have been downloaded and successfully used in training.

Running examples

Sorry guys, i am kinda crazy, and trying runing pilot on windows, but so far i even can't get some required files, f.e. on

>python deep.py interact skills/hcn_new/config.json

i am getting error

deeppavlov.core.common.errors.ConfigError: "Provided ser_path doesn't exist!"

Where can i get word_vocab.dict?

thanks!

Is Mac OSX supported?

Hi,

I know the readme says only Linux is supported, but since OSX is a Unix based operating system, it's unclear to me if it should be supported as well.

Thanks,

Patrick

Stuck post install

Hi,

I've gotten DeepPavlov set up on an Ubuntu server with a NVIDIA Tesla K80 GPU, but I can't get DeepPavlov to run any of the download or other commands. I'm at the point where I'm trying to run one of these:

python -m deeppavlov.deep download deeppavlov/configs/squad/squad.json -d
python -m deeppavlov.deep download deeppavlov/configs/go_bot/*.json

I've tried a number of things, but I always end up with this error:

/home/pboz/testenv3/local/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from 'float' to 'np.floating' is deprecated. In future, it will be treated as 'np.float64 == np.dtype(float).type'. from ._conv import register_converters as _register_converters Using TensorFlow backend. [nltk_data] Downloading package punkt to /home/pboz/nltk_data... [nltk_data] Package punkt is already up-to-date! [nltk_data] Downloading package stopwords to [nltk_data] /home/pboz/nltk_data... [nltk_data] Package stopwords is already up-to-date! 2018-05-19 21:35:24.547 INFO in 'gensim.utils'['utils'] at line 53: smart_open library not found; falling back to local-filesystem-only Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 183, in _run_module_as_main mod_name, mod_spec, code = _get_module_details(mod_name, _Error) File "/usr/lib/python3.6/runpy.py", line 109, in _get_module_details __import__(pkg_name) File "/home/pboz/DeepPavlov/deeppavlov/__init__.py", line 44, in <module> import deeppavlov.models.embedders.glove_embedder File "/home/pboz/DeepPavlov/deeppavlov/models/embedders/glove_embedder.py", line 22, in <module> from gensim.models import KeyedVectors File "/home/pboz/testenv3/local/lib/python3.6/site-packages/gensim/__init__.py", line 6, in <module> from gensim import parsing, matutils, interfaces, corpora, models, similarities, summarization File "/home/pboz/testenv3/local/lib/python3.6/site-packages/gensim/models/__init__.py", line 7, in <module> from .coherencemodel import CoherenceModel File "/home/pboz/testenv3/local/lib/python3.6/site-packages/gensim/models/coherencemodel.py", line 30, in <module> from gensim.models.wrappers import LdaVowpalWabbit, LdaMallet File "/home/pboz/testenv3/local/lib/python3.6/site-packages/gensim/models/wrappers/__init__.py", line 5, in <module> from .ldamallet import LdaMallet File "/home/pboz/testenv3/local/lib/python3.6/site-packages/gensim/models/wrappers/ldamallet.py", line 43, in <module> from smart_open import smart_open File "/home/pboz/testenv3/local/lib/python3.6/site-packages/smart_open/__init__.py", line 1, in <module> from .smart_open_lib import * File "/home/pboz/testenv3/local/lib/python3.6/site-packages/smart_open/smart_open_lib.py", line 34, in <module> import boto.s3.connection File "/home/pboz/testenv3/local/lib/python3.6/site-packages/boto/s3/connection.py", line 30, in <module> from boto.auth import detect_potential_s3sigv4 File "/home/pboz/testenv3/local/lib/python3.6/site-packages/boto/__init__.py", line 1216, in <module> boto.plugin.load_plugins(config) AttributeError: module 'boto' has no attribute 'plugin'

As a side note, I'm more of a Java guy, Python isn't my strongest language.

Any help is greatly appreciated.

GO_bot pretrained model checkpoint is broken

Hi, i've download all the data and got an exception once run interact mode

(tensorflow) D:\work\Pilot\deeppavlov>python deep.py interact skills/go_bot/config.json
Using TensorFlow backend.
C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\gensim\utils.py:865: UserWarning: detected Windows; aliasing chunkize to chunkize_serial
  warnings.warn("detected Windows; aliasing chunkize to chunkize_serial")
[loading vocabulary from `..\download\vocabs\word.dict`]
[loading vocabulary from `..\download\vocabs\token.dict`]
[loading vocabulary from `..\download\vocabs\tag.dict`]
[loading vocabulary from `..\download\vocabs\char.dict`]
[loading vocabulary from `..\download\vocabs\classes.dict`]
2018-01-30 20:24:57.237947: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE instructions, but these are available on your machine and could speed up CPU computations.
2018-01-30 20:24:57.238765: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE2 instructions, but these are available on your machine and could speed up CPU computations.
2018-01-30 20:24:57.241623: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
2018-01-30 20:24:57.243348: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2018-01-30 20:24:57.245338: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2018-01-30 20:24:57.246123: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2018-01-30 20:24:57.247209: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2018-01-30 20:24:57.247976: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.

:: initializing `GoalOrientedBotNetwork` from saved

:: restoring checkpoint from ..\download\go_bot_rnn\model-0

2018-01-30 20:24:57.578303: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\framework\op_kernel.cc:1152] Not found: Key lstm_cell/weights/Adadelta_1 not found in checkpoint
2018-01-30 20:24:57.583433: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\framework\op_kernel.cc:1152] Not found: Key lstm_cell/weights/Adadelta not found in checkpoint
2018-01-30 20:24:57.585337: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\framework\op_kernel.cc:1152] Not found: Key lstm_cell/weights not found in checkpoint
2018-01-30 20:24:57.587360: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\framework\op_kernel.cc:1152] Not found: Key lstm_cell/biases/Adadelta_1 not found in checkpoint
2018-01-30 20:24:57.589280: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\framework\op_kernel.cc:1152] Not found: Key lstm_cell/biases/Adadelta not found in checkpoint
2018-01-30 20:24:57.592492: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\framework\op_kernel.cc:1152] Not found: Key lstm_cell/biases not found in checkpoint
Exception in <class 'deeppavlov.skills.go_bot.network.GoalOrientedBotNetwork'>
Traceback (most recent call last):
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1039, in _do_call
    return fn(*args)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1021, in _run_fn
    status, run_metadata)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\contextlib.py", line 88, in __exit__
    next(self.gen)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: Key lstm_cell/weights/Adadelta_1 not found in checkpoint
         [[Node: save/RestoreV2_18 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_18/tensor_names, save/RestoreV2_18/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "deep.py", line 63, in <module>
    main()
  File "deep.py", line 49, in main
    interact_model(pipeline_config_path)
  File "D:\work\Pilot\deeppavlov\core\commands\infer.py", line 73, in interact_model
    model = build_model_from_config(config)
  File "D:\work\Pilot\deeppavlov\core\commands\infer.py", line 34, in build_model_from_config
    model = from_params(REGISTRY[model_name], model_config, vocabs=vocabs, mode=mode)
  File "D:\work\Pilot\deeppavlov\core\common\params.py", line 49, in from_params
    mode=kwargs['mode'])
  File "D:\work\Pilot\deeppavlov\core\common\params.py", line 52, in from_params
    model = cls(**dict(config_params, **kwargs))
  File "D:\work\Pilot\deeppavlov\core\models\tf_backend.py", line 47, in __call__
    obj.__init__(*args, **kwargs)
  File "D:\work\Pilot\deeppavlov\core\models\tf_backend.py", line 28, in _wrapped
    return func(*args, **kwargs)
  File "D:\work\Pilot\deeppavlov\skills\go_bot\network.py", line 49, in __init__
    self.load()
  File "D:\work\Pilot\deeppavlov\core\models\tf_backend.py", line 28, in _wrapped
    return func(*args, **kwargs)
  File "D:\work\Pilot\deeppavlov\core\models\tf_model.py", line 114, in load
    self._saver().restore(self.sess, ckpt.model_checkpoint_path)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1457, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 778, in run
    run_metadata_ptr)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 982, in _run
    feed_dict_string, options, run_metadata)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1032, in _do_run
    target_list, options, run_metadata)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1052, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key lstm_cell/weights/Adadelta_1 not found in checkpoint
         [[Node: save/RestoreV2_18 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_18/tensor_names, save/RestoreV2_18/shape_and_slices)]]

Caused by op 'save/RestoreV2_18', defined at:
  File "deep.py", line 63, in <module>
    main()
  File "deep.py", line 49, in main
    interact_model(pipeline_config_path)
  File "D:\work\Pilot\deeppavlov\core\commands\infer.py", line 73, in interact_model
    model = build_model_from_config(config)
  File "D:\work\Pilot\deeppavlov\core\commands\infer.py", line 34, in build_model_from_config
    model = from_params(REGISTRY[model_name], model_config, vocabs=vocabs, mode=mode)
  File "D:\work\Pilot\deeppavlov\core\common\params.py", line 49, in from_params
    mode=kwargs['mode'])
  File "D:\work\Pilot\deeppavlov\core\common\params.py", line 52, in from_params
    model = cls(**dict(config_params, **kwargs))
  File "D:\work\Pilot\deeppavlov\core\models\tf_backend.py", line 47, in __call__
    obj.__init__(*args, **kwargs)
  File "D:\work\Pilot\deeppavlov\core\models\tf_backend.py", line 28, in _wrapped
    return func(*args, **kwargs)
  File "D:\work\Pilot\deeppavlov\skills\go_bot\network.py", line 49, in __init__
    self.load()
  File "D:\work\Pilot\deeppavlov\core\models\tf_backend.py", line 28, in _wrapped
    return func(*args, **kwargs)
  File "D:\work\Pilot\deeppavlov\core\models\tf_model.py", line 114, in load
    self._saver().restore(self.sess, ckpt.model_checkpoint_path)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1056, in __init__
    self.build()
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1086, in build
    restore_sequentially=self._restore_sequentially)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 691, in build
    restore_sequentially, reshape)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 407, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 247, in restore_op
    [spec.tensor.dtype])[0])
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 669, in restore_v2
    dtypes=dtypes, name=name)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in __init__
    self._traceback = _extract_stack()

NotFoundError (see above for traceback): Key lstm_cell/weights/Adadelta_1 not found in checkpoint
         [[Node: save/RestoreV2_18 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_18/tensor_names, save/RestoreV2_18/shape_and_slices)]]

Should i train my own, is it wrong model or do i have wrong version of TF?

Keras==2.1.2
tensorflow==1.4.0
tensorflow-gpu==1.4.0

Thanks!

Pretrained model usage

There should be an easy way to apply a pretrained model (e.g. NER) to the user data by passing the same steps as in train_model_from_config except of train_on_batches. interact_model is by no means such a way since it requires redirecting stdin passing test sentences one by one.

How to write the component without cloning and patching the whole DeepPavlov repo?

Hi!

I'd like to write my own component in my private project.

It seems like I should clone the repo then patch it:

  1. By writing write my own dataset readers, iterators,... and putting them into DeepPavlov folder.
  2. Then I should put necessary imports into https://github.com/deepmipt/DeepPavlov/blob/master/deeppavlov/__init__.py.
  3. In the end I should write my config.json and run python -m deeppavlov...

Can I write separate component independently without patching source code of DeepPavlov?

Which operating systems are supported?

I just tried to install DeepPavlov on Windows and it fails due to pyfasttext requiring cysignals (which is not available for windows).
I could not find any mention about what operating systems are supported.
So is Windows supported in theory and just breaks because of that dependency, or was it never intended to be supported at all?

How to train goal oriented bot for my language?

Is it feasible? As I can see, the goal oriented part required, at least for english, a dataset designed specifically for it. So if I want to train it for Brazillian Portuguese I'd have to hire people to create, for example, restaurant dialogs?

Is there any work on support for other languages?

#question

Use tensorflow instead of tensorflow-gpu

Version 0.0.1 has tensorflow in requirements.txt.
0.0.3 now uses tensorflow-gpu.

It is common use case to write and test model in laptop without GPU and then deploy it to server with GPU.
So it will be great to support both CPU and GPU modes in the DeepPavlov lib.

python setup.py develop outputs

Searching for tensorflow-gpu==1.4.0
Reading https://pypi.python.org/simple/tensorflow-gpu/
No local packages or working download links found for tensorflow-gpu==1.4.0
error: Could not find suitable distribution for Requirement.parse('tensorflow-gpu==1.4.0')

pip 9.0.3
Python 3.6.1

Go_bot return #name

I interact to go_bot in config: configs/go_bot/gobot_dstc2.json
but it alway return some kind of: #name, #addrs,...
like this:
`x::i want some food
['i want some food']

I am sorry but there is no #food restaurant that matches your request.
x::i want sea food
['i want sea food']
What part of town do you have in mind?
x::north
['north']
Would you like something in the cheap, moderate, or expensive price range?
x::cheap
['cheap']
#name is a nice place in the north of town serving tasty seafood food.
x::ok thank
['ok thank']
You are welcome!
x::exit`

How can I make bot returning the suggested information instead of #name or #food?

Thanks.

Попробуйте

Вместо наложения маски на результат, добавить при обучении в функцию ошибки расстояние между вектором с маской и без.

Telegram interface bug

Alexander Seliverstov, [04.03.18 15:20]
/start

jhfirufoiug_bot, [04.03.18 15:20]
Welcome to DeepPavlov inference bot!

Alexander Seliverstov, [04.03.18 15:20]
Hi

jhfirufoiug_bot, [04.03.18 15:20]
['Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?']

Alexander Seliverstov, [04.03.18 15:20]
I want cheap russian food

jhfirufoiug_bot, [04.03.18 15:20]
['Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?', 'Hello, welcome to the Cambridge restaurant system. You can ask for restaurants by area, price range or food type. How may I help you?']

go_bot_all pretrained model not found

Hi,

I was trying to test the go_bot full model with the following command:

  python3 deep.py interact configs/go_bot/gobot_dstc2_all.json

And noticed that the download/go_bot_all directory is empty (PS: I've executed "python3 -m deeppavlov.download -all" and see all other existing model files).

As a result, the GoalOrientedBotNetwork._init_ method couldn't find the "go_bot_all/model" as specified in the gobot_dstc2_all.json config file.

Could you please upload the pretrained full model?

Thanks

Memory Error in kb.py

Class KnowledgeBaseEntityNormalizer , line 95:

 def normalize(self, key, tokens):
        utter = ' '.join(tokens)
        for entity, value in self.kb([key])[0]:
            utter = utter.replace(' '.join(value), entity)
        return utter.split()

Memory leak when value=[] leads to Memory Error

Discuss: UNIX-way or not

If we want to be able to pipe model's output to file or other programs (models), we should write logs to stderr instead of stdout or use some other form of logging.

How seq2seq_go_bot model was created?

Hi,

This is a very promising project, but I'm having troubles in understand how the models was created, e.g. seq2seq_go_bot?

What train/test/validation data was used?
How the model was built?

Thanks!

Problem with fastText

i have issue when start "python -m deeppavlov interactbot deeppavlov/configs/intents/intents_snips.json -d -t 597656151:AAGF3C9T8U8T7iyEIY8_u1t_u9Qv1_LQKUo"
AttributeError: module 'fastText' has no attribute 'load_model'
i use Windows 10, anaconda

can't install to a fresh virtual environment

`(.vipavlov) marat@dgx1:~/DeepPavlov$ pip3 install -r requirements.txt
Collecting Cython (from -r requirements.txt (line 1))
Downloading Cython-0.27.3-cp35-cp35m-manylinux1_x86_64.whl (3.0MB)
100% |████████████████████████████████| 3.0MB 538kB/s
Collecting lxml==4.1.1 (from -r requirements.txt (line 2))
Downloading lxml-4.1.1-cp35-cp35m-manylinux1_x86_64.whl (5.5MB)
100% |████████████████████████████████| 5.5MB 299kB/s
Collecting tqdm==4.19.4 (from -r requirements.txt (line 3))
Downloading tqdm-4.19.4-py2.py3-none-any.whl (50kB)
100% |████████████████████████████████| 51kB 11.7MB/s
Collecting requests==2.18.4 (from -r requirements.txt (line 4))
Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting gensim==2.3.0 (from -r requirements.txt (line 5))
Downloading gensim-2.3.0.tar.gz (17.2MB)
100% |████████████████████████████████| 17.2MB 90kB/s
Collecting tensorflow==1.4.0 (from -r requirements.txt (line 6))
Downloading tensorflow-1.4.0-cp35-cp35m-manylinux1_x86_64.whl (40.7MB)
100% |████████████████████████████████| 40.7MB 38kB/s
Collecting overrides==1.9 (from -r requirements.txt (line 7))
Downloading overrides-1.9.tar.gz
Collecting kenlm==0.0.0 (from -r requirements.txt (line 9))
Downloading kenlm-0.tar.gz (1.4MB)
100% |████████████████████████████████| 1.5MB 1.0MB/s
Collecting six==1.11.0 (from -r requirements.txt (line 10))
Using cached six-1.11.0-py2.py3-none-any.whl
Collecting h5py==2.7.1 (from -r requirements.txt (line 11))
Downloading h5py-2.7.1-cp35-cp35m-manylinux1_x86_64.whl (5.3MB)
100% |████████████████████████████████| 5.3MB 336kB/s
Collecting keras==2.1.2 (from -r requirements.txt (line 12))
Downloading Keras-2.1.2-py2.py3-none-any.whl (304kB)
100% |████████████████████████████████| 307kB 4.7MB/s
Collecting pandas==0.21.1 (from -r requirements.txt (line 13))
Downloading pandas-0.21.1-cp35-cp35m-manylinux1_x86_64.whl (25.7MB)
100% |████████████████████████████████| 25.7MB 57kB/s
Collecting fuzzywuzzy==0.16.0 (from -r requirements.txt (line 14))
Downloading fuzzywuzzy-0.16.0-py2.py3-none-any.whl
Collecting fasttext==0.8.3 (from -r requirements.txt (line 15))
Downloading fasttext-0.8.3.tar.gz (73kB)
100% |████████████████████████████████| 81kB 9.8MB/s
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-build-ym9i0nin/fasttext/setup.py", line 3, in
from Cython.Build import cythonize
ImportError: No module named 'Cython'

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-ym9i0nin/fasttext/
`

Can't download models with `python download.py`

(.vipavlov) marat@dgx1:~/DeepPavlov/deeppavlov$ python download.py /home/marat/data/taskdialog/.vipavlov/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from floattonp.floatingis deprecated. In future, it will be treated asnp.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters Using TensorFlow backend. Traceback (most recent call last): File "download.py", line 24, in <module> from deeppavlov.core.data.utils import download, download_decompress File "/home/marat/data/taskdialog/.vipavlov/lib/python3.5/site-packages/deeppavlov-0.0.1-py3.5.egg/deeppavlov/__init__.py", line 1, in <module> from deeppavlov.core.models.keras_model import KerasModel File "/home/marat/data/taskdialog/.vipavlov/lib/python3.5/site-packages/deeppavlov-0.0.1-py3.5.egg/deeppavlov/core/models/keras_model.py", line 38, in <module> class KerasModel(Trainable, Inferable, metaclass=TfModelMeta): File "/home/marat/data/taskdialog/.vipavlov/lib/python3.5/site-packages/deeppavlov-0.0.1-py3.5.egg/deeppavlov/core/models/keras_model.py", line 145, in KerasModel sample_weight_mode=None, weighted_metrics=None, target_tensors=None): File "/home/marat/data/taskdialog/.vipavlov/lib/python3.5/site-packages/overrides/overrides.py", line 70, in overrides method.__name__) AssertionError: No super class method found for "load"

deeppavlov.core.common.errors.ConfigError: 'Given fasttext model does NOT match fasttext model used previously to train loaded model'

I goy the following error

Traceback (most recent call last):
File "deep.py", line 63, in
main()
File "deep.py", line 55, in main
interact_model_by_telegram(pipeline_config_path, token)
File "/home/mdixie/.conda/envs/deeppavlov/lib/python3.6/site-packages/deeppavlov-0.0.1-py3.6.egg/telegram_utils/telegram_ui.py", line 57, in interact_model_by_telegram
model = build_model_from_config(config)
File "/home/mdixie/.conda/envs/deeppavlov/lib/python3.6/site-packages/deeppavlov-0.0.1-py3.6.egg/deeppavlov/core/commands/infer.py", line 34, in build_model_from_config
model = from_params(REGISTRY[model_name], model_config, vocabs=vocabs, mode=mode)
File "/home/mdixie/.conda/envs/deeppavlov/lib/python3.6/site-packages/deeppavlov-0.0.1-py3.6.egg/deeppavlov/core/common/params.py", line 49, in from_params
mode=kwargs['mode'])
File "/home/mdixie/.conda/envs/deeppavlov/lib/python3.6/site-packages/deeppavlov-0.0.1-py3.6.egg/deeppavlov/core/common/params.py", line 52, in from_params
model = cls(**dict(config_params, **kwargs))
File "/home/mdixie/.conda/envs/deeppavlov/lib/python3.6/site-packages/deeppavlov-0.0.1-py3.6.egg/deeppavlov/core/models/tf_backend.py", line 47, in call
obj.init(*args, **kwargs)
File "/home/mdixie/.conda/envs/deeppavlov/lib/python3.6/site-packages/deeppavlov-0.0.1-py3.6.egg/deeppavlov/core/models/tf_backend.py", line 28, in _wrapped
return func(*args, **kwargs)
File "/home/mdixie/.conda/envs/deeppavlov/lib/python3.6/site-packages/deeppavlov-0.0.1-py3.6.egg/deeppavlov/models/classifiers/intents/intent_model.py", line 138, in init
"Given fasttext model does NOT match fasttext model used previously to train loaded model")
deeppavlov.core.common.errors.ConfigError: 'Given fasttext model does NOT match fasttext model used previously to train loaded model'

ValueError: Cannot feed value of shape (1, 1, 802) for Tensor 'features:0', which has shape '(?, ?, 830)' ]

I fired the following command to train from scratch...

python -m deeppavlov.deep train deeppavlov/configs/go_bot/gobot_dstc2_all.json

I get the following error.

[Traceback (most recent call last):
  File "/home/mdixie/.conda/envs/dl/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/mdixie/.conda/envs/dl/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/n/w1-mdixie/Others_Models/DeepPavlov_Train/DeepPavlov/deeppavlov/deep.py", line 62, in <module>
    main()
  File "/n/w1-mdixie/Others_Models/DeepPavlov_Train/DeepPavlov/deeppavlov/deep.py", line 49, in main
    train_model_from_config(pipeline_config_path)
  File "/n/w1-mdixie/Others_Models/DeepPavlov_Train/DeepPavlov/deeppavlov/core/commands/train.py", line 127, in train_model_from_config
    _train_batches(model, iterator, train_config, metrics_functions)
  File "/n/w1-mdixie/Others_Models/DeepPavlov_Train/DeepPavlov/deeppavlov/core/commands/train.py", line 229, in _train_batches
    y_predicted = list(model(list(x)))
  File "/n/w1-mdixie/Others_Models/DeepPavlov_Train/DeepPavlov/deeppavlov/core/common/chainer.py", line 88, in __call__
    res = component(*[mem[k] for k in in_params])
  File "/n/w1-mdixie/Others_Models/DeepPavlov_Train/DeepPavlov/deeppavlov/skills/go_bot/bot.py", line 251, in __call__
    return [self._infer_dialog(x) for x in batch]
  File "/n/w1-mdixie/Others_Models/DeepPavlov_Train/DeepPavlov/deeppavlov/skills/go_bot/bot.py", line 251, in <listcomp>
    return [self._infer_dialog(x) for x in batch]
  File "/n/w1-mdixie/Others_Models/DeepPavlov_Train/DeepPavlov/deeppavlov/skills/go_bot/bot.py", line 245, in _infer_dialog
    res.append(self._infer(context['text'], context.get('db_result')))
  File "/n/w1-mdixie/Others_Models/DeepPavlov_Train/DeepPavlov/deeppavlov/skills/go_bot/bot.py", line 220, in _infer
    prob=True
  File "/n/w1-mdixie/Others_Models/DeepPavlov_Train/DeepPavlov/deeppavlov/skills/go_bot/network.py", line 62, in __call__
    self._action_mask: action_mask
  File "/home/mdixie/.conda/envs/dl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 889, in run
    run_metadata_ptr)
  File "/home/mdixie/.conda/envs/dl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1096, in _run
    % (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (1, 1, 802) for Tensor 'features:0', which has shape '(?, ?, 830)'
]

`show_examples` can help with model debug

Thain script has a show_examples option, which doesn't work now, but potentially can really help with model debug and improvement. (On show_examples=True script is expected to print wrongly classified examples)

Database connection failing with Telegram API

Hi,

When using pre-trained gobot_dstc2_best.json with interactbot, I receive the following error:

sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 4321387392 and this is thread id 123145327210496

However, it does work using interact. Would you know what is the problem with the Telegram API?

Thanks!

In Dataset class make separate index generation method

def batch_generator(self, batch_size: int, data_type: str = 'train',
                        shuffle: bool = None) -> Generator:
    r"""This function returns a generator, which serves for generation of raw
    (no preprocessing such as tokenization)
     batches
    Args:
        batch_size (int): number of samples in batch
        data_type (str): can be either 'train', 'test', or 'valid'
        shuffle (bool): whether to shuffle dataset before batching
    Returns:
        batch_gen (Generator): a generator, that iterates through the part (defined by data_type) of the dataset
    """
    if shuffle is None:
        shuffle = self.shuffle

    data = self.data[data_type]
    data_len = len(data)

    if data_len == 0:
        return

    order = list(range(data_len)) # HERE
    if shuffle: # HERE
        rs = random.getstate() # HERE
        random.setstate(self.random_state) # HERE
        random.shuffle(order) # HERE
        self.random_state = random.getstate() # HERE
        random.setstate(rs) # AND HERE
   # To separate method

    if batch_size < 0:
        batch_size = data_len

    for i in range((data_len - 1) // batch_size + 1):
        yield tuple(zip(*[data[o] for o in order[i * batch_size:(i + 1) * batch_size]]))

What is "'Chainer' object has no attribute 'infer'

2018-03-04 14:09:23,638 (util.py:64 WorkerThread2) ERROR - TeleBot: "AttributeError occurred, args=("'Chainer' object has no attribute 'infer'",)
Traceback (most recent call last):
File "/Users/developer/DeepPavlov/lib/python3.6/site-packages/telebot/util.py", line 58, in run
task(*args, **kwargs)
File "/Users/developer/Project/DeepPavlov/telegram_utils/telegram_ui.py", line 48, in handle_inference
pred = model.infer(context)
AttributeError: 'Chainer' object has no attribute 'infer'
"
2018-03-04 14:09:23.638 ERROR in 'TeleBot'['util'] at line 64: AttributeError occurred, args=("'Chainer' object has no attribute 'infer'",)
Traceback (most recent call last):
File "/Users/developer/DeepPavlov/lib/python3.6/site-packages/telebot/util.py", line 58, in run
task(*args, **kwargs)
File "/Users/developer/Project/DeepPavlov/telegram_utils/telegram_ui.py", line 48, in handle_inference
pred = model.infer(context)
AttributeError: 'Chainer' object has no attribute 'infer'

Traceback (most recent call last):
File "deep.py", line 60, in
main()
File "deep.py", line 56, in main
interact_model_by_telegram(pipeline_config_path, token)
File "/Users/developer/Project/DeepPavlov/telegram_utils/telegram_ui.py", line 58, in interact_model_by_telegram
init_bot_for_model(token, model)
File "/Users/developer/Project/DeepPavlov/telegram_utils/telegram_ui.py", line 52, in init_bot_for_model
bot.polling()
File "/Users/developer/DeepPavlov/lib/python3.6/site-packages/telebot/init.py", line 264, in polling
self.__threaded_polling(none_stop, interval, timeout)
File "/Users/developer/DeepPavlov/lib/python3.6/site-packages/telebot/init.py", line 288, in __threaded_polling
self.worker_pool.raise_exceptions()
File "/Users/developer/DeepPavlov/lib/python3.6/site-packages/telebot/util.py", line 107, in raise_exceptions
six.reraise(self.exc_info[0], self.exc_info[1], self.exc_info[2])
File "/Users/developer/DeepPavlov/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/Users/developer/DeepPavlov/lib/python3.6/site-packages/telebot/util.py", line 58, in run
task(*args, **kwargs)
File "/Users/developer/Project/DeepPavlov/telegram_utils/telegram_ui.py", line 48, in handle_inference
pred = model.infer(context)
AttributeError: 'Chainer' object has no attribute 'infer'

Hardware requirements for DeepPavlov

Dear Colleagues!
Have not found minimal hardware requirements for DeepPavlov installation.
(for example loading embeddings from wiki.en.bin takes about 20 minutes...)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.