iemldev / ieml Goto Github PK

IEML semantic language - a meaning-representation system based on semantic primitives and a regular grammar. Basic semantic relationships between concepts are automatically computed from syntactic similarities.

Home Page: https://dev.intlekt.io/

License: GNU General Public License v3.0

Python 71.60% Jupyter Notebook 27.87% Shell 0.10% TeX 0.43%

dictionary semantic usl topic ieml language meaning-representation semantic-relationships natural-language

ieml's People

Contributors

Stargazers

Watchers

Forkers

damons sidineyr scape1989 loup-brun bhutanmont hmltn2

ieml's Issues

E: is not a NullScript.

Faire la doc

relations errors

missing relations child d.i.- => d.i.-l.i.-' (the father is present)
relation opposed : d.i.-l.i.-' => l.i.-d.i.-t.u.-'

Error on projection

Trying to project posts give the following error:

Internal Server Error: /dictionary/I:/tables/
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/script/parser/parser.py", line 33, in t_parse
    return self.parser.parse(s, lexer=self.lexer)
  File "/usr/lib/python3.6/site-packages/ply/yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/usr/lib/python3.6/site-packages/ply/yacc.py", line 1199, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/usr/lib/python3.6/site-packages/ply/yacc.py", line 193, in call_errorfunc
    r = errorfunc(token)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/script/parser/parser.py", line 43, in p_error
    raise InvalidScript(msg)
ieml.exceptions.InvalidScript: Invalid arguments to create a script. Syntax error at ',' (1, 27)

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rest_framework/viewsets.py", line 86, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/django_intlekt-0.3.1-py3.6.egg/django_intlekt/views/dictionary.py", line 267, in wrapper
    ieml.dictionary.term(pk)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/tools.py", line 15, in term
    dictionary = Dictionary()
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 47, in __call__
    cls._instance = load_dictionary_from_cache(version)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/version.py", line 290, in load_dictionary_from_cache
    return pickle.load(fp)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 182, in __setstate__
    self._populate(scripts=state['scripts'], relations=state['relations'])
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 146, in _populate
    root = script(root)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/script/operator.py", line 18, in script
    s = ScriptParser().parse(arg)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/script/parser/parser.py", line 35, in t_parse
    raise CannotParse(s, str(e))
ieml.exceptions.CannotParse: Unable to parse the following string 'n.-S:.U:.-'B:.-'S:.-',M:.-',M:.-',_'. Invalid arguments to create a script. Syntax error at ',' (1, 27)
Internal Server Error: /dictionary/I:/relations/
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/script/parser/parser.py", line 33, in t_parse
    return self.parser.parse(s, lexer=self.lexer)
  File "/usr/lib/python3.6/site-packages/ply/yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/usr/lib/python3.6/site-packages/ply/yacc.py", line 1199, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/usr/lib/python3.6/site-packages/ply/yacc.py", line 193, in call_errorfunc
    r = errorfunc(token)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/script/parser/parser.py", line 43, in p_error
    raise InvalidScript(msg)
ieml.exceptions.InvalidScript: Invalid arguments to create a script. Syntax error at 'n' (1, 12)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rest_framework/viewsets.py", line 86, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/django_intlekt-0.3.1-py3.6.egg/django_intlekt/views/dictionary.py", line 267, in wrapper
    ieml.dictionary.term(pk)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/tools.py", line 15, in term
    dictionary = Dictionary()
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 47, in __call__
    cls._instance = load_dictionary_from_cache(version)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/version.py", line 290, in load_dictionary_from_cache
    return pickle.load(fp)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 182, in __setstate__
    self._populate(scripts=state['scripts'], relations=state['relations'])
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 146, in _populate
    root = script(root)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/script/operator.py", line 18, in script
    s = ScriptParser().parse(arg)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/script/parser/parser.py", line 35, in t_parse
    raise CannotParse(s, str(e))
ieml.exceptions.CannotParse: Unable to parse the following string 'f.o.-f.o.-',n.i.-f.i.-',x.-O:.-',_M:.-',_;+f.o.-f.o.-',n.i.-f.i.-',x.-O:.-',_E:F:.-',_;'. Invalid arguments to create a script. Syntax error at 'n' (1, 12)
Internal Server Error: /dictionary/
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/script/parser/parser.py", line 33, in t_parse
    return self.parser.parse(s, lexer=self.lexer)
  File "/usr/lib/python3.6/site-packages/ply/yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/usr/lib/python3.6/site-packages/ply/yacc.py", line 1199, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/usr/lib/python3.6/site-packages/ply/yacc.py", line 193, in call_errorfunc
    r = errorfunc(token)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/script/parser/parser.py", line 43, in p_error
    raise InvalidScript(msg)
ieml.exceptions.InvalidScript: Invalid arguments to create a script. Syntax error at '.' (1, 13)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rest_framework/viewsets.py", line 86, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/django_intlekt-0.3.1-py3.6.egg/django_intlekt/views/dictionary.py", line 292, in list
    data = serializers.TermSerializer(Dictionary(), many=True).data
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 47, in __call__
    cls._instance = load_dictionary_from_cache(version)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/version.py", line 290, in load_dictionary_from_cache
    return pickle.load(fp)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 182, in __setstate__
    self._populate(scripts=state['scripts'], relations=state['relations'])
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 146, in _populate
    root = script(root)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/script/operator.py", line 18, in script
    s = ScriptParser().parse(arg)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/script/parser/parser.py", line 35, in t_parse
    raise CannotParse(s, str(e))
ieml.exceptions.CannotParse: Unable to parse the following string 'f.o.-f.o.-',n.i.-f.i.-',x.-O:.-',_M:.-',_;+f.o.-f.o.-',n.i.-f.i.-',x.-O:.-',_E:F:.-',_;'. Invalid arguments to create a script. Syntax error at '.' (1, 13)
[22/Aug/2017 14:38:31] "GET /dictionary/I%3A/tables/ HTTP/1.1" 500 17000
[22/Aug/2017 14:38:31] "GET /dictionary/ HTTP/1.1" 500 16978
[22/Aug/2017 14:38:31] "GET /dictionary/I%3A/relations/ HTTP/1.1" 500 17116
Internal Server Error: /dictionary/projection/
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rest_framework/viewsets.py", line 86, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/django_intlekt-0.3.1-py3.6.egg/django_intlekt/views/dictionary.py", line 386, in projection
    usl = ieml.usl.tools.usl(post.usl.ieml)
  File "/usr/lib/python3.6/site-packages/ieml/usl/tools.py", line 21, in usl
    return USLParser().parse(arg)
  File "/usr/lib/python3.6/site-packages/ieml/usl/parser/parser.py", line 28, in parse
    return self.parser.parse(s, lexer=self.lexer)
  File "/usr/lib/python3.6/site-packages/ply/yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/usr/lib/python3.6/site-packages/ply/yacc.py", line 1118, in parseopt_notrack
    p.callable(pslice)
  File "/usr/lib/python3.6/site-packages/ieml/usl/parser/parser.py", line 42, in p_usl
    p[0] = Usl(IEMLParser().parse(p[1]))
  File "/usr/lib/python3.6/site-packages/ieml/syntax/parser/parser.py", line 53, in parse
    return self.parser.parse(s, lexer=self.lexer)
  File "/usr/lib/python3.6/site-packages/ply/yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/usr/lib/python3.6/site-packages/ply/yacc.py", line 1118, in parseopt_notrack
    p.callable(pslice)
  File "/usr/lib/python3.6/site-packages/ieml/syntax/parser/parser.py", line 97, in p_script
    p[0] = _build(term(p[1]))
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/tools.py", line 15, in term
    dictionary = Dictionary()
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 47, in __call__
    cls._instance = load_dictionary_from_cache(version)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/version.py", line 290, in load_dictionary_from_cache
    return pickle.load(fp)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 182, in __setstate__
    self._populate(scripts=state['scripts'], relations=state['relations'])
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 154, in _populate
    roots[root_ss[s.singular_sequences[0]]].append(s)

Can't post request to server on ieml_django

{
"authors": [],
 "created_on": null,
 "keywords": [],
 "language": "en",
 "title": "Indian stock investment Android app – MarketSmit India",
 "url": "https://marketsmithindia.com/mstool/investment-advisory-special-offers/seminar.html#/"
}

Make a coffee for the scrum master

Enjoy!

Freeing Terms.dict when their are used by usls when switching dictionary

Only one version of the dictionary can live in memory, when switching versions, the old terms are released to free memory. But they can be used by a Morphem in a usl and raise an exception.

Possible fix: make the terms "Global" objects with a mapping {dictionaryVersion -> {script, translation, relations, index, ...}. This allows multiples versions of the dictionary to live in memory and share the same pool of terms.
The translation of an usl between differents versions is then implicit.

Error generating table with the table endpoint

script_table('wa.F:F:F:.-')

Add a state field to posts to know if the document url still exists (404)

Some scoop posts have old urls that don't exist anymore, server returns 404. For now, no post is created for those urls but we could still create the post and add a state field that will give the information to the user that some of his documents were not retrieved because the url is not good anymore, giving him the information that his collection is not complete and that he should update the url if he still wants the document.

Remove self from relations.contained and relations.contains

USL translation uniqueness

Make sure two IEML words cannot have the same translation. For instance, this is not possible:

A+B -> joie
A+C -> joie

Look at slugs.

Leaves of Script trees (layer 0) have children

voir O:.E:M:.-

O:O:.O:O:.- is not inhibited in the relations collection but have an inhibition in the terms collection.

Url not properly encoded when posted to mongo

I have to fix the url encoder of the scraper so that all future problems are dealt with.

make gramatical_class on term return grammatical class name and not the class id

Store all parameters in forms

Currently, some parameters are stored in JS or HTML. For instance:

https://github.com/IEMLdev/ieml-api/blob/collection_models_django/intlekt/django/intlekt/collections_/static/collections_/ui.js#L337

https://github.com/IEMLdev/ieml-api/blob/collection_models_django/intlekt/django/intlekt/collections_/static/collections_/ui.js#L577

To have independent forms and to be able to send them without using the api, all data should be stored in fields.

Add form to edit a document

The button to display the form can appear in related post forms.

Loading the dictionary multiple times makes the server slow

Too much RAM is used and the server becomes really slow. Maybe do not load it if it is already in memory.

New AST not comparing o.O:M:.- and E:O:.T:M:.- correctly

database entries for both:

/* 1 */
{
    "_id" : ObjectId("55d220dc6653c32453c0a58e"),
    "CANONICAL" : [ 
        "adaghaaaa"
    ],
    "CLASS" : "1",
    "EN" : "low",
    "FR" : "bas",
    "IEML" : "E:O:.T:M:.-",
    "LAYER" : "2",
    "PARADIGM" : "0",
    "TAILLE" : "6"
}

/* 2 */
{
    "_id" : ObjectId("55d220dc6653c32453c0a59b"),
    "CANONICAL" : [ 
        "bfadhaaaa"
    ],
    "CLASS" : "2",
    "EN" : "want to perform",
    "FR" : "vouloir performer",
    "IEML" : "o.O:M:.-",
    "LAYER" : "2",
    "PARADIGM" : "0",
    "TAILLE" : "6"
}

It's probably the E and U comparison that's wrong

rename ieml.dictionary.script.operator

Add collection API documentation

With model docstrings and Field.help_text attribute.

Remove print calls

Not persistent, should use logs instead.

Moreover, raise an error on AWS as the Python process is not bound to a terminal.

Need a language field and a keyword field in the mongodb document collection

We need to add a language string field to store the detected language of the document as well as a keywords field that is list of strings to store the detected keywords of the document that will allow us to make tag suggestions in the future.

Add possibility to manage tag texts

From instance, if two automatically collected documents have the tags math and mathematics, we probably want them to refer to the same tag in the database.

Possibility to rename a tag.

@ogrergo I need to talk with you boy! Should we manage tags like we do for USLs?

Exception for invalid term

>>> ieml.tools.ieml("[u.-d.u.-'])]")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vincent/miniconda3/lib/python3.6/site-packages/ieml/tools.py", line 18, in ieml
    return IEMLParser().parse(arg)
  File "/home/vincent/miniconda3/lib/python3.6/site-packages/ieml/syntax/parser/parser.py", line 49, in parse
    self.parser.parse(s, lexer=self.lexer)
  File "/home/vincent/miniconda3/lib/python3.6/site-packages/ply/yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/home/vincent/miniconda3/lib/python3.6/site-packages/ply/yacc.py", line 1118, in parseopt_notrack
    p.callable(pslice)
  File "/home/vincent/miniconda3/lib/python3.6/site-packages/ieml/syntax/parser/parser.py", line 82, in p_term
    p[0] = _build(Dictionary().terms[p[2]])
KeyError: "u.-d.u.-'"

Should raise a custom exception.

Get cached dictionary version

get_default_dictionary_version loads the last version of the dictionary if set_default_dictionary_version has not been called. We should be able to retrieve the version in the cache.

Wrong factorization for "animal and vegetables"

Create a function to project posts on tables

Just move this function here: https://github.com/IEMLdev/django-intlekt/blob/master/django_intlekt/views/dictionary.py#L268

Internal Server Error: /dictionary/I:/tables/

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rest_framework/viewsets.py", line 86, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/django_intlekt-0.3.1-py3.6.egg/django_intlekt/views/dictionary.py", line 267, in wrapper
    ieml.dictionary.term(pk)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/tools.py", line 15, in term
    dictionary = Dictionary()
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 47, in __call__
    cls._instance = load_dictionary_from_cache(version)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/version.py", line 290, in load_dictionary_from_cache
    return pickle.load(fp)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 182, in __setstate__
    self._populate(scripts=state['scripts'], relations=state['relations'])
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 154, in _populate
    roots[root_ss[s.singular_sequences[0]]].append(s)
KeyError: <ieml.dictionary.script.script.MultiplicativeScript object at 0x7f014b068828>
Internal Server Error: /dictionary/
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rest_framework/viewsets.py", line 86, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/django_intlekt-0.3.1-py3.6.egg/django_intlekt/views/dictionary.py", line 292, in list
    data = serializers.TermSerializer(Dictionary(), many=True).data
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 47, in __call__
    cls._instance = load_dictionary_from_cache(version)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/version.py", line 290, in load_dictionary_from_cache
    return pickle.load(fp)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 182, in __setstate__
    self._populate(scripts=state['scripts'], relations=state['relations'])
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 154, in _populate
    roots[root_ss[s.singular_sequences[0]]].append(s)
KeyError: <ieml.dictionary.script.script.MultiplicativeScript object at 0x7f013bd87be0>
Internal Server Error: /dictionary/I:/relations/
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rest_framework/viewsets.py", line 86, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/lib/python3.6/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/django_intlekt-0.3.1-py3.6.egg/django_intlekt/views/dictionary.py", line 267, in wrapper
    ieml.dictionary.term(pk)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/tools.py", line 15, in term
    dictionary = Dictionary()
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 47, in __call__
    cls._instance = load_dictionary_from_cache(version)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/version.py", line 290, in load_dictionary_from_cache
    return pickle.load(fp)
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 182, in __setstate__
    self._populate(scripts=state['scripts'], relations=state['relations'])
  File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py", line 154, in _populate
    roots[root_ss[s.singular_sequences[0]]].append(s)
KeyError: <ieml.dictionary.script.script.MultiplicativeScript object at 0x7f014ce9e240>

Avoid duplicates in collection documents

Currently, the collection only has a list of document ids. It should be a set, to avoid duplicates.

Create Python packages

https://docs.djangoproject.com/en/1.11/intro/reusable-apps/

@ogrergo

Fix test distance

Cleanup the rels handler

remove the exists, visible ...

Headers for certain script are not valid when 3D.

M:M:.-O:M:.-E:.-+s.y.-'+M:M:.-M:O:.-E:.-+s.y.-'

Add endpoints for saving/editing/removing/query usls

Correct the rank algortihm

IEML from Rules returns valid when given rules with 1 term

I sent the following rules to the API with the /usls/from_rules endpoint
{rules: {r0: "U:"}}

and it returned no errors, but when I go to save the word, it gives me: "This word must not be a term."

Merge Additive children in additive root node for Script tree

Collection title not unique

Only unique for a user.

Bug in usls/from_rules/ endpoint

POST /usls/from_rules/
{
    "rules": {
        "r0": "A:O:.wo.t.-",
        "r1": "d.a.-l.a.-f.o.-'",
        "r2": "m.-M:.O:.-'m.-S:.U:.-'E:A:S:.-',",
        "f0": "b.o.-k.o.-s.u.-'",
        "f1": "n.u.-d.u.-d.u.-'"
    }
}

Error:

IEMLObjectResolutionError("Invalid ieml object definitions at:\n\t[] No compatible type found with the path r0 and the ieml object of type Table1D")

raised by ieml.tools.usl(rules)

Relations endpoints

rename the relations endpoints
order the relations entries
add a parameter to factorize the relations in their root paradigm.

Use date and not datetime for document creation date

Make the convertion with DRF.

Add possibility to add/remove USLs to tags

Tags are duplicated in mongodb

The list of tags is duplicated on mongodb, they are most probably duplicated when retrieved which could mean that everything is read twice.

Error on f.o.-f.o.-',n.i.-f.i.-',M:O:.-O:.-',_

IndexError at /dictionary/f.o.-f.o.-',n.i.-f.i.-',M:O:.-O:.-',_/tables/↵index 2 is out of bounds for axis 0 with size 2↵↵Request Method: GET↵Request URL: http://localhost:8000/dictionary/f.o.-f.o.-',n.i.-f.i.-',M:O:.-O:.-',_/tables/↵Django Version: 1.11.2↵Python Executable: /usr/bin/python↵Python Version: 3.6.2↵Python Path: ['/home/piggygenius/Studies/Ensimag/2A/Internship/django_project', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/lib/python3.6/site-packages', '/usr/lib/python3.6/site-packages/django_intlekt-0.3.1-py3.6.egg']↵Server time: Tue, 29 Aug 2017 20:55:38 +0000↵Installed Applications:↵['django.contrib.admin',↵ 'django.contrib.auth',↵ 'django.contrib.contenttypes',↵ 'django.contrib.sessions',↵ 'django.contrib.messages',↵ 'django.contrib.staticfiles',↵ 'rest_framework',↵ 'rest_framework_mongoengine',↵ 'corsheaders',↵ 'django_intlekt']↵Installed Middleware:↵['django.middleware.security.SecurityMiddleware',↵ 'django.contrib.sessions.middleware.SessionMiddleware',↵ 'django.middleware.common.CommonMiddleware',↵ 'django.middleware.csrf.CsrfViewMiddleware',↵ 'django.contrib.auth.middleware.AuthenticationMiddleware',↵ 'django.contrib.messages.middleware.MessageMiddleware',↵ 'django.middleware.clickjacking.XFrameOptionsMiddleware',↵ 'corsheaders.middleware.CorsMiddleware',↵ 'django.middleware.common.CommonMiddleware']↵↵↵Traceback: ↵↵File "/usr/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner↵ 41. response = get_response(request)↵↵File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response↵ 187. response = self.process_exception_by_middleware(e, request)↵↵File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response↵ 185. response = wrapped_callback(request, *callback_args, callback_kwargs)↵↵File "/usr/lib/python3.6/site-packages/django/views/decorators/csrf.py" in wrapped_view↵ 58. return view_func(*args, kwargs)↵↵File "/usr/lib/python3.6/site-packages/rest_framework/viewsets.py" in view↵ 86. return self.dispatch(request, args, **kwargs)↵↵File "/usr/lib/python3.6/site-packages/rest_framework/views.py" in dispatch↵ 489. response = self.handle_exception(exc)↵↵File "/usr/lib/python3.6/site-packages/rest_framework/views.py" in handle_exception↵ 449. self.raise_uncaught_exception(exc)↵↵File "/usr/lib/python3.6/site-packages/rest_framework/views.py" in dispatch↵ 486. response = handler(request, args, **kwargs)↵↵File "/usr/lib/python3.6/site-packages/django_intlekt-0.3.1-py3.6.egg/django_intlekt/views/dictionary.py" in wrapper↵ 272. return f(self, request, pk=pk, format=format)↵↵File "/usr/lib/python3.6/site-packages/django_intlekt-0.3.1-py3.6.egg/django_intlekt/views/dictionary.py" in tables↵ 341. return Response(get_tables_for_term(pk))↵↵File "/usr/lib/python3.6/site-packages/django_intlekt-0.3.1-py3.6.egg/django_intlekt/views/dictionary.py" in get_tables_for_term↵ 197. return __build_parallel_table(main, others, _get_rel_table_of(main), selected_cell=selected_cell)↵↵File "/usr/lib/python3.6/site-packages/django_intlekt-0.3.1-py3.6.egg/django_intlekt/views/dictionary.py" in __build_parallel_table↵ 96. ] for i in range(table.shape[0])↵↵File "/usr/lib/python3.6/site-packages/django_intlekt-0.3.1-py3.6.egg/django_intlekt/views/dictionary.py" in ↵ 96. ] for i in range(table.shape[0])↵↵File "/usr/lib/python3.6/site-packages/django_intlekt-0.3.1-py3.6.egg/django_intlekt/views/dictionary.py" in ↵ 95. } for j in range(table.shape[1])↵↵File "/usr/lib/python3.6/site-packages/django_intlekt-0.3.1-py3.6.egg/django_intlekt/views/dictionary.py" in ↵ 94. 'others': [_term_entry(tab[i, j]) for tab in parallel_terms] + others_rel(table[i, j])↵↵File "/usr/lib/python3.6/site-packages/ieml/dictionary/table.py" in __getitem__↵ 74. return self.cells[item]↵↵Exception Type: IndexError at /dictionary/f.o.-f.o.-',n.i.-f.i.-',M:O:.-O:.-',/tables/↵Exception Value: index 2 is out of bounds for axis 0 with size 2↵Request information:↵USER: AnonymousUser↵↵GET: No GET data↵↵POST: No POST data↵↵FILES: No FILES data↵↵COOKIES: No cookie data↵↵META:↵COLORTERM = 'gnome-terminal'↵CONTENT_LENGTH = ''↵CONTENT_TYPE = 'text/plain'↵CUDA_HOME = '/usr/local/cuda'↵DBUS_SESSION_BUS_ADDRESS = 'unix:path=/run/user/1000/bus'↵DESKTOP_SESSION = 'xfce'↵DISPLAY = ':0.0'↵DJANGO_SETTINGS_MODULE = 'django_project.settings'↵GATEWAY_INTERFACE = 'CGI/1.1'↵GLADE_CATALOG_PATH = ':'↵GLADE_MODULE_PATH = ':'↵GLADE_PIXMAP_PATH = ':'↵HISTCONTROL = 'ignoreboth:erasedups'↵HISTFILESIZE = '10000'↵HISTSIZE = '500'↵HOME = '/home/piggygenius'↵HTTP_ACCEPT = '/'↵HTTP_ACCEPT_ENCODING = 'gzip, deflate, br'↵HTTP_ACCEPT_LANGUAGE = 'en-US,en;q=0.8,fr;q=0.6'↵HTTP_CONNECTION = 'keep-alive'↵HTTP_HOST = 'localhost:8000'↵HTTP_ORIG… ('es-ni', 'Nicaraguan Spanish'), ('es-ve', 'Venezuelan Spanish'), ('et', 'Estonian'), ('eu', 'Basque'), ('fa', 'Persian'), ('fi', 'Finnish'), ('fr', 'French'), ('fy', 'Frisian'), ('ga', 'Irish'), ('gd', 'Scottish Gaelic'), ('gl', 'Galician'), ('he', 'Hebrew'), ('hi', 'Hindi'), ('hr', 'Croatian'), ('hsb', 'Upper Sorbian'), ('hu', 'Hungarian'), ('ia', 'Interlingua'), ('id', 'Indonesian'), ('io', 'Ido'), ('is', 'Icelandic'), ('it', 'Italian'), ('ja', 'Japanese'), ('ka', 'Georgian'), ('kk', 'Kazakh'), ('km', 'Khmer'), ('kn', 'Kannada'), ('ko', 'Korean'), ('lb', 'Luxembourgish'), ('lt', 'Lithuanian'), ('lv', 'Latvian'), ('mk', 'Macedonian'), ('ml', 'Malayalam'), ('mn', 'Mongolian'), ('mr', 'Marathi'), ('my', 'Burmese'), ('nb', 'Norwegian Bokmål'), ('ne', 'Nepali'), ('nl', 'Dutch'), ('nn', 'Norwegian Nynorsk'), ('os', 'Ossetic'), ('pa', 'Punjabi'), ('pl', 'Polish'), ('pt', 'Portuguese'), ('pt-br', 'Brazilian Portuguese'), ('ro', 'Romanian'), ('ru', 'Russian'), ('sk', 'Slovak'), ('sl', 'Slovenian'), ('sq', 'Albanian'), ('sr', 'Serbian'), ('sr-latn', 'Serbian Latin'), ('sv', 'Swedish'), ('sw', 'Swahili'), ('ta', 'Tamil'), ('te', 'Telugu'), ('th', 'Thai'), ('tr', 'Turkish'), ('tt', 'Tatar'), ('udm', 'Udmurt'), ('uk', 'Ukrainian'), ('ur', 'Urdu'), ('vi', 'Vietnamese'), ('zh-hans', 'Simplified Chinese'), ('zh-hant', 'Traditional Chinese')]↵LANGUAGES_BIDI = ['he', 'ar', 'fa', 'ur']↵LANGUAGE_CODE = 'en-us'↵LANGUAGE_COOKIE_AGE = None↵LANGUAGE_COOKIE_DOMAIN = None↵LANGUAGE_COOKIE_NAME = 'django_language'↵LANGUAGE_COOKIE_PATH = '/'↵LOCALE_PATHS = []↵LOGGING = {}↵LOGGING_CONFIG = 'logging.config.dictConfig'↵LOGIN_REDIRECT_URL = '/accounts/profile/'↵LOGIN_URL = '/accounts/login/'↵LOGOUT_REDIRECT_URL = None↵MANAGERS = []↵MEDIA_ROOT = ''↵MEDIA_URL = ''↵MESSAGE_STORAGE = 'django.contrib.messages.storage.fallback.FallbackStorage'↵MIDDLEWARE = ['django.middleware.security.SecurityMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'corsheaders.middleware.CorsMiddleware', 'django.middleware.common.CommonMiddleware']↵MIDDLEWARE_CLASSES = ['django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware']↵MIGRATION_MODULES = {}↵MONTH_DAY_FORMAT = 'F j'↵NUMBER_GROUPING = 0↵PASSWORD_HASHERS = ''↵PASSWORD_RESET_TIMEOUT_DAYS = ''↵PREPEND_WWW = False↵REST_FRAMEWORK = {'DEFAULT_PAGINATION_CLASS': 'django_intlekt.pagination.PageNumberPagination', 'PAGE_SIZE': 100}↵ROOT_URLCONF = 'django_project.urls'↵SECRET_KEY = '****************'↵SECURE_BROWSER_XSS_FILTER = False↵SECURE_CONTENT_TYPE_NOSNIFF = False↵SECURE_HSTS_INCLUDE_SUBDOMAINS = False↵SECURE_HSTS_PRELOAD = False↵SECURE_HSTS_SECONDS = 0↵SECURE_PROXY_SSL_HEADER = None↵SECURE_REDIRECT_EXEMPT = []↵SECURE_SSL_HOST = None↵SECURE_SSL_REDIRECT = False↵SERVER_EMAIL = 'root@localhost'↵SESSION_CACHE_ALIAS = 'default'↵SESSION_COOKIE_AGE = 1209600↵SESSION_COOKIE_DOMAIN = None↵SESSION_COOKIE_HTTPONLY = True↵SESSION_COOKIE_NAME = 'sessionid'↵SESSION_COOKIE_PATH = '/'↵SESSION_COOKIE_SECURE = False↵SESSION_ENGINE = 'django.contrib.sessions.backends.db'↵SESSION_EXPIRE_AT_BROWSER_CLOSE = False↵SESSION_FILE_PATH = None↵SESSION_SAVE_EVERY_REQUEST = False↵SESSION_SERIALIZER = 'django.contrib.sessions.serializers.JSONSerializer'↵SETTINGS_MODULE = 'django_project.settings'↵SHORT_DATETIME_FORMAT = 'm/d/Y P'↵SHORT_DATE_FORMAT = 'm/d/Y'↵SIGNING_BACKEND = 'django.core.signing.TimestampSigner'↵SILENCED_SYSTEM_CHECKS = []↵STATICFILES_DIRS = []↵STATICFILES_FINDERS = ['django.contrib.staticfiles.finders.FileSystemFinder', 'django.contrib.staticfiles.finders.AppDirectoriesFinder']↵STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage'↵STATIC_ROOT = None↵STATIC_URL = '/static/'↵TEMPLATES = [{'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [], 'APP_DIRS': True, 'OPTIONS': {'context_processors': ['django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages']}}]↵TEST_NON_SERIALIZED_APPS = []↵TEST_RUNNER = 'django.test.runner.DiscoverRunner'↵THOUSAND_SEPARATOR = ','↵TIME_FORMAT = 'P'↵TIME_INPUT_FORMATS = ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M']↵TIME_ZONE = 'UTC'↵USE_ETAGS = False↵USE_I18N = True↵USE_L10N = True↵USE_THOUSAND_SEPARATOR = False↵USE_TZ = True↵USE_X_FORWARDED_HOST = False↵USE_X_FORWARDED_PORT = False↵WSGI_APPLICATION = 'django_project.wsgi.application'↵X_FRAME_OPTIONS = 'SAMEORIGIN'↵YEAR_MONTH_FORMAT = 'F Y'↵↵↵You're seeing this error because you have DEBUG = True in your↵Django settings file. Change that to False, and Django will↵display a standard page generated by the handler for this status code.↵↵

Add USL hint

If all tags have one and only one USL: concatenate
Else if at least one tag has no USLs: concatenate and tell there are missing translations
Else if at least one tag has multiple USLs: add possibility to choose one usl then concatenate

Error when creating a new paradigm

Traceback (most recent call last):
File "/home/ubuntu/envs/dictionary/lib/python3.5/site-packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/home/ubuntu/envs/dictionary/lib/python3.5/site-packages/django/core/handlers/base.py", line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/home/ubuntu/envs/dictionary/lib/python3.5/site-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/ubuntu/envs/dictionary/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/home/ubuntu/envs/dictionary/lib/python3.5/site-packages/django/views/generic/base.py", line 68, in view
return self.dispatch(request, *args, **kwargs)
File "/home/ubuntu/envs/dictionary/lib/python3.5/site-packages/rest_framework/views.py", line 489, in dispatch
response = self.handle_exception(exc)
File "/home/ubuntu/envs/dictionary/lib/python3.5/site-packages/rest_framework/views.py", line 449, in handle_exception
self.raise_uncaught_exception(exc)
File "/home/ubuntu/envs/dictionary/lib/python3.5/site-packages/rest_framework/views.py", line 486, in dispatch
response = handler(request, *args, **kwargs)
File "./dictionary/views.py", line 451, in post
new_version = create_dictionary_version(DictionaryVersion.from_file_name(version), add=to_add)
AttributeError: type object 'DictionaryVersion' has no attribute 'from_file_name'

Everything fails

Traceback:  

File "/usr/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  41.             response = get_response(request)

File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  187.                 response = self.process_exception_by_middleware(e, request)

File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/usr/lib/python3.6/site-packages/django/views/decorators/csrf.py" in wrapped_view
  58.         return view_func(*args, **kwargs)

File "/usr/lib/python3.6/site-packages/rest_framework/viewsets.py" in view
  86.             return self.dispatch(request, *args, **kwargs)

File "/usr/lib/python3.6/site-packages/rest_framework/views.py" in dispatch
  489.             response = self.handle_exception(exc)

File "/usr/lib/python3.6/site-packages/rest_framework/views.py" in handle_exception
  449.             self.raise_uncaught_exception(exc)

File "/usr/lib/python3.6/site-packages/rest_framework/views.py" in dispatch
  486.             response = handler(request, *args, **kwargs)

File "/usr/lib/python3.6/site-packages/django_intlekt-0.3.1-py3.6.egg/django_intlekt/views/dictionary.py" in list
  292.         data = serializers.TermSerializer(Dictionary(), many=True).data

File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py" in __call__
  47.                 cls._instance = load_dictionary_from_cache(version)

File "/usr/lib/python3.6/site-packages/ieml/dictionary/version.py" in load_dictionary_from_cache
  290.         return pickle.load(fp)

File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py" in __setstate__
  182.         self._populate(scripts=state['scripts'], relations=state['relations'])

File "/usr/lib/python3.6/site-packages/ieml/dictionary/dictionary.py" in _populate
  154.             roots[root_ss[s.singular_sequences[0]]].append(s)

Exception Type: KeyError at /dictionary/
Exception Value: <ieml.dictionary.script.script.MultiplicativeScript object at 0x7f785d710b00>
Request information:
USER: AnonymousUser

Paradigmatic table for I: isn't working

Je sais pas si c'est normal ou pas, mais ça fait planter le serveur, donc faut gérer ce cas là.

Article publish date is often None

The newspaper module seems to be working well except for the publication date that is often null, we could add the scoop publication date when the module can't find the correct date to at least have an indication. To be decided.