kunansy / rnc

API for Russian National Corpus

Kirill kunansy

rnc's Issues

Add docs to examples to README

Use ABC

Make Corpus and Example classes abstract.

Del example by index

Create special exception classes

Findall and finditer

Implement these methods, use generators

Update setting a language in parallel corpus

Now it set as mode=para-eng/fra/ger etc.

Change email address

Query with gram params doesn't work

Corpus returns 'Not found...' if the query is a dict.
But there're results, corp.open_url() shows the page with them.

Pages

Задать через класм Pages. количество страниц: ≤, все, случайная страница

Specify requirements' versions

Use generators in page parsers

Add log message

RNC/rnc/examples.py

Line 735 in 2a3e5a4

try:

Use namedtuples where it's possible

Almanac

Here the are some minor changes, fixes and improvements

• Use Path.mkdir(exist_ok=True) instead of

  RNC/rnc/__init__.py

  Line 47 in 0ef7f28

  os.makedirs(LOG_FOLDER, exist_ok=True)

Remove some useless validating requests to RNC

[ERROR] [corpora_requests:whether_result_found] The request is not correct

bug
[ERROR] [corpora_requests:whether_result_found] The request is not correct

code

    import rnc
    
    ru = rnc.MainCorpus(
        query='корпус', 
        p_count=5,
        marker=str.upper)
    
    ru.request_examples()

logs

    [28.03.2023 15:24:07,632] [DEBUG] [corpora_requests:is_request_correct] Validating that everything is OK
    [28.03.2023 15:24:07,634] [DEBUG] [corpora_requests:whether_result_found] Validating that the request is OK
    [28.03.2023 15:24:07,635] [INFO] [corpora_requests:get_htmls] Requested to 'https://processing.ruscorpora.ru/search.xml' [0;1) with params {'env': 'alpha', 'api': '1.0', 'lang': 'en', 'dpp': 5, 'spd': 10, 'text': 'lexgramm', 'out': 'normal', 'sort': 'i_grtagging', 'nodia': 1, 'lex1': 'корпус', 'mode': 'main'}
    [28.03.2023 15:24:07,636] [ERROR] [corpora_requests:whether_result_found] The request is not correct: {'env': 'alpha', 'api': '1.0', 'lang': 'en', 'dpp': 5, 'spd': 10, 'text': 'lexgramm', 'out': 'normal', 'sort': 'i_grtagging', 'nodia': 1, 'lex1': 'корпус', 'mode': 'main'}
    [28.03.2023 15:24:07,636] [ERROR] [corpora_requests:is_request_correct] HTTP request is wrong
    [28.03.2023 15:24:07,637] [ERROR] [corpora:request_examples] Query = ['корпус'], 5, {'env': 'alpha', 'api': '1.0', 'lang': 'en', 'dpp': 5, 'spd': 10, 'text': 'lexgramm', 'out': 'normal', 'sort': 'i_grtagging', 'nodia': 1, 'lex1': 'корпус', 'mode': 'main'}
    e = {'env': 'alpha', 'api': '1.0', 'lang': 'en', 'dpp': 5, 'spd': 10, 'text': 'lexgramm', 'out': 'normal', 'sort': 'i_grtagging', 'nodia': 1, 'lex1': 'корпус', 'mode': 'main'}
    Output exceeds the [size limit](command:workbench.action.openSettings?[). Open the full output data [in a text editor](command:workbench.action.openLargeOutput?df6b87e8-4758-497a-a54d-7a456297814e)
    ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    File ~/.conda-envs/venv2/lib/python3.8/site-packages/rnc/corpora_requests.py:200, in whether_result_found(url, **kwargs)
        199 try:
    --> 200     page_html = get_htmls(url, **kwargs)[0]
        201 except Exception:
    
    File ~/.conda-envs/venv2/lib/python3.8/site-packages/rnc/corpora_requests.py:164, in get_htmls(url, start, stop, **kwargs)
        162 coro_start = time.time()
    --> 164 html_codes = asyncio.run(
        165     get_htmls_coro(url, start, stop, **kwargs)
        166 )
        168 logger.info("Request was successfully completed")
    
    File ~/.conda-envs/venv2/lib/python3.8/asyncio/runners.py:33, in run(main, debug)
         32 if events._get_running_loop() is not None:
    ---> 33     raise RuntimeError(
         34         "asyncio.run() cannot be called from a running event loop")
         36 if not coroutines.iscoroutine(main):
    
    RuntimeError: asyncio.run() cannot be called from a running event loop
    
    During handling of the above exception, another exception occurred:
    
    RuntimeError                              Traceback (most recent call last)
    ...
    --> 294     raise WrongHTTPRequest(f"{kwargs}")
        295 logger.debug("HTTP request is correct, result found")
        297 logger.debug("Validating that the last page exists")
    
    WrongHTTPRequest: {'env': 'alpha', 'api': '1.0', 'lang': 'en', 'dpp': 5, 'spd': 10, 'text': 'lexgramm', 'out': 'normal', 'sort': 'i_grtagging', 'nodia': 1, 'lex1': 'корпус', 'mode': 'main'}

Я так понимаю с текущей версией сайта не библиотека не работает? Ошибка при обрашении к https://processing.ruscorpora.ru/search.xml

Add log when there's 429

Use workers and queue instead of aiojobs

aiojobs uses deprecated args for event_loop. It will be removed in Python 3.10.

Change log messages format

Like
[date time] [level] [line] [message]

Add `flashtext` instead of regexp

Gramm, flags and semantic

RNC should get only this params in query, raise and exception if sth went wrong.

ujson instead of json

Use ujson instead of json, it is quicker

Optimize working with files

Use

... open() as f:
    for line in f:
        ...

Use utf-8 instead of utf-16

Remove censure

Неверное распределение тэг – язык в || корпусе при чтении из файла.

Log to file

Let the user to turn the logging to file off.

Data folder

Do not create folder 'data' while it is not demanded

Semantic properties to query

Request two or more words with one str.

Parsing docs or examples to Multiprocessing

Parsing in MultilingualParaCorpus gets a lot of time.
Which of the parse method to Multiprocessing: parse_doc or parse_example? Profile the project to know.

Add offset

Method `filter` to class Corpus, applying to Example objects.

More tests

Add cases when the result not found, last page doesn't exist etc

Information about text

Get full info about the document

Create class Query

Class Query will be used instead of dict with tags.

Distance between words doesn't set