letuananh / chirptext

ChirpText is a collection of text processing tools for Python.

Hi there 👋

Le Tuan Anh is a semanticist who is deeply interested in well-being, languages, and free software. His main research interests are in computational meaning and theory of language. He is the author of the coolisf software, which provides computational deep semantic analysis by combining structural semantics from construction grammars and lexical semantics from ontologies in a single representation. The dissertation (2019), source code, and data are available on the Open Science Framework.

In his free time, Tuan Anh develops jamdict - a free and open-source Python 3 library for manipulating Japanese linguistic resources: Jim Breen’s JMdict, KanjiDic2, JMnedict, and kanji-radical mappings. There is an online demo available at: https://jamdict.herokuapp.com/

From Mar 2019 to Apr 2022, Tuan Anh was a research fellow at Nanyang Technological University (NTU), Singapore and contributed to the development of BELA - a pathway for creating, checking, visualizing, and analysing multilingual corpuses of natural languages.

chirptext's Issues

Update documentation about new deko API

Revamp TTL APIs for more complex usecases

• simplify multi-tag handling (i.e. sense candidates, chunk languages, annotators, etc.)
• Built-in support for CoNLL
• use first tag slot for scalar tags (i.e. POS, lemma, surface, languages)
• Re-design TTL JSON

FileNotFoundError [WinError 2] The system cannot find the file specified

>>> from chirptext import deko
>>> sent = deko.parse('猫が好きです。')
>>> sent.tokens
===========================
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-15-bd2eed5e1672> in <module>
      1 from chirptext import deko
----> 2 sent = deko.parse('猫が好きです。')
      3 sent.tokens

~\Anaconda3\lib\site-packages\chirptext\deko.py in txt2mecab(text, **kwargs)
    250 def txt2mecab(text, **kwargs):
    251     ''' Use mecab to parse one sentence '''
--> 252     mecab_out = _internal_mecab_parse(text, **kwargs).splitlines()
    253     tokens = [MeCabToken.parse(x) for x in mecab_out]
    254     return MeCabSent(text, tokens)

~\Anaconda3\lib\site-packages\chirptext\dekomecab.py in parse(content, *args, **kwargs)
     65         return MeCab.Tagger(*args).parse(content)
     66     else:
---> 67         return run_mecab_process(content, *args, **kwargs)
     68 
     69 

~\Anaconda3\lib\site-packages\chirptext\dekomecab.py in run_mecab_process(content, *args, **kwargs)
     55     output = subprocess.run(proc_args,
     56                             input=content.encode(encoding),
---> 57                             stdout=subprocess.PIPE)
     58     output_string = os.linesep.join(output.stdout.decode(encoding).splitlines())
     59     return output_string

~\Anaconda3\lib\subprocess.py in run(input, capture_output, timeout, check, *popenargs, **kwargs)
    464         kwargs['stderr'] = PIPE
    465 
--> 466     with Popen(*popenargs, **kwargs) as process:
    467         try:
    468             stdout, stderr = process.communicate(input, timeout=timeout)

~\Anaconda3\lib\subprocess.py in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors, text)
    767                                 c2pread, c2pwrite,
    768                                 errread, errwrite,
--> 769                                 restore_signals, start_new_session)
    770         except:
    771             # Cleanup if the child failed starting.

~\Anaconda3\lib\subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, unused_restore_signals, unused_start_new_session)
   1170                                          env,
   1171                                          os.fspath(cwd) if cwd is not None else None,
-> 1172                                          startupinfo)
   1173             finally:
   1174                 # Child is launched. Close the parent's copy of those pipe

FileNotFoundError: [WinError 2] The system cannot find the file specified

Make sure that there is no hard-coded logging to INFO in any file

use TTL 2.0 for all deko-* parses

• dekomecab
• dekoigo
• dekohebi

add chunk support to TTL for BELA

Chunks are

• taggable (i.e. languages, fonts, etc.)
• optional links to tokens (or inferred from (cfrom, cto)
• Ref: https://blipntu.github.io/belacon/

Add dialect support to TextReport

• Markdown
• RST
• org-mode
• HTML?
• LaTeX?

chio module, move mode before encoding

Reason: encoding is almost always UTF-8 but default file access mode is changed quite often

review ttl.Sentence.fix_cfrom_cto() function

review ttl.Sentence.fix_cfrom_cto() function

Determine whether to lose, migrate somewhere else, or refactor it for more general cases?

Release chirptext 0.1

Asking for a new release on PyPi

Hi,

Version 0.1a18 is a bit outdated, could you update a newer version to PyPi?

I only need this commit, but since long time is passed i think most of the master change are stable.

Consider to port back AppConfig.potentials() from version 0.2

chirptext/chirptext/leutile.py

Line 613 in 12a8997

LOC_TEMPLATE = ['{wd}/.{n}.{mode}', '{wd}/{n}.{mode}',

support more Japanese parsers

Order of engines to use

1. mecab
2. janome
3. igo-python

failed to import HIRAGANA and KATAKANA from chirptext.deko

This code from jamdict failed to run with chirptext 0.2a2

from chirptext.deko import HIRAGANA, KATAKANA
...

Add support for Leipzig and Penn Treebank tagset

• Leipzig
  • Reference: https://www.eva.mpg.de/lingua/resources/glossing-rules.php
• Penn Treebank tag set
  • Version 1: https://www.sketchengine.eu/penn-treebank-tagset/
  • Version 2: https://www.sketchengine.eu/english-treetagger-pipeline-2

add encoding keyword to chio.write_csv and chio.read_csv