afpy / padpo

Linter for gettext files (*.po)

padpo

Linter for gettext files (*.po)

Created to help the translation of official Python docs in French: https://github.com/python/python-docs-fr

Il faut demander aux traducteurs s'ils n'ont pas de pot quand ils traduisent, maintenant ils ont padpo… :smile: :laughing: :stuck_out_tongue_winking_eye: :joy: (note : il était tard le soir quand j'ai trouvé le nom).

License

BSD 3-clause

Pull request are welcome.

Padpo is part of poutils!

Poutils (.po utils) is a metapackage to easily install useful Python tools to use with po files and padpo is a part of it! Go check out Poutils to discover the other tools!

Usage

Using the activated virtual environment created during the installation:

For a local input file:

padpo --input-path a_file.po

or for a local input directory:

padpo --input-path a_directory_containing_po_files

or for a pull request in python-docs-fr repository (here pull request #978)

padpo --python-docs-fr 978

or for a pull request in a GitHub repository (here python/python-docs-fr/pull/978)

padpo --github python/python-docs-fr/pull/978

Color

By default, the output is colorless, and formatted like GCC messages. You can use -c or --color option to get a colored output.

Installation

Automatic installation

pip install padpo

Manual installation

1. Install dependencies

   poetry install

   Note: this uses poetry that you can get here: https://poetry.eustace.io/docs/

2. Use virtual environment$

   poetry shell

Update on PyPI

./deliver.sh

Changelog

v0.11.0 (2021-02-02)

• update glossary (fix #58)

v0.10.0 (2020-12-04)

• use pygrammalecte v1.3.0
• use GitHub Actions

v0.9.0 (2020-09-07)

• use pygrammalecte default message for spelling errors

v0.8.0 (2020-08-25)

• use pygrammalecte
• add continuous integration
• fix #12, #13, #14, #15, #17, #18, #20
• add --color CLI option to get a colored output (default is colorless)

v0.7.0 (2019-12-11)

• add --version CLI option to display the current version of padpo
• --input-path CLI option now accepts several paths as in padpo --input-path file1.po file2.po directory1 directory2 or padpo -i file1.po file2.po directory1 directory2

v0.6.0 (2019-12-9)

• check errors against defined glossaries

v0.5.0 (2019-12-3)

• check spelling errors with grammalecte
• tag releases!

v0.4.0 (2019-12-2)

• use poetry: https://poetry.eustace.io/docs/
• add some tests with tox and pytests
• fix some false positive issues with grammalecte

padpo's Issues

--version

Would enjoy a padpo --version for our CI.

relax dependencies constraints in pyproject.toml

see travis result for python/python-docs-fr#1117

Transfer to AFPy organization

@Seluj78 @JulienPalard

• transfer the project to AFPy organization in GitHub
• update metadata
• change rights in PyPI
• change rights in CodeClimate?

Do you see any other subjects?

Packaging for PyPI

Any idea why grammalecte is not on PyPI? it would be nice so we can package padpo with grammalecte as an install dependency.

Pinning setuptools as a dependency

This does not looks like a good idea, and it's wrong: setuptools is not a runtime dependency of your project, just a dependency of your build system.

It's also not nice as it could conflict with pip itself.

faux positif

padpo -p 1015

library/re.po:1321 [ Grammalecte ] Guillemets typographiques ouvrants. => ###s ASCII et aux 4 caractères non ASCII : 'İ' (« U+01###

Malformed po file in fixtures

the https://github.com/AFPy/padpo/blob/master/tests/po_without_warnings/abc.po file is not compliant with po file format

False positive on hyperlink

2019-12-01 23:30:04,425 [ERROR   ]             padpo.py(147):display_warnings     :: using/windows.po:2453 [     NBSP     ] There should be a non-breakable space before ":": between ###cf. `Cygwin package source <ftp### and ###://ftp.uni-erlangen.de/pub/pc/g###

found using padpo on using/windows.po (obviously on python-docs-fr).

French Personal dict is mandatory fetched every startup

padpo/padpo/checkers/grammalecte.py

Line 103 in 429ef81

def get_personal_dict(self):

get_personal_dict is run at startup, and fetches https://raw.githubusercontent.com/python/python-docs-fr/3.9/dict (c.f. https://github.com/python/python-docs-fr/blob/3.9/dict ) which

a) should be cached somewhere and not refetched at startup every time, and
b) should be configurable to assist using with other projects (i.e. not python-docs-fr) or other languages

padpo doesn't like `*n*-uplet` on msgstr start

As you can see, *n*-uplet is a valid word but padpo throws an error when it sees that word at the beginning of a translation

bug on empty patch

padpo -p 1059 fails:

padpo -p 1059
Traceback (most recent call last):
  File "/home/vincent/Documents/programmation/padpo/venv/bin/padpo", line 11, in <module>
    load_entry_point('padpo', 'console_scripts', 'padpo')()
  File "/home/vincent/Documents/programmation/padpo/padpo/padpo.py", line 115, in main
    pull_request_info = pull_request_files(pull_request)
  File "/home/vincent/Documents/programmation/padpo/padpo/github.py", line 59, in pull_request_files
    pr.add_file(filename, temp_file, fileinfo["patch"])
KeyError: 'patch'

A huge file may have no patch in GitHub API response.

Could -i accept multiple files?

Would be nice in a CI also, like:
padpo -i *.po

double spaces management

Do not emit a warning for double spaces, according to the decision made in python/python-docs-fr#1113

false positive

As said in python/python-docs-fr#1059

#: ../Doc/reference/expressions.rst:1427
msgid ""
"``None`` and ``NotImplemented`` are singletons.  :PEP:`8` advises that "
"comparisons for singletons should always be done with ``is`` or ``is not``, "
"never the equality operators."
msgstr ""
"``None`` et ``NotImplemented`` sont des singletons. :PEP:`8` conseille de "
"toujours comparer les singletons en utilisant soit ``is`` soit ``is not``, "
"jamais les autres opérateurs."

2019-12-12 11:20:14,289 [ERROR   ]            pofile.py(153):display_warnings     :: reference/expressions.po:2148 [     NBSP     ] There should be a non-breakable space before ":": between ###emented » sont des singletons. ### and ###:PEP:`8` conseille de toujours ###

I need to process tcomment field in pofile entries

padpo implements its own parser of .po files and entries.
The idea is to enable the translator to tell padpo that he deliberately knows there is a false positive. The translator would do it in writing a pragma in the "translator comment" field.
The way padpo parses the .po file does not enable this feature.
I would recommend to use the polib library which give access to "translator comment" field.

404 while running padpo

Tried padpo v0.11 today on my laptop, got:

$ padpo -i  *.po
<frozen importlib._bootstrap>:1049: ImportWarning: VendorImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:673: ImportWarning: VendorImporter.exec_module() not found; falling back to load_module()
<frozen importlib._bootstrap>:1049: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:673: ImportWarning: _SixMetaPathImporter.exec_module() not found; falling back to load_module()
/home/mdk/clones/python/python-docs-fr/.venv/lib/python3.11/site-packages/pkg_resources/_vendor/pyparsing.py:87: DeprecationWarning: module 'sre_constants' is deprecated
  import sre_constants
<frozen importlib._bootstrap>:1049: ImportWarning: VendorImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:1049: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:1049: ImportWarning: VendorImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:1049: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:1049: ImportWarning: VendorImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:1049: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:1049: ImportWarning: VendorImporter.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:1049: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
Traceback (most recent call last):
  File "/home/mdk/clones/python/python-docs-fr/.venv/lib/python3.11/site-packages/pygrammalecte/pygrammalecte.py", line 108, in grammalecte_file
    result = _run_grammalecte(filename)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mdk/clones/python/python-docs-fr/.venv/lib/python3.11/site-packages/pygrammalecte/pygrammalecte.py", line 139, in _run_grammalecte
    return subprocess.run(
           ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1024, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.11/subprocess.py", line 1901, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'grammalecte-cli.py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mdk/clones/python/python-docs-fr/.venv/bin/padpo", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/mdk/clones/python/python-docs-fr/.venv/lib/python3.11/site-packages/padpo/padpo.py", line 131, in main
    errors, warnings = check_paths(path, pull_request_info=pull_request_info)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mdk/clones/python/python-docs-fr/.venv/lib/python3.11/site-packages/padpo/padpo.py", line 54, in check_paths
    errors, warnings = check_path(path, pull_request_info)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mdk/clones/python/python-docs-fr/.venv/lib/python3.11/site-packages/padpo/padpo.py", line 46, in check_path
    return check_file(path, pull_request_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mdk/clones/python/python-docs-fr/.venv/lib/python3.11/site-packages/padpo/padpo.py", line 23, in check_file
    checker.check_file(pofile)
  File "/home/mdk/clones/python/python-docs-fr/.venv/lib/python3.11/site-packages/padpo/checkers/grammalecte.py", line 40, in check_file
    self.manage_warnings(warnings, pofile)
  File "/home/mdk/clones/python/python-docs-fr/.venv/lib/python3.11/site-packages/padpo/checkers/grammalecte.py", line 48, in manage_warnings
    for warning in warnings:
  File "/home/mdk/clones/python/python-docs-fr/.venv/lib/python3.11/site-packages/pygrammalecte/pygrammalecte.py", line 97, in grammalecte_text
    yield from grammalecte_file(tmpfile)
  File "/home/mdk/clones/python/python-docs-fr/.venv/lib/python3.11/site-packages/pygrammalecte/pygrammalecte.py", line 112, in grammalecte_file
    _install_grammalecte()
  File "/home/mdk/clones/python/python-docs-fr/.venv/lib/python3.11/site-packages/pygrammalecte/pygrammalecte.py", line 163, in _install_grammalecte
    download_request.raise_for_status()
  File "/home/mdk/clones/python/python-docs-fr/.venv/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://grammalecte.net/grammalecte/zip/Grammalecte-fr-v1.12.0.zip

Liste des terms RST

https://github.com/vpoulailleau/padpo/blob/82604c068cf57b398dec048c8e37484081b2086c/padpo.py#L83

Est tu sur que cette liste est complète ?

404 au lancement de padpo vide

(python-docs-i18n) ➜  python-docs-fr git:(3.8) pip install --upgrade pip padpo potodo pospell powrap
Looking in indexes: https://pypi.python.org/simple, https://pypi.apple.com/simple
Requirement already up-to-date: pip in /Users/seluj78/.venvs/python-docs-i18n/lib/python3.7/site-packages (19.3.1)
Collecting padpo
  Downloading https://files.pythonhosted.org/packages/cc/78/34e0096714a5412acccb78c92fff709dbd1c599f901596a000da15c7edee/padpo-0.3.0-py3-none-any.whl
Requirement already up-to-date: potodo in /Users/seluj78/.venvs/python-docs-i18n/lib/python3.7/site-packages (0.0.9)
Requirement already up-to-date: pospell in /Users/seluj78/.venvs/python-docs-i18n/lib/python3.7/site-packages (1.0.3)
Requirement already up-to-date: powrap in /Users/seluj78/.venvs/python-docs-i18n/lib/python3.7/site-packages (0.3.0)
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl
Collecting simplelogging
  Using cached https://files.pythonhosted.org/packages/63/00/e173efe1df19b62a599b2c543245fc01976ce7cdc2c32f6d16c1288b8fd6/simplelogging-0.10.0-py2.py3-none-any.whl
Requirement already satisfied, skipping upgrade: requests in /Users/seluj78/.venvs/python-docs-i18n/lib/python3.7/site-packages (from padpo) (2.21.0)
Requirement already satisfied, skipping upgrade: polib in /Users/seluj78/.venvs/python-docs-i18n/lib/python3.7/site-packages (from potodo) (1.1.0)
Requirement already satisfied, skipping upgrade: regex in /Users/seluj78/.venvs/python-docs-i18n/lib/python3.7/site-packages (from pospell) (2019.8.19)
Requirement already satisfied, skipping upgrade: docutils>=0.11 in /Users/seluj78/.venvs/python-docs-i18n/lib/python3.7/site-packages (from pospell) (0.14)
Requirement already satisfied, skipping upgrade: tqdm in /Users/seluj78/.venvs/python-docs-i18n/lib/python3.7/site-packages (from powrap) (4.36.1)
Collecting colorlog==4.0.2
  Using cached https://files.pythonhosted.org/packages/68/4d/892728b0c14547224f0ac40884e722a3d00cb54e7a146aea0b3186806c9e/colorlog-4.0.2-py2.py3-none-any.whl
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /Users/seluj78/.venvs/python-docs-i18n/lib/python3.7/site-packages (from requests->padpo) (2018.11.29)
Requirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1 in /Users/seluj78/.venvs/python-docs-i18n/lib/python3.7/site-packages (from requests->padpo) (1.24.1)
Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /Users/seluj78/.venvs/python-docs-i18n/lib/python3.7/site-packages (from requests->padpo) (3.0.4)
Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /Users/seluj78/.venvs/python-docs-i18n/lib/python3.7/site-packages (from requests->padpo) (2.8)
Installing collected packages: wheel, colorlog, simplelogging, padpo
Successfully installed colorlog-4.0.2 padpo-0.3.0 simplelogging-0.10.0 wheel-0.33.6
(python-docs-i18n) ➜  python-docs-fr git:(3.8) padpo
Traceback (most recent call last):
  File "/Users/seluj78/.venvs/python-docs-i18n/bin/padpo", line 8, in <module>
    sys.exit(main())
  File "/Users/seluj78/.venvs/python-docs-i18n/lib/python3.7/site-packages/padpo/padpo.py", line 522, in main
    path, pull_request_info = pull_request_files(pull_request)
  File "/Users/seluj78/.venvs/python-docs-i18n/lib/python3.7/site-packages/padpo/padpo.py", line 458, in pull_request_files
    request.raise_for_status()
  File "/Users/seluj78/.venvs/python-docs-i18n/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.github.com/repos//files
(python-docs-i18n) ➜  python-docs-fr git:(3.8)

Corrections automatiques ?

Suite à la discussion sur python/python-docs-fr#978, il pourrait être intéressant d'appliquer en automatique les corrections dont on est sûr.

Colorless, narrower output

Logs are hard to read:

I think they should be narrower and maybe with less colors, at least no background color. why not sticking to gcc-style errors, as an example:

2019-12-12 21:27:53,569 [ERROR   ]            pofile.py(153):display_warnings     :: distutils/apiref.po: 618 [     NBSP     ] There should be a non-breakable space before ":": between ###ralement exactement un symbole ### and ###: « init + nom_de_l'extension »###
2019-12-12 21:27:54,937 [WARNING ]            pofile.py(156):display_warnings     :: faq/library.po: 938 [ Grammalecte  ] Unknown word "CGI" in ###mment reproduire un envoi de formulaire CGI (« METHOD###
2019-12-12 21:27:58,011 [WARNING ]            pofile.py(156):display_warnings     :: faq/programming.po:  72 [ Double space ] Double spaces detected between ###s développés sans *PythonWin*.### and ###*PythonWin* est disponible dan###
2019-12-12 21:27:58,026 [WARNING ]            pofile.py(156):display_warnings     :: faq/programming.po:1979 [ Grammalecte  ] Pour le “t” euphonique, il faut deux traits d’union. Pas d’apostrophe. Pas d’espace. => ###Pourquoi « a_tuple[i] += ['item'] » lève-t'il une exc###
2019-12-12 21:27:58,812 [WARNING ]            pofile.py(156):display_warnings     :: faq/windows.po: 354 [ Grammalecte  ] Unknown word "DLL" in ###avez « pas » besoin de créer un fichier DLL, et cela ###

Could be more readable as:

distutils/apiref.po:618: error: There should be a non-breakable space before ":": between ###ralement exactement un symbole ### and ###: « init + nom_de_l'extension »###
faq/library.po:938: warning: Unknown word "CGI" in ###mment reproduire un envoi de formulaire CGI (« METHOD###
faq/programming.po:72: warning: Double spaces detected between ###s développés sans *PythonWin*.### and ###*PythonWin* est disponible dan###
faq/programming.po:1979: warning: Pour le “t” euphonique, il faut deux traits d’union. Pas d’apostrophe. Pas d’espace. => ###Pourquoi « a_tuple[i] += ['item'] » lève-t'il une exc###
faq/windows.po:354: warning: Unknown word "DLL" in ###avez « pas » besoin de créer un fichier DLL, et cela ###

Note that in github the wide version is so wide we have literally no relevant information in the screen :(

Bonus: By sticking to gcc error format we could easily use padpo in vim/emacs :]

False positive

#: ../Doc/howto/unicode.rst:344
msgid ""
"The syntax is inspired by Emacs's notation for specifying variables local to "
"a file.  Emacs supports many different variables, but Python only supports "
"'coding'.  The ``-*-`` symbols indicate to Emacs that the comment is "
"special; they have no significance to Python but are a convention.  Python "
"looks for ``coding: name`` or ``coding=name`` in the comment."
msgstr ""
"La syntaxe s'inspire de la notation d'*Emacs* pour spécifier les variables "
"locales à un fichier. *Emacs* supporte de nombreuses variables différentes, "
"mais Python ne gère que *coding*. Les symboles ``-*-`` indiquent à *Emacs* "
"que le commentaire est spécial ; ils n'ont aucune signification pour Python "
"mais sont une convention. Python cherche ``coding: name`` ou "
"``coding=name`` dans le commentaire."

2019-12-13 16:29:00,094 [WARNING ]            pofile.py(156):display_warnings     :: howto/unicode.po: 615 [ Grammalecte  ] Incohérence : l’espace insécable à côté de l’espace sécable n’a pas d’effet. => ###ing ». Les symboles « -« - » indiquent à  »Emacs* qu###

Suggestions automatiques

Ca serait ouf que padpo fasse des suggestions dans github automatiquement, en tant que bot même, sans avoir a le lancer (genre un cron ou un bot comme miss-illington)

Speedup padpo

Padpo, on the fake-docs repository, takes about 5 minutes and a half to complete :/ How can it be sped up ?
https://github.com/PyDocTeur/fake-docs/pull/18/checks?check_run_id=1493393281

False positive

<NBSP>:<space>:: is OK but padpo tells some problem about NBSP

Dependabot couldn't authenticate with https://pypi.python.org/simple/

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

False positive on internal links

Words ending with _ are internal links in rst, it raises a false positive, like in https://travis-ci.org/python/python-docs-fr/builds/624345482:

2019-12-12 21:24:48,786 [WARNING ]            pofile.py(156):display_warnings     :: library/argparse.po: 936 [ Grammalecte  ] Unknown word "choices_" in ###Voir le chapitre choices_ pour plus###