polib mutilates escape sequences about polib

[izimobil]

Original comment by qx0monster (Bitbucket: qx0monster, GitHub: Unknown):

---

Sorry, I'm out of this. I will not be able to provide a patch as discussed, so please don't wait for me. Close or proceed as you see fit.

[izimobil]

Original comment by qx0monster (Bitbucket: qx0monster, GitHub: Unknown):

---

I'm terribly lagging behind ...

[izimobil]

Original comment by David Jean Louis (Bitbucket: izi, GitHub: izi):

---

Issue #35 was marked as a duplicate of this issue.

[izimobil]

Original comment by David Jean Louis (Bitbucket: izi, GitHub: izi):

---

Hi qx0monster, any progress on this ? Thanks.

[izimobil]

Original comment by Jakub Wilk (Bitbucket: jwilk, GitHub: jwilk):

---

The correct solution is to fix the parser to decode all escape sequences (and possibly signal an error on unknown ones).

AFAICS it is true that with qx0monster's patch polib indeed round-trips properly. It's just the Python objects that are supposed to represent the PO file contents don't make sense. For example, for this file:

msgid ""
msgstr ""

msgid "\\n"
msgstr ""

I get:

>>> polib.pofile('t.po')[0].msgid[-1].isspace()
True

which is wrong: the msgid didn't contain any whitespace.

[izimobil]

Original comment by David Jean Louis (Bitbucket: izi, GitHub: izi):

---

It would be cool if you could send a pull request (with tests and docs), thanks in advance !

[izimobil]

Original comment by qx0monster (Bitbucket: qx0monster, GitHub: Unknown):

---

Would you suggest I'll take a look at it? Or is this something you want to do yourself?

[izimobil]

Original comment by David Jean Louis (Bitbucket: izi, GitHub: izi):

---

I think this should be done in polib.pofile and polib.mofile, but also in polib.POFile and polib.MOFile classes, as it's done for the check_for_duplicates option.

[izimobil]

Original comment by qx0monster (Bitbucket: qx0monster, GitHub: Unknown):

---

Fine with me. Where would this option be set? On the POFile/_BaseFile object? Will you add it to the params of the pofile() factory?

[izimobil]

Original comment by David Jean Louis (Bitbucket: izi, GitHub: izi):

---

Hi, thanks for the explanations, it makes perfect sense indeed. That said, doing this by default will totally break backwards compatibility, so I'm +1 on this if it's an explicit option.
Regards.

[izimobil]

Original comment by qx0monster (Bitbucket: qx0monster, GitHub: Unknown):

---

To add a bit more rational to this issue:

• Compatibility with gettext tools: The gettext parser supports (a) octal numbers of the form '\ooo', (b) hex numbers of the form '\x...' and (c) some additional control seq's like '\b', '\f', '\v' and '\a' (while most of the latter are warned about as "not suitable" for internationalization efforts).
• The gettext parser is strict, in that it does not allow unknown escape sequences (it bombs). It also seems to replace escape sequences with their binary equivalent, e.g. '\x61' is replaced with 'a'. Leaving octal, hex and the other allowed control seq's untouched seems like a good interop measure, for people using both polib and gettext tools.
• Unicode support: gettext tools don't support unicode escapes (Funnily, there is a code comment in po-lex.c, saying "FIXME: \u and \U are not handled"! http://git.savannah.gnu.org/cgit/gettext.git/tree/gettext-tools/src/po-lex.c?id=b75a50bc5f30666fb36469a90245e1d7856d672c#n851). But as I said support for unicode escapes is crucial for us, and I would appreciate if polib was getting ahead of gettext in this respect. Being just lenient with unknown escape seq's would do the job.