pogrep fails when encoding missing from header about translate

[davidfraser]

This might or might not be fixed by the latest encoding changes, can you verify
and if not provide a file as a test case…

[dwaynebailey]

Tested this using:

pogrep -e “[ḒḓḼḽṄṅṊṋṰṱ]” —errorlevel=traceback ve duh

(All Venda special characters. And it works.

Then again with:

pogrep “ṅ” —errorlevel=traceback ve duh

(n with a dot above)

Everything works correctly

[dwaynebailey]

If the files contains no valid PO header and the contents contains non ascii
characters then the grep will fail.

We should fail more gracefully with a warning about the missing header.

[dwaynebailey]

Created [attachment 52](http://bugs.locamotion.org/attachment.cgi?id=52)

Small segment of headerless PO file with Unicode chars

[dwayne@laptop-wifi ve]$ pogrep —search=msgstr “ṇ”
—errorlevel=traceback browser-head.po browser-grep.po
processing 1 files…

pogrep: warning: Error processing: input browser-head.po, output
browser-grep.po, template None: Traceback (most recent call last):

File “/usr/lib/python2.4/site-packages/translate/misc/optrecurse.py”, line

418, in recursiveprocess
success = self.processfile(fileprocessor, options, fullinputpath,
fulloutputpath, fulltemplatepath)

File “/usr/lib/python2.4/site-packages/translate/misc/optrecurse.py”, line

471, in processfile
if fileprocessor(inputfile, outputfile, templatefile,
**passthroughoptions):

File “/usr/lib/python2.4/site-packages/translate/tools/pogrep.py”, line 142,

in runpogrep
tofile = checkfilter.filterfile(fromfile)

File “/usr/lib/python2.4/site-packages/translate/tools/pogrep.py”, line 91,

in filterfile
matches = self.filterelement(thepo)

File “/usr/lib/python2.4/site-packages/translate/tools/pogrep.py”, line 84,

in filterelement
if self.matches(unquotedstr): return True

File “/usr/lib/python2.4/site-packages/translate/tools/pogrep.py”, line 56,

in matches
found = teststr.find(self.searchstring) != -1

UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xe1 in position 5: ordinal
not in range(128)

[dwaynebailey]

pogrep is now fine – HEAD when we grep for headerless entries in non regexp mode.

However, if you search using —regexp with a Unicode search string then it fails
to detect a matching unicode entry in the Message Unit.

[dwaynebailey]

The changes to manage headerless files that Friedel committed (I think) has
fixed this problem.

The encoding of search strings has been manage by forcing a decode from UTF-8.
This will create problems for anyone not using UTF-8. We’ll fix that when teh
problem arrises.