joungkyun / python-chardet Goto Github PK

A file with a UTF-8 BOM is detected as 'utf-8' when it should ideally be 'utf-8-sig'. This is really important because lots of tools re-open the file using the detected encoding, and 'utf-8-sig' will strip the bom but 'utf-8' will not, and the BOM will cause breakages.

I realise that utf-8-sig is a Python-ism, but the libchardet could provide some extra flags in its results which python-chardet could check to known when to append the -sig.

Other differences when compared with other libraries:

utf16* are detected as 'utf-16le' and 'utf-16be', which is great because many detection libraries just report 'utf-16'. It is odd that this library is reporting the endianness of utf-16, but is not reporting the -sig when the BOM appears in utf-8.

Curly quotes in ascii are detected as 'windows-1250', which decodes correctly. \o/ Libraries which detect this often detect it as 'windows-1252', but that is an internal/arbitrary choice not based on the input text.

UTF-7 problems like reported at https://github.com/PyYoshi/uchardet/issues/4 , and the other BOMs reported there, also occur in this library.

What is your problem?

Error when building using setup.py

OS and Architecture

Enter the OS and architecture information using the following command: CentOS 7

[user@host ~]$ python -c 'import platform; print(platform.uname())'

Python version

Enter the your version of PYTHON using the following command: 2.7.5

[user@host ~]$ python --version

python-chardet version

Enter the your version of PYTHON using the following command: 60f2c39

[user@host ~]$ python -c 'import chardet; print(chardet.__version__)'

Expected behavior

no error

Actual behavior

[user@host python-chardet-master]$ python setup.py build
Traceback (most recent call last):
  File "setup.py", line 45, in <module>
    long_description=readme(),
  File "setup.py", line 12, in readme
    with open('README') as f:
IOError: [Errno 2] No such file or directory: 'README'

Steps to reproduce the behavior

[user@host ~]$ cd
[user@host ~]$ git clone https://github.com/Joungkyun/python-chardet.git
Cloning into 'python-chardet'...
remote: Counting objects: 185, done.
remote: Total 185 (delta 0), reused 0 (delta 0), pack-reused 185
Receiving objects: 100% (185/185), 36.91 KiB | 0 bytes/s, done.
Resolving deltas: 100% (117/117), done.
[user@host python-chardet]$ cd python-chardet/
[user@host python-chardet]$ python setup.py build
Traceback (most recent call last):
  File "setup.py", line 45, in <module>
    long_description=readme(),
  File "setup.py", line 12, in readme
    with open('README') as f:
IOError: [Errno 2] No such file or directory: 'README'
[root@an3 python-chardet]#

This package should try to maintain compatibility with the traditional chardet, or document any differences.

It would be nice to run a modified copy of their test suite, and periodically sync it.

chardet master has introduced detect_all which returns an array of results.

I couldnt find this published on PyPI.

That is a prerequisite for packaging in most distros, as their package naming convention usually names the package after the PyPI name.

openSUSE has libchardet packages, so publishing this Python package should be really easy.

setup.py uses chardet-config which is nice, except that it is broken on openSUSE and the maintainers recommend using pkgconfig instead. see https://build.opensuse.org/request/show/720085

libchardet does include a .pc , so it could be used instead.

What is your problem?

detector.done can't check

if change code as follows:

--- a/src/universaldetector.c
+++ b/src/universaldetector.c
@@ -64,6 +64,8 @@ static void Universal_set_result (Universal * self, char * encoding, float confi
    else
        PyDict_Clear (self->result);

+   printf ("**** encoding => %s, bom => %d\n", encoding, bom);
+
    if ( strcmp (encoding, "UTF-8") == 0 && bom == 1 ) {
        int buflen = sizeof (char) * (strlen (encoding) + 5);
        buf = malloc (buflen);

can find that can't detect done

URL utf-8-bom.txt        : **** encoding => None, bom => 0
**** encoding => UTF-8, bom => 1 =======================> detect UTF-8 bom, but can't stop
**** encoding => UTF-8, bom => 0
encoding: UTF-8          , confidence: 1.00

OS and Architecture

ALL

Python version

ALL

python-chardet version

2.0.0 and before

Expected behavior

detact UTF-8-SIG

Actual behavior

detact UTF-8

Steps to reproduce the behavior

f = open ('./utf-8-bom.txt', 'r')
detector.reset()
while True:
    line = f.readline()
    if not line : break
    detector.feed(line)
    if detector.done: break
f.close()
detector.close()