joungkyun / python-chardet Goto Github PK
View Code? Open in Web Editor NEWPython Universal Character Encoding Detector C-binding module (Compatible and faster than py-chardet)
License: Other
Python Universal Character Encoding Detector C-binding module (Compatible and faster than py-chardet)
License: Other
setup.py
uses chardet-config
which is nice, except that it is broken on openSUSE and the maintainers recommend using pkgconfig instead. see https://build.opensuse.org/request/show/720085
libchardet does include a .pc , so it could be used instead.
This package should try to maintain compatibility with the traditional chardet, or document any differences.
It would be nice to run a modified copy of their test suite, and periodically sync it.
chardet master has introduced detect_all
which returns an array of results.
Error when building using setup.py
Enter the OS and architecture information using the following command: CentOS 7
[user@host ~]$ python -c 'import platform; print(platform.uname())'
Enter the your version of PYTHON using the following command: 2.7.5
[user@host ~]$ python --version
Enter the your version of PYTHON using the following command: 60f2c39
[user@host ~]$ python -c 'import chardet; print(chardet.__version__)'
no error
[user@host python-chardet-master]$ python setup.py build
Traceback (most recent call last):
File "setup.py", line 45, in <module>
long_description=readme(),
File "setup.py", line 12, in readme
with open('README') as f:
IOError: [Errno 2] No such file or directory: 'README'
[user@host ~]$ cd
[user@host ~]$ git clone https://github.com/Joungkyun/python-chardet.git
Cloning into 'python-chardet'...
remote: Counting objects: 185, done.
remote: Total 185 (delta 0), reused 0 (delta 0), pack-reused 185
Receiving objects: 100% (185/185), 36.91 KiB | 0 bytes/s, done.
Resolving deltas: 100% (117/117), done.
[user@host python-chardet]$ cd python-chardet/
[user@host python-chardet]$ python setup.py build
Traceback (most recent call last):
File "setup.py", line 45, in <module>
long_description=readme(),
File "setup.py", line 12, in readme
with open('README') as f:
IOError: [Errno 2] No such file or directory: 'README'
[root@an3 python-chardet]#
I couldnt find this published on PyPI.
That is a prerequisite for packaging in most distros, as their package naming convention usually names the package after the PyPI name.
openSUSE has libchardet packages, so publishing this Python package should be really easy.
detector.done can't check
if change code as follows:
--- a/src/universaldetector.c
+++ b/src/universaldetector.c
@@ -64,6 +64,8 @@ static void Universal_set_result (Universal * self, char * encoding, float confi
else
PyDict_Clear (self->result);
+ printf ("**** encoding => %s, bom => %d\n", encoding, bom);
+
if ( strcmp (encoding, "UTF-8") == 0 && bom == 1 ) {
int buflen = sizeof (char) * (strlen (encoding) + 5);
buf = malloc (buflen);
can find that can't detect done
URL utf-8-bom.txt : **** encoding => None, bom => 0
**** encoding => UTF-8, bom => 1 =======================> detect UTF-8 bom, but can't stop
**** encoding => UTF-8, bom => 0
encoding: UTF-8 , confidence: 1.00
ALL
ALL
2.0.0 and before
detact UTF-8-SIG
detact UTF-8
f = open ('./utf-8-bom.txt', 'r')
detector.reset()
while True:
line = f.readline()
if not line : break
detector.feed(line)
if detector.done: break
f.close()
detector.close()
A file with a UTF-8 BOM is detected as 'utf-8' when it should ideally be 'utf-8-sig'. This is really important because lots of tools re-open the file using the detected encoding, and 'utf-8-sig' will strip the bom but 'utf-8' will not, and the BOM will cause breakages.
I realise that utf-8-sig
is a Python-ism, but the libchardet could provide some extra flags in its results which python-chardet
could check to known when to append the -sig
.
Other differences when compared with other libraries:
utf16* are detected as 'utf-16le' and 'utf-16be', which is great because many detection libraries just report 'utf-16'. It is odd that this library is reporting the endianness of utf-16, but is not reporting the -sig
when the BOM appears in utf-8.
Curly quotes in ascii are detected as 'windows-1250', which decodes correctly. \o/ Libraries which detect this often detect it as 'windows-1252', but that is an internal/arbitrary choice not based on the input text.
UTF-7 problems like reported at https://github.com/PyYoshi/uchardet/issues/4 , and the other BOMs reported there, also occur in this library.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.