woodgern / confusables Goto Github PK
View Code? Open in Web Editor NEWA python package providing functionality for matching words using different characters but appearing to be a similar/the same word.
License: MIT License
A python package providing functionality for matching words using different characters but appearing to be a similar/the same word.
License: MIT License
There are a few characters that get display as an empty string like those for example:
\u200b
\u200c
\u200d
\u200e
\u200f
They can be mixed into any string and thus bypass the confusable detection since they are not visible the strings look the same
>>> print("f\u200boo")
fโoo
When using the normalize function for 'rn', 'r' followed by 'n', it doesn't include 'm' in the returned list despite the fact that applying confusable_characters to 'm' includes 'rn' in the list, presumably normalize is only applying to single characters e.g:
>>> normalize('rn')
['rn'] # should return ['m', 'rn']
>>> confusable_characters('m')
['๐', 'ฮผ', 'แฐ', 'แธพ', '๐', 'โฒ', '๐', 'โณ', '๐ด', '๐', 'ะผ', '๐ณ', '๐', '๐ผ', '๐ง', 'แท', '๐', '๐ฎ', '๐', '๐ถ', '๐', '๏ฝ', '๐บ', '๐', '๐', 'แ', '๐ญ', 'โ
ฏ', 'M', 'rn', 'ยต', '๐ข', 'แน', '๐ฆ', '๊ญ', 'แน', '๐ธ', 'โฒ', '๐', 'แธฟ', 'ฯป', '๊', '๏ผญ', 'm', '๊ฎ', 'โ
ฟ', '๐ฐ', '๐', 'ฯบ', 'ฮ', '๐ก', 'แน', '๐ฌ', '๐', '๐', '๐ช', '๐', '๐', 'แน', '๐ฃฃ', 'ะ', '๐', '๐ ']
If you run normalize on this string:
abing๐ช|C-01 |๐inv100+ Lv16ๆจ่ต1300
then it will return an EXTREMELY long (3981312 entries) list.
It must be getting caught on something, because a 3+ million list attempting to normalize a string is absurd.
To reproduce:
import confusables
foo = "abing๐ช|C-01 |๐inv100+ Lv16ๆจ่ต1300"
x = confusables.normalize(foo)
print(len(x))
print(x)
When trying to create a confusable regex from a string like "t[e]st", it will fail to match the string "t[e]st". I believe this is because the characters are not escaped when added to the regex.
In parse.py when creating the confusable sets, there is an assumption that if a certain character is confusable with another, it will also be confuseable with the upper/lower case version. This isn't always the case.
For example this causes "latin small letter v" (v) to be confusable with "latin capital letter n" (N), as "greek small letter nu" is confusable with "latin small letter v" (03BD) and calling string.upper()
on charater 03BD produces "latin capital letter n".
Hello woodgern and thanks for the amazing library. I'm working on porting this code to Golang so that others can benefit from your work, and I had a few questions about the code.
In parse.py on line 50, you check to see if str1 only contains 1 character. However, this appears unnecessary, because all of the characters in the left-most column of confusing.txt contain only 1 character (and 1 code point). Is there some other reason that you have included this?
In parse.py on line 59, you check to see if str2 only contains 1 character. Why exactly is this? For example, consider the following line of confusables.txt:
0191 ; 0046 0326 ; MA # ( ฦ โ Fฬฆ ) LATIN CAPITAL LETTER F WITH HOOK โ LATIN CAPITAL LETTER F, COMBINING COMMA BELOW # โFฬกโ
This seems like it should be included in a list of "look-alike" characters (since it case-inverts into "fฬฆ"), but it is now skipped because it has a length of two. Is this intentional?
unicode_confusable_map[str1].add(case_change)
be instead:
unicode_confusable_map[str1].add(case_change)
unicode_confusable_map[str2].add(case_change)
?
The latest one at the moment is version 13.0.0. Could be updated also in README.md.
Hi,
I'm playing around with flake8 plugins, including flake8-confusables. But installing that (and as a dependency, this package) raises a FileNotFoundError: [Errno 2] No such file or directory: '/home/yggdrasil/.virtualenvs/svhclient/lib/python3.6/site-packages/confusables/assets/confusable_mapping.json'
. Manually copying the assets
directory there resolves this.
Might I suggest adding a MANIFEST.in :)
Thanks
Hi!
There used to be confusables for python that I forked and no longer exists. It had one very useful function skeleton()
. It returned string that transformed all confusables to normalized values. So you could detect that original strings don't match but normalized strings match. Like A
in latin and ะ
in azbuka.
https://github.com/don-mums/confusables/blob/master/confusables.py
Is this something you would like to include into your project? The only thing I found that remotely resembles skeleton()
is _asciify()
but this also removes all non ascii characters, which is not very useful for us. As you can see on the example below.
'Veฤพkรฝ Krtรญลก'.encode('ascii', 'ignore').decode('ascii') == 'Vek Krt'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.