woodgern / confusables Goto Github PK

There are a few characters that get display as an empty string like those for example:

\u200b
\u200c
\u200d
\u200e
\u200f

They can be mixed into any string and thus bypass the confusable detection since they are not visible the strings look the same

>>> print("f\u200boo")
f​oo

When using the normalize function for 'rn', 'r' followed by 'n', it doesn't include 'm' in the returned list despite the fact that applying confusable_characters to 'm' includes 'rn' in the list, presumably normalize is only applying to single characters e.g:

>>> normalize('rn')
['rn']  # should return ['m', 'rn']
>>> confusable_characters('m')
['𝑚', 'μ', 'ᗰ', 'Ḿ', '𝘔', 'Ⲙ', '𝓂', 'ℳ', '𝑴', '𝓜', 'м', '𝚳', '𝚖', '𝙼', '𝜧', 'Ꮇ', '𝑀', '𝘮', '𝕞', '𝓶', '𑜀', 'ｍ', '𝗺', '𝞛', '𝗆', 'ᛖ', '𝛭', 'Ⅿ', 'M', 'rn', 'µ', '𝙢', 'ṃ', '𝐦', 'ꭑ', 'Ṃ', '𝕸', 'ⲙ', '𝒎', 'ḿ', 'ϻ', 'ꓟ', 'Ｍ', 'm', 'ꮇ', 'ⅿ', '𐊰', '𝔐', 'Ϻ', 'Μ', '𝝡', 'Ṁ', '𝖬', '𝐌', '𐌑', '𝔪', '𝙈', '𝕄', 'ṁ', '𑣣', 'М', '𝖒', '𝗠']

If you run normalize on this string:

abing🪀|C-01 |🍏inv100+ Lv16推赞1300

then it will return an EXTREMELY long (3981312 entries) list.

It must be getting caught on something, because a 3+ million list attempting to normalize a string is absurd.

To reproduce:

import confusables

foo = "abing🪀|C-01 |🍏inv100+ Lv16推赞1300"

x = confusables.normalize(foo)

print(len(x))
print(x)

When trying to create a confusable regex from a string like "t[e]st", it will fail to match the string "t[e]st". I believe this is because the characters are not escaped when added to the regex.

In parse.py when creating the confusable sets, there is an assumption that if a certain character is confusable with another, it will also be confuseable with the upper/lower case version. This isn't always the case.

For example this causes "latin small letter v" (v) to be confusable with "latin capital letter n" (N), as "greek small letter nu" is confusable with "latin small letter v" (03BD) and calling string.upper() on charater 03BD produces "latin capital letter n".

Hello woodgern and thanks for the amazing library. I'm working on porting this code to Golang so that others can benefit from your work, and I had a few questions about the code.

1. In parse.py on line 50, you check to see if str1 only contains 1 character. However, this appears unnecessary, because all of the characters in the left-most column of confusing.txt contain only 1 character (and 1 code point). Is there some other reason that you have included this?

2. In parse.py on line 59, you check to see if str2 only contains 1 character. Why exactly is this? For example, consider the following line of confusables.txt:

0191 ;	0046 0326 ;	MA	# ( Ƒ → F̦ ) LATIN CAPITAL LETTER F WITH HOOK → LATIN CAPITAL LETTER F, COMBINING COMMA BELOW	# →F̡→

This seems like it should be included in a list of "look-alike" characters (since it case-inverts into "f̦"), but it is now skipped because it has a length of two. Is this intentional?

3. On lines 40-48, you add both sides to each other's map entry. However, you don't do the same thing on lines 50-66. In other words, shouldn't

unicode_confusable_map[str1].add(case_change)

be instead:

unicode_confusable_map[str1].add(case_change)
unicode_confusable_map[str2].add(case_change)

?

The latest one at the moment is version 13.0.0. Could be updated also in README.md.

Hi,

I'm playing around with flake8 plugins, including flake8-confusables. But installing that (and as a dependency, this package) raises a FileNotFoundError: [Errno 2] No such file or directory: '/home/yggdrasil/.virtualenvs/svhclient/lib/python3.6/site-packages/confusables/assets/confusable_mapping.json'. Manually copying the assets directory there resolves this.
Might I suggest adding a MANIFEST.in :)

Thanks

Hi!
There used to be confusables for python that I forked and no longer exists. It had one very useful function skeleton(). It returned string that transformed all confusables to normalized values. So you could detect that original strings don't match but normalized strings match. Like A in latin and А in azbuka.

https://github.com/don-mums/confusables/blob/master/confusables.py

Is this something you would like to include into your project? The only thing I found that remotely resembles skeleton() is _asciify() but this also removes all non ascii characters, which is not very useful for us. As you can see on the example below.

'Veľký Krtíš'.encode('ascii', 'ignore').decode('ascii') == 'Vek Krt'