Potentially incorrect tokenization about clearnlp

[GoogleCodeExporter]

Additional tokenization differences.

Text: I said at 4:45pm.
Expected: [I, said, at, 4:45, pm, .]
ClearNLP: [I, said, at, 4:45pm, .]

Text:
I can't believe they wanna keep 40% of that."
        ``Whatcha think?''
        "I don't --- think so...,"
Expected: [I, ca, n't, believe, they, wan, na, keep, 40, %, of, that, ., ", ``, 
Wha, t, cha, think, ?, '', ", I, do, n't, ---, think, so, ..., ,, "]
ClearNLP: [I, ca, n't, believe, they, wan, na, keep, 40, %, of, that, ., ", `, 
`, Whatcha, think, ?, ', ', ", I, do, n't, ---, think, so, ..., ,, "]

Text: 
You `paid' US$170,000?!
        You should've paid only$16.75.
Expected: [You, `, paid, ', US$, 170,000, ?, !, You should, 've, paid, only, $, 
16.75, .]
ClearNLP: [You, `, paid, ', US, $, 170,000, ?!, You, should, 've, paid, only, 
$, 16.75, .]


Text: 
1. Buy a new Chevrolet (37%-owned in the U.S..) . 15%
Expected: [1, ., Buy, a, new, Chevrolet, (, 37%-owned, in, the, U.S, .., ), ., 
15, %]
ClearNLP: [1, ., Buy, a, new, Chevrolet, (, 37, %, -, owned, in, the, U.S, .., 
), ., 15, %]
** Though for this one, ClearNLP disagrees with the ClearTK tokenizer, and 
agrees with the Stanford parser on "37%-owned". **

Original comment by lee.becker on 27 Oct 2012 at 7:48

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Lee,

Here are my answers.  All the bugfixes will be applied to the following 
version, which is coming in next 2 days.  Thanks for finding these  bugs and 
please let me know if you find more.  I expect the tokenizer  to keep evolving!

1. The cases of `` and '' are bugs, they are fixed now.
2. The case of "AT&T" is a known bug; the LDC tokenizer handles it as the 
ClearNLP tokenizer does.  I made a simple heuristic to get around with it 
("\\p{Upper}\\&\\p{Upper}"); I didn't want to go with the dictionary approach 
for this case since that can cover only so far.
3. The case of "whatcha" should be "what cha" instead of "wha t cha".  This is 
also fixed now.
4. The case of "4:45pm." is a bug; it is fixed now.
5. "US" and "$" are now joined as "US$".
6. The case of "?!" is intentional; all period-like punctuation (e.g., ".", 
"!", and "?") are grouped as one unit in the ClearNLP tokenizer.
7. For the last example ("1. Buy …"), the ClearNLP is doing the same as the 
LDC tokenizer.  The hyphenation is a change that LDC made few years ago which I 
don't think was applied to the OpenNLP.

Jinho

Original comment by [email protected] on 29 Oct 2012 at 11:08