remove_words! fails for long terms & terms with punctuation about textanalysis.jl

[enkiv2]

It sounds like this would be substantially faster than my patch, so I approve :)

…

On Tue, May 14, 2019 at 2:28 AM Ayush Kaushal ***@***.***> wrote: I think a better way to solve this issue could be using a TokenBuffer which is currently used by the tokenizers in WordTokenizers.jl. ( #143 <#143> ) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#74?email_source=notifications&email_token=AADXUGMGAAHFI2D2ZJYMGRLPVJLYFA5CNFSM4EZJCJYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVKMXUQ#issuecomment-492096466>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADXUGKGXUUES4IGO6TL3X3PVJLYFANCNFSM4EZJCJYA> .

[aviks]

Might be a good idea. Will need some thought on how to deprecate the existing behaviour. Care to do a PR?

[enkiv2]

Well, I have a naive solution that works for StringDocument; unfortunately, it has major performance problems. Should I submit a PR or wait until I've figured out how to get reasonable performance out of it?

[aviks]

I'd say submit a PR. We can figure out performance later. Slow code is better than no code.

[enkiv2]

PR: #76

Because of string copying overhead, for situations big enough for the regex size to matter, it's unusably slow. However, in the absence of reliable regex escaping, this has reliability benefits.

[Ayushk4]

I think a better way to solve this issue could be using a TokenBuffer which is currently used by the tokenizers in WordTokenizers.jl. ( #143 )