snukky / wikiedits Goto Github PK

Hello @snukky, I wanted some clarity on this piece of code

def __levenshtein_ratio(self, old_sent, new_sent):
    old_words = old_sent.split()
    new_words = new_sent.split()

   min_words = min(len(old_words), len(new_words))
   dist = self.__levenshtein_on_words(old_words, new_words)
   ratio = dist / float(min_words) * math.log(min_words,
   self.LEVENSHTEIN_RATIO_LOG_BASE)

particularly how the ratio is calculated and how it can be interpreted
Is it a scaled version of - how much in ratio will the smaller string need to be operated on given the distance?
Could you point me to relevant literature if available, Thanks

Hi,
I am having problems downloading the ready datasets, wiked-v1.0.en.tgz and wiked-v1.0.en.prepro.tgz - do the links work?
If I were to process the dump from scratch, which dump should I download? I am looking at the page of the current dump https://dumps.wikimedia.org/enwiki/20220920/, is it the pages-articles file?
Thanks!

Anna

1. In bin/txt_edits.py, the text in input and output text files are being read as a whole in the lines
   old_text = " ".join(line.rstrip() for line in file.readlines()) and
   new_text = " ".join(line.rstrip() for line in file.readlines())
   which doesn't work for large files(such as datasets, whose size is in GBs).

2. Also in setup.py, the line has to be corrected from
   long_description=open('README.txt').read(), to
   long_description=open('README.md').read(),

The paper mentions +Select corpus which filters only edits that are like those that appear in NUCLE corpus. Any script available to do that?

Here's an error to be corrected (and noted in the edit history):

The scripts work independently form Wiki Edits

It should be from.

Hi!
What would be the best way to consider/filter only edits performed by users with:

• a high level of language proficiency (e.g. in English-EN https://en.wikipedia.org/wiki/Category:User_en-5 , https://en.wikipedia.org/wiki/Category:User_en-N
• a high level of activity as editor: https://en.wikipedia.org/wiki/Wikipedia:Service_awards

and possibly:

• consider only main namespace-related edits only
• consider edits labeled as grammar/spelling fixes in the tag

?

I'm asking because by looking at wiked-v1.0.en.prepro.tgz, spelling error corrections, grammatical error corrections and stylistic changes seem to be outnumbered by edits that are less useful for grammatical error correction tasks (some of the edits are non-vandalizing regressions, generally poor, or anyway fail to correct all problems in the sentences). Also, my gut feeling is that "meta" discussions/talk might have a lower language quality.

What would be the best way to implement this in the code? I was thinking of labeling the unedited/edited snippets with username + comment, by making wiki_dump_parser.py or edit_filter.py so to print out namespace/username/comment-filtered edits (but I am not sure of what would be the easiest approach).