snukky / wikiedits Goto Github PK
View Code? Open in Web Editor NEWAutomatic extraction of edited sentences from text edition histories.
License: Other
Automatic extraction of edited sentences from text edition histories.
License: Other
Hello @snukky, I wanted some clarity on this piece of code
def __levenshtein_ratio(self, old_sent, new_sent):
old_words = old_sent.split()
new_words = new_sent.split()
min_words = min(len(old_words), len(new_words))
dist = self.__levenshtein_on_words(old_words, new_words)
ratio = dist / float(min_words) * math.log(min_words,
self.LEVENSHTEIN_RATIO_LOG_BASE)
particularly how the ratio is calculated and how it can be interpreted
Is it a scaled version of - how much in ratio will the smaller string need to be operated on given the distance?
Could you point me to relevant literature if available, Thanks
Hi,
I am having problems downloading the ready datasets, wiked-v1.0.en.tgz and wiked-v1.0.en.prepro.tgz - do the links work?
If I were to process the dump from scratch, which dump should I download? I am looking at the page of the current dump https://dumps.wikimedia.org/enwiki/20220920/, is it the pages-articles file?
Thanks!
Anna
In bin/txt_edits.py, the text in input and output text files are being read as a whole in the lines
old_text = " ".join(line.rstrip() for line in file.readlines())
and
new_text = " ".join(line.rstrip() for line in file.readlines())
which doesn't work for large files(such as datasets, whose size is in GBs).
Also in setup.py, the line has to be corrected from
long_description=open('README.txt').read(),
to
long_description=open('README.md').read(),
The paper mentions +Select corpus which filters only edits that are like those that appear in NUCLE corpus. Any script available to do that?
Here's an error to be corrected (and noted in the edit history):
The scripts work independently form Wiki Edits
It should be from.
Hi!
What would be the best way to consider/filter only edits performed by users with:
and possibly:
?
I'm asking because by looking at wiked-v1.0.en.prepro.tgz, spelling error corrections, grammatical error corrections and stylistic changes seem to be outnumbered by edits that are less useful for grammatical error correction tasks (some of the edits are non-vandalizing regressions, generally poor, or anyway fail to correct all problems in the sentences). Also, my gut feeling is that "meta" discussions/talk might have a lower language quality.
What would be the best way to implement this in the code? I was thinking of labeling the unedited/edited snippets with username + comment, by making wiki_dump_parser.py or edit_filter.py so to print out namespace/username/comment-filtered edits (but I am not sure of what would be the easiest approach).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.