kanyalee / assignment0_dsw

assignment0_dsw

How to run the code:

1. Install and Set up Spark on the local machine
2. Download the wiki.txt file as an input
3. Download the python file (name wordcount.py) from Canvas where it contain the main code -- did not update the file name but it contains bigrams
4. Run the code: spark-submit wordcount.py wiki.txt output_bigramprb1 output_bigramcount1
5. The output script will be two text files: one containing the bigram probability and one another containing bigrams counts 5.1 the bigram probability file shows each bigram along with its probability 5.2 the bigram counts file shows how each bigram is presented with the number of count

Link to output for Bigram Count file https://drive.google.com/file/d/1UJG1XBB5i-IDn6MvC2_mdkEmL-KBB36T/view?usp=sharing

Link to output for Bigram Probability file https://drive.google.com/file/d/1H5YCstLnpsAReifCQk30hl5Js-oHOI2I/view?usp=sharing

Link to output for Word Count file https://drive.google.com/file/d/1ACSW877aWQeYvEDG0C9qM8ctqTEbqBvV/view?usp=sharing