jeanineharb / big-data-analysis-with-scala-and-spark

My submissions for the Coursera MOOC "Big Data Analysis with Scala and Spark" given by EPFL.

Big Data Analysis with Scala and Spark

This project contains my submissions for the Coursera MOOC Big Data Analysis with Scala and Spark, given by EPFL, and taught by Prof. Heather C. Miller.

Timeline

• Date Started: March 14, 2017

• Date Completed: March 24, 2017

Assignments

Programming Assignment 1: Wikipedia

• Week: 1

• Lesson: Basics of Spark's RDDs

• Description: "In this assignment, we'll use our full-text data from Wikipedia to produce a rudimentary metric of how popular a programming language is, in an effort to see if our Wikipedia-based rankings bear any relation to the popular Red Monk rankings."

• Grade: 10 / 10

Programming Assignment 2: StackOverflow

• Week: 2 (two-week long assignment)

• Lesson: Reduction Operations & Distributed Key-Value Pairs

• Description: "The overall goal of this assignment is to implement a distributed k-means algorithm which clusters posts on the popular question-answer platform StackOverflow according to their score. Moreover, this clustering should be executed in parallel for different programming languages, and the results should be compared."

• Grade: 10 / 10

Programming Assignment 3: Time Usage

• Week: 4

• Lesson: SQL, Dataframes, and Datasets

• Description: "Our goal is to identify three groups of activities: primary needs (sleeping and eating), work, other (leisure). And then to observe how do people allocate their time between these three kinds of activities, and if we can see differences between men and women, employed and unemployed people, and young (less than 22 years old), active (between 22 and 55 years old) and elder people."

• Grade: 10 / 10

Note

Resource files have to be unzipped for the code to work.

big-data-analysis-with-scala-and-spark's Issues

I follow your code but I still get some errors. Please give me some hints to fix it

`Tests that were aborted took too long too complete or crashed the
JVM. Such crashes can arise due to infinite non-terminating
loops or recursion (StackOverflowException) or excessive memory
consumption (OutOfMemoryException).

======== LOG OF FAILED TESTS ========

[Test Description] 'kmeans' running on the sampled stackoverflow dataset should return correct centroids (10pts)(stackoverflow.StackOverflowSuite)
[Observed Error] assertion failed:
Expected: List((0,458), (0,2), (0,115), (50000,2), (50000,929), (50000,141), (100000,42), (100000,385), (100000,1), (150000,70), ...)
Actual: List((0,4), (0,0), (0,1), (50000,1), (50000,3), (50000,2), (100000,0), (100000,2), (100000,0), (150000,0), ...)
The given collection contains some unexpected elements: List((300000,0), (700000,1), (300000,8), (50000,1), (0,0), ...)

[Test Description] 'kmeans' running just 1 iteration on the sampled stackoverflow dataset should return correct centroids (10pts)(stackoverflow.StackOverflowSuite)
[Observed Error] assertion failed:
Expected: List((0,11), (0,0), (0,1), (50000,0), (50000,10), (50000,2), (100000,0), (100000,5), (100000,0), (150000,0), ...)
Actual: List((0,4), (0,0), (0,1), (50000,1), (50000,3), (50000,2), (100000,0), (100000,2), (100000,0), (150000,0), ...)
The given collection contains some unexpected elements: List((300000,0), (700000,1), (300000,8), (50000,1), (350000,144), ...)
`
score of submition is 7/10