Mastering Spark for Data Science, published by Packt
This is the code repository for Mastering Spark for Data Science, published by Packt. It contains all the supporting project files necessary to work through the book from start to finish.
Data science seeks to transform the world using data and this is typically achieved through disrupting and changing real processes in real industries. In order to operate at this level you need to be able to build data science solutions of substance – solutions that solve real problems and that run so relialbly that people trust and actualize them. Spark has emerged as the big data platform of choice for data scientists due to its speed, scalability, and easy-to-use APIs.
This book deep dives into Spark to deliver production grade data science solutions that are innovative, disruptive, and reliable. This process is demonstrated by exploring the construction of a sophisticated global news analysis service that uses Spark to generate continuous geopolitical and current affairs insights. You will learn all about the core Spark APIs and take a comprehensive tour of advanced libraries including: Spark SQL, visual streaming, MLlib, and more.
All of the code is organized into folders. Each folder starts with a number followed by the application name. For example, Chapter02.
The code will look like the following:
import org.apache.spark.sql.functions._
val rdd = rawDS map GdeltParser.toCaseClass
val ds = rdd.toDS()
// DataFrame-style API
ds.agg(avg("goldstein")).as("goldstein").show()
Spark 2.0 is used throughout the book along with Scala 2.11, Maven and Hadoop. This is the basic environment required, there are many other technologies used which are introduced in the relevant chapters.
Click here if you have any feedback or suggestions.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent