jgperrin / net.jgp.books.spark.ch17

Spark in Action, 2nd edition - chapter 16 - exporting data, using delta lake

This repository contains the Java labs as well as their Scala and Python ports of the code used in Manning Publication’s Spark in Action, 2nd edition, by Jean-Georges Perrin.

---

Spark in Action, 2nd edition – Java, Python, and Scala code for chapter 17

Chapter 17 is about exporting data.

This code is designed to work with:

• Apache Spark v3.0.0.
• Delta Lake v0.7.0.

Labs

Lab #100

Adapted from one of IBM's Code for Call starter kit, but leveraging the wildfires category.

Lab #200

Feeds data into Delta Lake.

Lab #210, #220

Run analytics on data stored in Delta Lake.

Lab #250

Feeds data into Delta Lake.

Lab #900

Append using primary key.

Lab #910

Export widlfire data to Elasticsearch.

Datasets

Dataset(s) used in this chapter:

• https://www.data.gouv.fr/en/datasets/donnees-ouvertes-du-grand-debat-national/#_
• Populations légales 2016 https://www.insee.fr/fr/statistiques/3677785?sommaire=3677855

Lab #100

The ExportWildfiresApp application does the following:

1. It acquires a session (a SparkSession).
2. It asks Spark to load (ingest) a dataset in CSV format.
3. Spark stores the contents in a dataframe, do some transformations and export final reports as CSV data.

Running the lab in Java

For information on running the Java lab, see chapter 1 in Spark in Action, 2nd edition.

Running the lab using PySpark

Prerequisites:

You will need:

• git.
• Apache Spark (please refer Appendix P - 'Spark in production: installation and a few tips').

1. Clone this project

git clone https://github.com/jgperrin/net.jgp.books.spark.ch17

2. Go to the lab in the Python directory

cd net.jgp.books.spark.ch17/src/main/python/lab100_orders/

3. Execute the following spark-submit command to create a jar file to our this application

spark-submit orderStatisticsApp.py

NOTE:- If you want to run delta lake examples, please use the following spark-submit command:

spark-submit --driver-class-path /tmp/jars/io.delta_delta-core_2.12-0.7.0.jar  --packages io.delta:delta-core_2.12:0.7.0 feedDeltaLakeApp.py

If you want to run PostgreSQL example, please use the following spark-submit command:

spark-submit --driver-class-path /tmp/jars/org.postgresql_postgresql-42.1.4.jar  --packages org.postgresql:postgresql:42.1.4 appendDataJdbcPrimaryKeyApp.py

Running the lab in Scala

Prerequisites:

You will need:

• git.
• Apache Spark (please refer Appendix P - "Spark in production: installation and a few tips").

1. Clone this project

git clone https://github.com/jgperrin/net.jgp.books.spark.ch17

2. cd net.jgp.books.spark.ch17

3. Package application using sbt command

 sbt clean assembly

4. Run Spark/Scala application using spark-submit command as shown below:

spark-submit --class net.jgp.books.spark.ch17.lab100_export.ExportWildfiresScalaApplication target/scala-2.12/SparkInAction2-Chapter17-assembly-1.0.0.jar  

Notes:

1. [Java] Due to renaming the packages to match more closely Java standards, this project is not in sync with the book's MEAP prior to v10 (published in April 2019).
2. [Scala, Python] As of MEAP v14, we have introduced Scala and Python examples (published in October 2019).

---

Follow me on Twitter to get updates about the book and Apache Spark: @jgperrin. Join the book's community on Facebook or in Manning's community site.