This is the forth project of Udacity's Data Engineering Nanodegree๐.
The purpose is to build an ETL pipeline that extracts data from S3
, processes them using Spark, and load the data back into S3(or hdfs)
as a set of dimensional tables.
A startup called ๐ตSparkify wants to analyze the data they've been collecting on songs and user activity on their new music streaming app.
The analytics team is particularly interested in understanding what songs users are listening to.
Currently, they don't have an easy way to query their data, which resides in a directory of JSON logs on user activity on the app, as well as a directory with JSON metadata on the songs in their app.
etl.py
loads data from S3, process that data usign Spark, and writes them back to S3(or hdfs).sample_parquet_data.ipynb
is a notebook to show sample data of each table in parquet file.
tablename | tbl_rows | sortkey1 | diststyle |
---|---|---|---|
artists | 9553 | artist_id | ALL |
songplays | 309 | start_time | KEY(song_id) |
songs | 14896 | None | KEY(song_id) |
time | 6813 | start_time | ALL |
users | 96 | user_id | ALL |
To use this, you need to set
output_data
path inetl.py
to your own S3 or hdfs
- Copy
$ git clone https://github.com/kjh7176/data_lakes_with_spark
# change current working directory
$ cd data_lakes_with_spark
- Execute ETL process
$ python etl.py
- Test
Opensample_parquet_data.ipynb
in order to check if data is inserted correctly.