sadamingh / msds697_distributed_data_systems_2022

2022 USF MSDS697 Class Examples

2022 USF MSDS697 Class Examples

Day 1 Example

postgres json example and its limitation

The need of a NoSQL database in some cases

---

Day 2 Example

MongoDB data types

Basic MongoDB CRUD Operatons (will be continued in Day3)

---

Day 3 Example

Design and implement more complex queries - focusing on query and projection operators.

---

Day 4 Example

Design and create aggregation pipeline.

Create and drop index to improve query performance.

---

Day 6 Example

Load data from S3 and preprocess using Pyspark.

Store created aggregates into MongoDB.

---

Day 7 Example

Creating DataFrames (2 ways)

DataFrame API Basics

---

Day 8 Example

Apply SQL functions with DataFrame - Scalar/Aggregate/Window/UDF/Join

Registering DataFrame in the Table Catalog

Loading/Writing Data using SparkSQL - csv, parquet, json

---

Day 9 Example

Load/Write data to S3/MongoDB locally.

Apply logistic regression to adult.dat using Spark ML.

---

Day 10 Example

Apply decision tree/Random Forest/ and KMeans using penbased.dat

Create Spark ML pipeline.

---

Day 12 Example

Create H2O dataframes.

Apply H2O AutoML and evaluate/explain models.