npto04 / pyspark-pandas-challenge

Challenge appointed as assignment from Data Engineering course at SoulCode Academy

Data wrangling challenge with Python and GCP

Challenge appointed as an assignment from Data Engineering course at SoulCode Academy

Infrastructural level

• The Dataset must be saved in a cloud environment (Cloud Storage)
• The original and transformed data files must be saved in MongoDB Atlas in different collections
• All DataFrames must be saved in a Cloud Storage bucket

Pandas level

• The file is in another language and must have its data translated into Portuguese-BR
• Perform extraction correctly for a data frame
• Check for inconsistent data and perform cleanup to NaN or NA
• Perform the drop (if necessary) of columns from the data frame by commenting on the reason for the deletion
• All steps must be commented

Spark level

• The DataFrame structure must be assembled using the StructType.
• Check for inconsistent, null data and clean it up.
• Check the need for dropping out columns or rows. If necessary, comment on why.
• Perform the name change of at least 2 columns
• You must create at least two new columns containing some relevant information about the other existing columns (Grouping, Aggregation or Joins Functions). (Use your analytical skills)
• You should use filters, sorting and grouping, bringing data relevant to the business in question. (Use your analytical skills)
• Use at least two Window Functions
• Use at least 5 different queries using SparkSQL, commenting on why you choose these functions and explaining what each query does.