Challenge appointed as an assignment from Data Engineering course at SoulCode Academy
- The Dataset must be saved in a cloud environment (Cloud Storage)
- The original and transformed data files must be saved in MongoDB Atlas in different collections
- All DataFrames must be saved in a Cloud Storage bucket
- The file is in another language and must have its data translated into Portuguese-BR
- Perform extraction correctly for a data frame
- Check for inconsistent data and perform cleanup to NaN or NA
- Perform the drop (if necessary) of columns from the data frame by commenting on the reason for the deletion
- All steps must be commented
- The DataFrame structure must be assembled using the StructType.
- Check for inconsistent, null data and clean it up.
- Check the need for dropping out columns or rows. If necessary, comment on why.
- Perform the name change of at least 2 columns
- You must create at least two new columns containing some relevant information about the other existing columns (Grouping, Aggregation or Joins Functions). (Use your analytical skills)
- You should use filters, sorting and grouping, bringing data relevant to the business in question. (Use your analytical skills)
- Use at least two Window Functions
- Use at least 5 different queries using SparkSQL, commenting on why you choose these functions and explaining what each query does.