In this project, a ETL data pipeline was built on AWS using Apache Airflow to automate the process.
First an Instance was created on EC2.
On the image above, the instance is running and a public IPv4 address and some other information were generated including the Instance ID.
Then a S3 bucket named kaggle-dataset-team-bucket-ylm was created on the instance. This bucket houses the data to be used in this bucket.
The data needed for this project will be uploaded in the S3 bucket.