tiao553 / bigdata-k8s-kafka

In this repository i will allocate my architecture project, all on a kubernetes cluster.

📌 About me

Well, I'm 22 years old, living in Ipanema, Minas Gerais, Brazil.

I'm graduating in Control and Automation Engineering and I've always been passionate about technology but the career opportunities always took me to another path, until I finally decided to drop everything and go after my dream. I am currently working as a Data Scientist at precato, a fintech company focused on the purchase of precatórios, where I work directly with python projects and test automation and model implementation.

Currently I invest my time in learning about data engineering, an area I fell in love with.

🛠️ Programing languages

---

Projects about the Data Science

• Analyzing the Violence in Rio de Janeiro [PT-BR]: link

  A descriptive analysis was made with the objective of understanding which were the main influencers of the high violence observed in Rio de Janeiro.

• Airbnb Data Analysis, Buenos Aires, Argentina [PT-BR]: link

  Descriptive analysis on the data provided by Airbnb in order to understand which neighborhoods are the best to rent according to price and location.

• Features engineering, learn interactively [EN-US]: link

  One way to improve the performance of a model is feature enginerring. In this article I show how to apply this technique in a simple interactive way.

Projects about the Data engineer

• [AWS] Building a lambda pipeline with CDK [PT-BR] : Link

  This pipeline architecture is widely used in cases where the data is integrated all in one zone. In addition, it has 3 storage stages (bronze, silver, gold). Curious? Go to the repositor at link above.

• [GCP] Hackathon A3data [PT-BR] : link

  A very cool challenge that contemplated the treatment and analysis of a large set of data. In this repository you can see my project carried out in this competition.

• Data Collection Pipeline in Yahoo Finance [PT-BR] : link

  Back to the origins. I entered the market with many cloud-oriented solutions. With that in mind I worked on this project creating instances of Hadoop, spark, and Hive to get an experience with these tools.

---

• Machine Learning and Data Science with Python of A to Z: Certificate

  With this course I was able to learn about a good part of a data scientist's pipeline considering from data preparation to application of supervised to unsupervised algorithms. In the course it was also proposed several use cases.

• TensorFlow: Machine Learning and Deep Learning with Python: Certificate

  Theory and practice of how to build artificial neural networks to solve real problems of the day with convolutional neural networks, recurrent neural networks, autoencoders, and robust generative adversarial networks using TensorFlow were proposed in the course.

• SQL and NoSQL Databases from basic to advanced: Certificate

  With course I learned to use different SQL and NoSQL Database Management Systems, model relational databases applying the five normal ways. [MySQL, PostgreSQL, SQLite, MongoDB,Redis, CouchDB,and Firehouse]

• HOWBootcamps Engenharia de dados: Certificate

  In this bootcamp it was proposed to build a pipeline with lambda architecture. Where we use the infrastructure as AWS CDK code, create a datalake and a datawarehouse, process data with databricks, orchestrate tasks with airflow and automate tests and processes.

---

✅ Get in contact with me! ✉️