Data Engineering and DataOps

Data Engineering and DataOps Course: IDS 706, Fall 2022 by Noah Gift

Contact & Bio
This is the course syllabus.
These are the course projects
This is the week by week schedule
These are the resources
A guide to SQL Mastery
Question and Answer
Guest Speakers
These are the lecture notes
Example community recipes/example student projects
Notes for Coursera Course: Foundations of Data Engineering
Office Hours Via Zoom: Tuesday 5-7pm recorded

Course Description

This course covers servers as a method of encapsulating many courses in the program. Students learn to apply Data Engineering to a real-world project. This manifests itself through several goals: development of non-linear life-long learning, community building, portfolio development, and software engineering best practices.

Course Objectives

By the end of the course, you will be able to:

Develop data engineering solutions with a minimal and essential subset of the Python language and the Linux environment
Design scripts to connect and query a SQL database using Python
Use a scraping library in Python to read, identify and extract data from websites
Setup a provisioned Python project environment
Develop non-linear life-long learning skills
Build, share and present compelling portfolios using: Github, YouTube, and Linkedin.
Get Certified both Cloud Computing and a Big Data SQL solution

Prerequisites

Basic programming skills as well as basic linux skills. See optional readings/media to self-learn before class starts.

Diversity Statement

As educators and learners, we must share a commitment to diversity and equity, removing barriers to education so that everyone may participate fully in the community. In this course, we respect and embrace the unique experiences that brought each person here, including backgrounds, identities, learning styles, ways of expression, and academic interests. The broad spectrum of perspectives represented by our students enrich everyone’s experiences, and we strive to meet each perspective with openness and respect.

Required Readings & Media

Python

Data Science, Pandas, and Colab

https://learning.oreilly.com/videos/python-for-data/62062021VIDEOPAIML/

Python and DevOps
https://learning.oreilly.com/videos/python-devops-in/61272021VIDEOPAIML/
Python Command-line Tools
https://learning.oreilly.com/videos/learn-python-command-line/50102VIDEOPAIML/

Linux and Systems Engineering

Docker containers:
https://learning.oreilly.com/videos/learn-docker-containers/50103VIDEOPAIML/
Learn the Vim Text Editor:

https://learning.oreilly.com/videos/learn-vim-in/50100VIDEOPAIML/

Assignment Overview and Grading Breakdown

Grading and feedback turnaround will be one week from the due date. You will be notified if the turnaround will be longer than one week.

The discussion forums, written assignments, demo videos, and final project will be graded based on specific criteria or a rubric. The criteria or rubric for each type of assessment will be available in the course. To view the discussion forum rubric, click the gear icon in the upper right corner of the page and choose Show Rubric. The Written Assignment Rubric and Final Project Rubric will automatically appear on the page.

Activity	Percent of Final Grade
Weekly Async Discussion & Assignment	40%
Individual Projects (4)	40%
Team Final Project (1)	20%

Grade	Percentage	Total Points (out of 500)
A	93%–100%	465– 500 points
A-	90%–92%	450 – 464 points
B+	87%–89%	435 – 449 points
B	83%–86%	415 – 434 points
B-	80%–82%	400 – 414 points
C+	77%–79%	385 – 399 points
C	73%–76%	365 – 384 points
C-	70%–72%	350 – 364 points
F	0%–69%	0 – 349 points

Grading Scale

A	100%	to 93%
A-	< 93%	to 90%
B+	< 90%	to 87%
B	< 87%	to 83%
B-	< 83%	to 80%
C+	< 80%	to 77%
C	< 77%	to 73%
C-	< 73%	to 70%
F	< 70%	to 0%

Late Work Policy

Late work will be accepted only in the event of an instructor-approved absence. Contact your instructor as soon as possible, at least 24 hours in advance.

Online Communication and Interaction Expectations

Discussion Forums

The purpose of the discussion boards is to allow students to freely exchange ideas. It is imperative to remain respectful of all viewpoints and positions and, when necessary, agree to respectfully disagree. While active and frequent participation is encouraged, cluttering a discussion board with inappropriate, irrelevant, or insignificant material will not earn additional points and may result in receiving less than full credit. Frequency matters, but contributing content that adds value is paramount. Please remember to cite all sources—when relevant—in order to avoid plagiarism. Please post your viewpoints first and then discuss others’ viewpoints.

The quality of your posts and how others view and respond to them are the most valued. A single statement mostly implying “I agree” or “I do not agree” is not counted as a post. Explain, clarify, politely ask for details, provide details, persuade, and enrich communications for a great discussion experience. Please note, there is a requirement to respond to at least two fellow class members’ posts. Also, remember to cite all sources—when relevant—in order to avoid plagiarism.

Online Communication Etiquette

Beyond interacting with your instructor and peers in discussions, you will be expected to communicate by Canvas message, email, and sync session. Your instructor may also make themselves available by phone or text. In all contexts, keep your communication professional and respect the instructor’s posted availability.

Just as you expect a response when you send a message to your instructor, please respond promptly when your instructor contacts you. Your instructor will expect a response within two business days. This will require that you log into the course site regularly and set up your notifications to inform you when the instructor posts an announcement, provides feedback on work or sends you a message.

Participation and Attendance

This course will not meet at a particular time each week. All course goals, session learning objectives, and assessments are supported through classroom elements that can be accessed at any time. To measure class participation (or attendance), your participation in threaded discussion boards is required, graded, and paramount to your success in this course. Please note that any scheduled synchronous meetings are optional. While your attendance is highly encouraged, it is not required and you will not be graded on your attendance or participation.

Course Technology

This course will involve a number of different types of interactions. These interactions will take place primarily through Microsoft Teams. Please take the time to navigate through the course and become familiar with the course syllabus, structure, and content and review the list of resources below.

Required Technical Skills

Students in an online program should be able to do the following:

Communicate Teams discussion forums.
Use web browsers and navigate the World Wide Web.
Use the learning management system Teams.
Use Teams
Use applications to create documents and presentations (e.g., Microsoft Word, PowerPoint).
Use applications to share files (e.g., Box, Google Drive).

Required Digital Literacy Skills

In order to be successful in an online course, students should be able to locate, evaluate, apply, create, and communicate information using technology.

Students in this online course should be able to do the following:

Create, name, compose, upload, and attach documents.
Download, modify, upload, attach document templates.
Create, name, design, and upload presentations.
Access and download Course Reserve readings; read and review PDF documents.
Access and use a digital textbook.
Record and upload video taken with a webcam or smartphone.
Use the library website for scholarly research tasks.
Search the Internet strategically and assess the credibility of Internet sources.
Participate in threaded discussions by contributing text responses, uploading images, sharing links.
Coordinate remote work with peers, which may include contacting each other by e-mail, phone, video conference, or shared document.
Edit and format pages in the course site using a WYSIWIG (What You See is What You Get) editor or basic HTML.
Using a quizzing tool to answer multiple choice, true/false, matching, and short response questions within a given time period.
Follow directions to engage with a remote proctor by text, webcam, and audio.
Use a video player to review content, including pausing and restarting video.

kurhula / data-engineering-and-dataops Goto Github PK