tdhock / cs570-spring-2022

Computer Science CS 570-002 (10752) at Northern Arizona University, Spring 2022

Topic: Advanced Intelligent Systems (Deep Learning)

Dates: Jan 10, 2022-May 6, 2022

Meeting time/place: TuTh 8:00AM - 9:15AM, SBS-Raul H. Castro, Rm 310.

Syllabus: Google Doc.

Discord: https://discord.gg/gP3kChNmxd

Textbooks

These provide background/theory about the algorithms we study in this class.

The readings are from Deep Learning by Goodfellow, et al., which is freely available online.

You should have some knowledge of computational complexity (big O notation), so please read the following if you need to review:

• The SICP book, 1.2.3 “Orders of Growth,” has a brief description in general terms (not specific to machine learning).
• The CLRS book has a more detailed description in Chapter 3, “Growth of Functions” (not specific to machine learning).

Weekly schedule of Homeworks and reading

Homework topics and readings for each week are listed below. The date of the Monday of each week is written. Each homework is due Friday of that week, 11:59PM.

• Jan 10, Homework week 1: installing software, reading CSV, data visualization.
  • Installation instructions, slides.
• Jan 17, Homework week 2: k-fold cross-validation, nearest neighbors and linear models.
  • Slides.
  • Reading DL5 (The most important parts are 5.2 and 5.3).
• Jan 24, Homework week 3: nearest neighbors.
  • Slides.
  • Reading DL5.
• Jan 31, Homework week 4: linear model with early stopping regularization.
  • Slides, interactive figure showing gradient descent for regression.
  • Reading DL5, DL4.3.
• Feb 7, week 5 review and exam.
  • Exam is 10 Feb 2022, in class. During the exam you are allowed to use one page of handwritten notes.
  • Practice exam PDF.
• Feb 14, Homework week 6: backpropagation algorithm, training neural networks from scratch. Other losses (multi-class, regression).
  • Slides.
  • Reading DL6.
• Feb 21, Homework week 7: implementing a simple auto-grad system.
  • Slides, section Automatic differentiation.
  • Reading DL6.5.
• Feb 28, Homework week 8: training neural networks using torch.
  • Slides, section Torch.
• Mar 7, week 9 review and exam.
  • Exam is 10 Mar 2022, in class. During the exam you are allowed to use one page of handwritten notes.
  • Practice exam 2 PDF.
• Mar 14, spring break (no homework).
• Mar 21, Homework week 10: regularizing neural networks.
  • Reading DL7.
• Mar 28, Homework week 11: optimization.
  • Slides.
  • Reading DL8.
• Apr 4, Homework week 12: convolutional networks.
  • Slides.
  • Reading DL9.
• Apr 11, Homework week 13: recurrent networks.
  • Reading DL10.
• Apr 18, Homework week 14: interpretability via attention mechanism.
  • Readings: Yang et al, 2016, GRU vs LSTM, PyTorch translation attention tutorial using GRU, Another attention tutorial using LSTM.
• Apr 25, reading week, exam review Thurs Apr 28.
• Final exam, Thursday May 5, 8-9:30AM.
• Other topics:
  • Residual Learning
  • Pre-trained word embeddings. Word embeddings tutorial, How to use Pre-trained Word embeddings in Pytorch, Efficient Estimation of Word Representations in Vector Space, tutorial on implementing that paper.

Rubric for homeworks

Your content and responses to each homework question will be graded as follows

• Full credit for figures which show correct results, along with code which is correct and is of high quality.
• This General Usage Rubric will be used to grade the code quality/efficiency in each of your homeworks, -5 for each violation of these good coding rules.
• Some code and figures/results/answers, but clearly incorrect, -5 to -10.
• Missing code or figure/results/answers, -10 to -20.
• Missing code and figure/results/answers, -20 to -40.

Software

The links below provide practical advice about how to write the code necessary for the homeworks, and please read my instructions to install all of the necessary software.

Python documentation and introductory tutorials:

• numpy is a python module for multi-dimensional arrays (vectors, matrices, etc). It is useful for storing numeric data sets and doing various computations in learning algorithms (vectorized functions, matrix multiplication, etc). NumPy: the absolute basics for beginners, W3Schools, TutorialsPoint.
• pandas is a python module for data reading and manipulation. Whereas every column in a numpy matrix is the same type (typically double), columns in a pandas DataFrame can be different types (string, int, etc). 10 minutes to pandas, W3Schools, TutorialsPoint.
• janitor is a python module for data cleaning and reshaping. It is useful for getting data into the right format for learning or visualization.
• plotnine is a python module for data visualization based on the grammar of graphics, similar to ggplot2 in R: docs, tutorials, My 2021 blog post comparing R and python libraries for data reshaping and visualization.
• Scikit-learn is a python module which implements various standard machine learning algorithms. User guide: Nearest Neighbors, Linear Models.
• torch is a python module which is similar to numpy, but with two key differences which are useful for machine learning: (1) makes it easy to use automatic differentiation, and (2) computations can be easily performed on GPU for increased speed. Introduction to pytorch tutorial.

General Questions and Answers (FAQ)

• Are there any materials online from previous versions of this class which may be useful? Here are some video screencasts from Spring 2020 (R/keras was used instead of python/numpy/torch).
  • Neural networks using keras in R.
  • Number of hidden units is a regularization parameter.
  • Convolutional Neural Networks in R.
• Can I copy/modify the code demos from in class and from your screencast videos? Yes you can copy/modify these code demos for your homework, since they are a part of the class material. But in general, copying without giving a clear citation of your source is plagiarism (and will be pursued as an academic integrity violation).
• Can I collaborate with my classmates on the homework? Yes, as long as your share ideas and not code/results. More specifically, homeworks are individual assignments which should be your own work, so it is strictly forbidden to copy code/results from classmates or internet sources. However it is encouraged to discuss ideas related to lectures and homework solutions with classmates.

How to ace this class

Before class you should prepare by doing the suggested readings/videos. When you do that, write a summary in your own words of every section. Also write questions that you have during your reading so you can ask in class or office hours.

During class, take notes by writing what you understood in your own words. Also I would suggest to ask questions in class as soon as you need clarification.

After class, you should review your notes with one of your classmates (ask one of the students who seem to be correctly answering a lot of questions in class). Ask each other questions and try to teach/summarize some of the material with each other – that is one of the best ways to learn.

Finally after doing all of the above, please come to office hours (see syllabus), or email me to schedule a meeting.