nomad128 / msai22

Optimization methods course @ MIPT online MSc program.

title	permalink	nav_order
🏡 Home	/	1

Optimization methods. MSAI 2022

The course is a summary of state-of-the-art results and approaches in solving applied optimization problems. Despite the focus on applications, the course contains the necessary set of theoretical foundations to understand why and how given methods work.

Classes are taken online twice a week for an hour and a half. In the lecture session a brief theoretical introduction to the topic is discussed, in the practical interactive session students solve problems on the topic on their own with Q&A.

Program

Introductory session. 📝 Notes. 📼 Video

Week 1

🦄 Lecture	🏛 Seminar
Brief recap of matrix calculus. 📄 presentation 📝 notes 📼 video	Examples of matrix and vector derivatives. 📼 video 🐍 code
Idea of automatic differentiation. 📄 presentation 📝 notes 🐍 code 📼 video	Work with automatic differentiation libraries - jax, pytorch, autograd. 📼 video🐍 code

Week 2

🦄 Lecture	🏛 Seminar
Markowitz portfolio theory 📄 presentation 📝 notes 📼 video🐍 code	Building a portfolio based on a real-world data. 📼 video 🐍 code

Week 3

🦄 Lecture	🏛 Seminar
Applications of linear programming. 📄 presentation 📝 notes 📼 video🐍 code	LP applications exercises: selecting TED talks as LP, production planning. 📼 video🐍 code

Week 4

🦄 Lecture	🏛 Seminar
Zero order methods: simulated annealing, evolutionary algorithms, genetic algorithm. Idea of Nelder Mead algorithm. 🐍 code ML models hyperparameter search with nevergrad 🐍 code and optuna 🐍 code. 📄 presentation 📝 notes 📼 video	ML models Hyperparameter search with optuna and keras. 📼 video🐍 code

Week 5

🦄 Lecture	🏛 Seminar
Newton method. 🐍 code Quasi-Newton methods. 🐍 code 📄 presentation 📝 notes 📼 video	Implementation of the damped Newton method. Finding the analytical center of a set. Convergence study. Comparison with other methods. Benchmarking of quasi-Newtonian methods. 📼 video🐍 code

Week 6

🦄 Lecture	🏛 Seminar
Stochastic gradient descent method. Batches, epochs, schedulers. Nesterov Momentum and Polyak Momentum. Accelerated gradient method. Adaptive stochastic methods. Adam, RMSProp, AdaDelta. 🐍 code 📄 presentation 📝 notes 📼 video	A convergence study of the SGD. Hyperparameter tuning. Convergence study of accelerated methods in neural network training. Convergence study of adaptive methods in neural network training. 📼 video🐍 code

Week 7

🦄 Lecture
The landscape of the loss function of a neural network. Neural network fine-tuning aka transfer learning. Neural style transfer. 🐍 code Using GANs to train density distribution on the plane. Generating new pokemons using deep neural networks. 🐍 code Visualizing the projection of the loss function of a neural network on a straight line and a plane. 🐍 code 📼 video

• 📧 Chat
• 👨‍💻 Github