EPFL Course - Optimization for Machine Learning - CS-439

May 1, 2026 · View on GitHub

Official coursebook information

Lectures: Fri 13:15-15:00 in CO2

Exercises: Fri 15:15-17:00 in BC01

This course teaches an overview of modern mathematical optimization methods, for applications in machine learning and data science. In particular, scalability of algorithms to large datasets will be discussed in theory and in implementation.

Team

Contents:

Convexity, Gradient Methods, Proximal algorithms, Subgradient Methods, Stochastic and Online Variants of mentioned methods, Coordinate Descent, Frank-Wolfe, Accelerated Methods, Primal-Dual context and certificates, Lagrange and Fenchel Duality, Second-Order Methods including Quasi-Newton Methods, Derivative-Free Optimization.

Advanced Contents:

Parallel and Distributed Optimization Algorithms, Federated Learning

Computational Trade-Offs (Time vs Data vs Accuracy), Lower Bounds

Non-Convex Optimization: Convergence to Critical Points, Alternating minimization, Neural network training

Program:

NrDateTopicMaterialsExercises
120.2.Introduction, Convexityslideslab00
227.2.Gradient Descentslideslab01
36.3.Projected Gradient Descentslideslab02
413.3.Proximal and Subgradient Descentslideslab03
520.3.Stochastic Gradient Descent, Non-Convex Optimizationslideslab04
627.3.Non-Convex Optimizationslideslab05
.3.4.Easter vacation-
.10.4.Easter vacation-
717.4.Newton's Method & Quasi-Newtonslideslab06
824.4.Frank-Wolfe & Muonslideslab07
91.5.Coordinate Descent-lab08
108.5.Lower Bounds and Accelerated Gradient Descent-lab09
1115.5.(Ascension Day Bridge Day)-Q & A Projects
1222.5.Gradient free and adaptive methods--
1329.5.Optimization for real-world AI model training--

Lecture Notes:

The course is based on the following lecture notes.

Videos:

The videos of the lectures for each week will be available.

Exercises:

The weekly exercises consist of a mix of theoretical and practical Python exercises for the corresponding topic each week (starting week 2). Solutions to exercises are available in the lab folder.

Forum:

Discussion forum (EPFL internal)

Project:

A mini-project will focus on the practical implementation: Here we encourage students to investigate the real-world performance of one of the studied optimization algorithms or variants, helping to provide solid empirical evidence for some behaviour aspects on a real machine-learning task. The project is mandatory and done in groups of 3 students. It will count 30% to the final grade. Project reports (3 page PDF) are due June 12th. Here is a detailed project description.

Assessment:

Session Exam. Exam Date: Thursday 18.06.2026 from 09h15 to 12h15 (CE 1 515).

Format: Closed book. Theoretical questions similar to exercises. You are allowed to bring one cheat sheet (A4 size paper, both sides can be used).

For practice: