6.881 Optimization for Machine Learning

INTRODUCTION

Welcome to 6.881. This is the homepage for the course: Optimization for Machine Learning (OPTML) that I am teaching in Spring 2020. OPTML covers topics from convex, nonconvex, continuous and combinatorial optimization. The topics covered range from foundational material to research-edge topics.

ORGANIZATION

Lectures: MW 1:00PM–2:30PM
Virtual Location: (Touchstone reqd): https:mit.zoom.usj263388385
Historical Location: 32-155
Instructor: Suvrit Sra
Teaching Assistants: Sarath Pattathil and Jingzhao Zhang
Grading policy: Homeworks (40%), Project (50%), Virus (10%)
Course site on stellar
Homework Policy
Project Guidelines
Main References:
• Yurii Nesterov. Lectures on convex optimization. Springer 2018
• Stephen Boyd, Lieven Vandenberghe. Convex Optimization. Cambridge University Press. 2003.

Schedule (Subject to change)

Lec Date Topic Links/Notes
1 Feb 3 Foundations: convex sets, functions L1A; L1B; HW1 Out
2 Feb 5 Foundations: conjugates, subdifferentials. Basic problemsL2
3 Feb 10 Foundations: Weak duality, strong duality, minimax theorem L3
4 Feb 12 Foundations: Optimality conditions, KKT, conic duality L4
5 Feb 18 Nonconvex optimality & stationarity L5
6 Feb 19 Tractable nonconvex problems L6; HW1 In; HW2 Out
7 Feb 24 First-order methods: gradient descent and more L7
Feb 26 CANCELED
8 Mar 02 FOMs: acceleration and subgradient method L8
9 Mar 04 Stochastic gradient methods L9
10 Mar 09 Proximal methods, operator splitting L10; HW2 In; HW3 Out
11 Mar 11 Frank-Wolfe methods L11
Mar 16 COVID CANCELED US
Mar 18 COVID CANCELED US
Mar 23 Spring Break
Mar 25 Spring Break
12 Mar 30 Coordinate descent, block CD L12; Video
13 Apr 01 EM method, CCCP, and related methods
14 Apr 06 Scaling up via parallel, distributed optimization Project midterm reports
15 Apr 08 Zeroth order optimization L14; HW3 In; HW4 Out
16 Apr 13 Neural network optimization (SGD heuristics and tricks) L15
17 Apr 15 NN optimization: escaping saddle points L16;
Apr 20 No class, Patriots Day
18 Apr 22 Min-Max Problems I (convex-concave)
19 Apr 27 Min-Max Problems II (nonconvex)
20 Apr 29 Lower bounds and complexity
21 May 04 Non-Euclidean Optimization I HW4 In
22 May 06 Non-Euclidean Optimization II
23 May 11 Perspectives, Other Directions, Open Problems II Projects due