6.881 Optimization for Machine Learning

INTRODUCTION

Welcome to 6.881. This is the homepage for the course: Optimization for Machine Learning (OPTML) that I am teaching in Spring 2020. OPTML covers topics from convex, nonconvex, continuous and combinatorial optimization. The topics covered range from foundational material to research-edge topics.

ORGANIZATION

Lectures: MW 1:00PM–2:30PM
Location: 4-265
Instructor: Suvrit Sra
Teaching Assistant: Jingzhao Zhang
Grading policy: Homeworks (40%), Quiz (10%), Project (50%)
Course site on stellar
Homework Policy
Project Guidelines
Main References:
• Yurii Nesterov. Lectures on convex optimization. Springer 2018
• Stephen Boyd, Lieven Vandenberghe. Convex Optimization. Cambridge University Press. 2003.

Schedule (Under construction! Not finalized yet!)

Lec Date Topic Links/Notes
1 Feb 3 Foundations: convex sets, functions Notes; HW1 Out
2 Feb 5 Foundations: conjugates, subdifferentials. Basic problems
3 Feb 10 Foundations: Weak duality, strong duality, minimax theorem
4 Feb 12 Foundations: Optimality conditions, KKT, conic duality
5 Feb 18 Nonconvex optimality & stationarity
6 Feb 19 Tractable nonconvex problems HW1 In; HW2 Out
7 Feb 24 First-order methods: gradient and stochastic gradient methods
8 Feb 26 Faster stochastic gradient methods
9 Mar 02 First-order methods: nonsmooth optimization
10 Mar 04 Proximal methods, operator splitting methods
11 Mar 09 Frank-Wolfe methods HW2 In; HW3 Out
12 Mar 11 Coordinate descent, block CD
13 Mar 16 EM method, CCCP, and related methods
14 Mar 18 QUIZ (in class)
Mar 23 No class, Spring Break
Mar 25 No class, Spring Break
15 Mar 30 Scaling up via parallel, distributed optimization
16 Apr 01 Zeroth order optimization HW3 In; HW4 Out
17 Apr 06 Neural network optimization (SGD heuristics and tricks)
18 Apr 08 NN optimization: escaping saddle points
19 Apr 13 Min-Max Problems I (convex-concave)
20 Apr 15 Min-Max Problems II (nonconvex)
Apr 20 No class, Patriots Day
21 Apr 22 Lower bounds and complexity HW4 In
22 Apr 27 Non-Euclidean Optimization I
23 Apr 29 Non-Euclidean Optimization II
24 May 04 Submodular optimization I
25 May 06 Submodular optimization II
26 May 11 Perspectives, Other Directions, Open Problems II Projects due