Welcome to OPTML++! This is the homepage for the research seminar plus reading group on: Optimization for Machine Learning (the (++) stands for related areas (statistics, signal processing, computer vision, robotics, information theory, engineering, among others).


Date Topic Links Presenter
14 Sep Introduction; Outline of topics; Recap of basics Slides Suvrit
21 Sep First-order methods, smooth and nonsmooth Slides Suvrit
28 Sep Operator splitting methods Slides Suvrit
05 Oct Summary of stochastic, incremental methods Slides Suvrit
12 Oct No meeting - -
19 Oct Sparse-data ML algorithms Slides Suvrit
26 Oct Locality sensitive hashing Slides Ilya
02 Nov Online learning, game theory, and combinatorial optimization TBA Swati
09 Nov Matrix sketching, randomized linear algebra Slides Chengtao
16 Nov Online Methods for Prediction in Evolving (Social) Networks TBA Sasha
23 Nov Incremental Methods: Why Random Reshuffling Beats Stochastic Gradient Descent Paper 1; Paper 2 Mert
30 Nov Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints Slides Matt


  • Basic convex analysis and optimization
    • Basics of convex sets and functions
    • Key convex functions
    • Constructing and recognizing convex functions
    • Some convex geometric problems
  • Scalable convex optimization
    • First-order methods
    • Parallel
    • Distributed
    • Online learning -- maybe Sasha Rakhlin can help!
    • Sublinear methods
    • Incremental gradient methods
    • Stochastic gradient methods
    • Large-scale Semidefinite programming
  • Scalable nonconvex optimization
    • online optimization
    • dictionary learning models and algorithms
    • deep learning algorithms
    • nonconvex SGD, incremental gradient, and other algorithms
  • Non-Euclidean optimization
    • Hadamard manifolds
    • Riemannian manifolds
    • Metric spaces of nonpositive curvature
    • Wasserstein metric based problems (probability, geometry)
    • Explicit examples, such as large scale Gaussian mixtures
  • Interplay of convex and combinatorial optimization
    • Submodular problems -- potentially led by Stefanie Jegelka
    • Max-flow, graph algorithms -- potentially led by Aleksander MÄ…dry
    • Convex relaxations
  • Applications
    • Large-scale regression problems
    • Optimization in deep learning
    • Dictionary learning
    • Applications from machine learning, statistics
    • Applications from finance, signal processing, control, information theory, ...
  • Software, design and implementation
    • Data Science platforms: AWS, Azure, Spark, MLBase, Parameter Server, Dato
    • Lessons in implementing asynchronous optimization algorithms
    • Multicore CPU considerations
    • GPU and GPGPU considerations
    • Delay sensitive algorithms and design
    • Other useful software and hardware ideas
  • Sublinear algorithms, randomized methods
    • Randomized linear algebra
    • Property testing
    • Hashing, SimHash, dimensionality reduction, etc.
    • Approximation algorithms
  • Exploiting symmetry in optimization
    • Optimizing over permutations
    • Schur-convexity, majorization
    • Optimizing over groups
    • Scattering transforms, application to deep learning
  • Large-scale sampling, MCMC
    • Parallel Gibbs sampling procedures
    • Large-scale multivariate gaussians
    • MCMC via the optimization lens
    • Scalable determinantal sampling
  • Open problems in optimization, ML, statistics, Information theory
    • Entropy inequalities
    • Convexity conjectures
    • Global optimization problems