Optimization: introduction

• Optimization considers the problem $$ \begin{eqnarray*} \text{minimize } f(\mathbf{x}) \\ \text{subject to constraints on } \mathbf{x} \end{eqnarray*} $$

• Possible confusion:

  • We (statisticians) talk about maximization: $\max \, L(\mathbf{\theta})$.
  • People talk about minimization in the optimization world: $\min \, f(\mathbf{x})$.

• Why is optimization important in statistics?

  • Maximum likelihood estimation (MLE).
  • Maximum a posteriori (MAP) estimation in Bayesian framework.
  • Machine learning: minimize a loss + certain regularization.
  • ...

• Our major goal (or learning objectives) is to

  • have a working knowledge of some commonly used optimization methods:
    • Newton type algorithms
    • expectation-maximization (EM) algorithm
    • majorization-minimization (MM) algorithm
    • quasi-Newton algorithm
    • conjugate gradient (CG) type algorithms
    • convex programming with emphasis in statistical applications
  • implement some of them in homework
  • get to know some optimization tools in Julia

• What's not covered in this course:

  • Optimality conditions
  • Convergence theory
  • Convex analysis
  • Modern algorithms for large scale problems (ADMM, CD, proximal gradient, stochastic gradient, ...)
  • Combinatorial optimization
  • Stochastic algorithms
  • Many others

• You must take advantage of the great resources at UCLA.

  • Lieven Vandenberghe: EE236A (Linear Programming), EE236B (Convex Optimization), EE236C (Optimization Methods for Large-scale Systems). One of the best places to learn convex programming.
  • Kenneth Lange: Biomath 210 (Optimization Methods in Biology). The best place to learn MM type algorithms.
  • Wotao Yin in math.