Optimization: introduction

  • Optimization considers the problem $$ \begin{eqnarray*} \text{minimize } f(\mathbf{x}) \\ \text{subject to constraints on } \mathbf{x} \end{eqnarray*} $$

  • Possible confusion:

    • We (statisticians) talk about maximization: $\max \, L(\mathbf{\theta})$.
    • People talk about minimization in the field of optimization: $\min \, f(\mathbf{x})$.
  • Why is optimization important in statistics?

    • Maximum likelihood estimation (MLE).
    • Maximum a posteriori (MAP) estimation in Bayesian framework.
    • Machine learning: minimize a loss + certain regularization.
    • ...
  • Our major goal (or learning objectives) is to

    • have a working knowledge of some commonly used optimization methods:
      • Newton type algorithms
      • expectation-maximization (EM) algorithm
      • majorization-minimization (MM) algorithm
      • quasi-Newton algorithm
      • conjugate gradient (CG) type algorithms
      • convex programming with emphasis in statistical applications
    • implement some of them in homework
    • get to know some optimization tools in Julia
  • What's not covered in this course:

    • Optimality conditions
    • Convergence theory
    • Convex analysis
    • Modern algorithms for large scale problems (ADMM, CD, proximal gradient, stochastic gradient, ...)
    • Combinatorial optimization
    • Stochastic algorithms
    • Many others
  • You must take advantage of the great resources at UCLA.

    • Lieven Vandenberghe: EE236A (Linear Programming), EE236B (Convex Optimization), EE236C (Optimization Methods for Large-scale Systems). One of the best places to learn convex programming.
    • Kenneth Lange: Biomath 210 (Optimization Methods in Biology). The best place to learn MM type algorithms.
    • Wotao Yin in math.