Quasi-Newton methods for stochastic optimization

In statistics, response surface methodology (RSM) is a popular approach to stochastic optimization. RSM uses least-squares regression to construct local linear or quadratic approximations of the objective function. In standard practice, the objective function is assumed to be quadratic and several iterations using linear approximations culminate in a final iteration using a quadratic approximation. If the objective function is more complicated, then it is natural to construct a sequence of quadratic approximations. We study two techniques for constructing such a sequence. One uses quadratic regression to construct second-order approximations directly from noisy function values; the other uses linear regression to construct first-order approximations from noisy function values, then approximates second-order terms by the BFGS updating formula. Results from numerical experiments suggest that the second approach performs more efficiently than the first approach. Pathologies occasionally occur. We argue that these pathologies motivate the use of various safeguards.