Math @ Duke

Publications [#354089] of Holden Lee
Papers Published
 Kuditipudi, R; Wang, X; Lee, H; Zhang, Y; Li, Z; Hu, W; Arora, S; Ge, R, Explaining landscape connectivity of lowcost solutions for multilayer nets,
Advances in Neural Information Processing Systems, vol. 32
(January, 2019)
(last updated on 2022/01/28)
Abstract: Mode connectivity (Garipov et al., 2018; Draxler et al., 2018) is a surprising phenomenon in the loss landscape of deep nets. Optima'at least those discovered by gradientbased optimization'turn out to be connected by simple paths on which the loss function is almost constant. Often, these paths can be chosen to be piecewise linear, with as few as two segments. We give mathematical explanations for this phenomenon, assuming generic properties (such as dropout stability and noise stability) of welltrained deep nets, which have previously been identified as part of understanding the generalization properties of deep nets. Our explanation holds for realistic multilayer nets, and experiments are presented to verify the theory.


dept@math.duke.edu
ph: 919.660.2800
fax: 919.660.2821
 
Mathematics Department
Duke University, Box 90320
Durham, NC 277080320

