Math @ Duke

Publications [#361655] of Holden Lee
Papers Published
 Lee, H; Pabbaraju, C; Sevekari, A; Risteski, A, Universal Approximation for Logconcave Distributions using
Wellconditioned Normalizing Flows
(July, 2021)
(last updated on 2022/05/22)
Abstract: Normalizing flows are a widely used class of latentvariable generative
models with a tractable likelihood. Affinecoupling (Dinh et al, 201416)
models are a particularly common type of normalizing flows, for which the
Jacobian of the latenttoobservablevariable transformation is triangular,
allowing the likelihood to be computed in linear time. Despite the widespread
usage of affine couplings, the special structure of the architecture makes
understanding their representational power challenging. The question of
universal approximation was only recently resolved by three parallel papers
(Huang et al.,2020;Zhang et al.,2020;Koehler et al.,2020)  who showed
reasonably regular distributions can be approximated arbitrarily well using
affine couplings  albeit with networks with a nearlysingular Jacobian. As
illconditioned Jacobians are an obstacle for likelihoodbased training, the
fundamental question remains: which distributions can be approximated using
wellconditioned affine coupling flows?
In this paper, we show that any logconcave distribution can be approximated
using wellconditioned affinecoupling flows. In terms of proof techniques, we
uncover and leverage deep connections between affine coupling architectures,
underdamped Langevin dynamics (a stochastic differential equation often used to
sample from Gibbs measures) and H\'enon maps (a structured dynamical system
that appears in the study of symplectic diffeomorphisms). Our results also
inform the practice of training affine couplings: we approximate a padded
version of the input distribution with iid Gaussians  a strategy which
Koehler et al.(2020) empirically observed to result in betterconditioned
flows, but had hitherto no theoretical grounding. Our proof can thus be seen as
providing theoretical evidence for the benefits of Gaussian padding when
training normalizing flows.


dept@math.duke.edu
ph: 919.660.2800
fax: 919.660.2821
 
Mathematics Department
Duke University, Box 90320
Durham, NC 277080320

