Math @ Duke
|
Publications [#361655] of Holden Lee
Papers Published
- Lee, H; Pabbaraju, C; Sevekari, A; Risteski, A, Universal Approximation for Log-concave Distributions using
Well-conditioned Normalizing Flows
(July, 2021)
(last updated on 2022/08/06)
Abstract: Normalizing flows are a widely used class of latent-variable generative
models with a tractable likelihood. Affine-coupling (Dinh et al, 2014-16)
models are a particularly common type of normalizing flows, for which the
Jacobian of the latent-to-observable-variable transformation is triangular,
allowing the likelihood to be computed in linear time. Despite the widespread
usage of affine couplings, the special structure of the architecture makes
understanding their representational power challenging. The question of
universal approximation was only recently resolved by three parallel papers
(Huang et al.,2020;Zhang et al.,2020;Koehler et al.,2020) -- who showed
reasonably regular distributions can be approximated arbitrarily well using
affine couplings -- albeit with networks with a nearly-singular Jacobian. As
ill-conditioned Jacobians are an obstacle for likelihood-based training, the
fundamental question remains: which distributions can be approximated using
well-conditioned affine coupling flows?
In this paper, we show that any log-concave distribution can be approximated
using well-conditioned affine-coupling flows. In terms of proof techniques, we
uncover and leverage deep connections between affine coupling architectures,
underdamped Langevin dynamics (a stochastic differential equation often used to
sample from Gibbs measures) and H\'enon maps (a structured dynamical system
that appears in the study of symplectic diffeomorphisms). Our results also
inform the practice of training affine couplings: we approximate a padded
version of the input distribution with iid Gaussians -- a strategy which
Koehler et al.(2020) empirically observed to result in better-conditioned
flows, but had hitherto no theoretical grounding. Our proof can thus be seen as
providing theoretical evidence for the benefits of Gaussian padding when
training normalizing flows.
|
|
dept@math.duke.edu
ph: 919.660.2800
fax: 919.660.2821
| |
Mathematics Department
Duke University, Box 90320
Durham, NC 27708-0320
|
|