Abstract
The Langevin equation is a stochastic differential equation that allows for the sampling of exponential distributions. The velocity field is proportional to the energy gradient of the distribution; in this case, the Langevin equation has a unique steady state equal to this distribution. Transport via the Langevin equation converges to this steady state, but this convergence can be very slow when the energy is not convex. This approach is therefore difficult to apply for sampling distributions with non-convex energy in high dimensions.
The score diffusion with denoising algorithm is an alternative approach based on a homotopic principle. The original probability distribution is transported toward a Gaussian white noise distribution using a stochastic Ornstein–Uhlenbeck equation that gradually adds noise. Through the Fokker–Planck equation, it is shown that this transport of the probability distribution is invertible. The inverse transport is obtained using a stochastic differential equation whose velocity field depends on the score—that is, on the gradient of the log-probability of the noisier distribution at the given time. The score diffusion algorithm implements this inverse transport, which allows new data to be generated from Gaussian white noise.
The main challenge of this approach is calculating the score of the probability distribution. We prove the Tweedie–Robbins–Miyasawa identity, which relates the score to the conditional expectation of a signal, given that the signal is contaminated by additive white noise. This is the denoising estimator that minimizes the mean squared error.
