Abstract
Gibbs energy probabilistic models are central to statistical physics. The probability density depends on an energy that is parameterized. The challenge is to estimate these parameters and sample the resulting distribution. So-called exponential distributions have an energy that is linearly dependent on the parameters. The parameters are estimated by maximizing the likelihood. We show that this likelihood is concave. Maximization can therefore be performed using a convergent upward gradient algorithm. However, this algorithm requires sampling the probability distribution at each gradient iteration.
The main challenge of this approach is sampling these exponential distributions. This can be done using a Monte Carlo Markov Chain (MCMC) algorithm, which guarantees convergence but can be extremely time-consuming and computationally intensive when the energy is not convex. This has significantly limited the use of this approach for generating high-dimensional data.
Generative Adversarial Networks (GANs) offer an alternative approach, which has achieved spectacular results in image generation. A first neural network generates an image from a sample of Gaussian white noise. A second network is trained to distinguish the images generated by the first network from the original samples of the probability distribution. A loss function pits the two networks against each other so that the generative network produces images that are difficult for the second network to distinguish from the original images. It has been shown that the optimal result is achieved with a generative network that exactly replicates the probability distribution of the data.
