Integrating Exponential Dispersion Models to Latent Structures

Abstract

Latent variable models have two basic components: a latent structure encoding a hypothesized complex pattern and an observation model capturing the data distribution. With the advancements in machine learning and increasing availability of resources, we are able to perform inference in deeper and more sophisticated latent variable models. In most cases, these models are designed with a particular application in mind; hence, they tend to have restrictive observation models. The challenge, surfaced with the increasing diversity of data sets, is to generalize these latent models to work with different data types. We aim to address this problem by utilizing exponential dispersion models (EDMs) and proposing mechanisms for incorporating them into latent structures.

In Chapter 2, we show that the density of steep EDMs can be expressed with a Bregman divergence. Based on this relationship, we parametrize families of steep EDMs for various data types. We then use these families in the mixture model setting and propose an expectation-maximization algorithm (AdaCluster) that can identify the underlying distribution of each attribute in a heterogeneous data set.

In Chapter 3, we generalize hierarchical Poisson factorization, a Bayesian non-negative matrix factorization model, by compounding the original Poisson output with EDMs. We show that the proposed model is particularly effective for large data sets with extreme sparsity and arbitrary data distribution.

In Chapter 4, we use the compound-Poisson-EDM structure within the context of missing data. We show that an arbitrary data-generating model with EDM output---such as Gaussian mixture model, probabilistic matrix factorization, Poisson mixture model or linear regression model---can be coupled with a Poisson factorization encoding the missing-data pattern through compounding. In particular, we argue that the heteroscedastic impact of missing-data pattern on the dispersion of observation variable can be captured with the proposed model.