ELBO: maximizing a lower bound is minimizing KL divergence
Published:
In latent-variable models, we often want to maximize the marginal likelihood \(p_\theta(x) = \int p_\theta(x, z)\,dz\), but the integral is usually intractable. Variational inference replaces the true posterior \(p_\theta(z \mid x)\) with a tractable approximation \(q_\phi(z \mid x)\), and turns the problem into maximizing a lower bound.
This post derives the standard identity:
\[\log p_\theta(x) = \underbrace{\mathcal{L}(\theta,\phi; x)}_{\text{ELBO}} \;+\; \underbrace{KL\!\left(q_\phi(z\mid x)\,\|\,p_\theta(z\mid x)\right)}_{\ge 0}\]so maximizing the ELBO is equivalent to minimizing a KL divergence.
Step 1: start from the log marginal likelihood
We begin with:
\[\log p_\theta(x) = \log \int p_\theta(x, z)\,dz.\]Introduce any distribution \(q_\phi(z\mid x)\) whose support is contained in the support of \(p_\theta(x,z)\), and multiply/divide inside the integral:
\[\log p_\theta(x) = \log \int q_\phi(z\mid x)\,\frac{p_\theta(x,z)}{q_\phi(z\mid x)}\,dz = \log \mathbb{E}_{z\sim q_\phi(z\mid x)}\!\left[\frac{p_\theta(x,z)}{q_\phi(z\mid x)}\right].\]Step 2: apply Jensen’s inequality to get a lower bound
Since \(\log\) is concave, Jensen’s inequality gives:
\[\log \mathbb{E}_{q}\!\left[\frac{p_\theta(x,z)}{q(z\mid x)}\right] \ge \mathbb{E}_{q}\!\left[\log \frac{p_\theta(x,z)}{q(z\mid x)}\right].\]Define the evidence lower bound (ELBO):
\[\mathcal{L}(\theta,\phi; x) \;:=\; \mathbb{E}_{z\sim q_\phi(z\mid x)}\!\left[\log p_\theta(x,z) - \log q_\phi(z\mid x)\right].\]So we have the fundamental inequality:
\[\log p_\theta(x) \ge \mathcal{L}(\theta,\phi; x).\]Step 3: express the gap as a KL divergence
The cleanest way to see what the ELBO is doing is to subtract it from \(\log p_\theta(x)\):
\[\begin{align*} \log p_\theta(x) - \mathcal{L}(\theta,\phi; x) &= \log p_\theta(x) - \mathbb{E}_{q}\!\left[\log p_\theta(x,z) - \log q(z\mid x)\right] \\ &= \mathbb{E}_{q}\!\left[\log p_\theta(x)\right] - \mathbb{E}_{q}\!\left[\log p_\theta(x,z)\right] + \mathbb{E}_{q}\!\left[\log q(z\mid x)\right] \\ &= \mathbb{E}_{q}\!\left[\log q(z\mid x) - \log \frac{p_\theta(x,z)}{p_\theta(x)}\right] \\ &= \mathbb{E}_{q}\!\left[\log q(z\mid x) - \log p_\theta(z\mid x)\right] \\ &= KL\!\left(q(z\mid x)\,\|\,p_\theta(z\mid x)\right). \end{align*}\]Because KL divergence is always non-negative, this simultaneously proves:
- Lower bound: \(\log p_\theta(x) \ge \mathcal{L}(\theta,\phi; x)\)
- Tightness condition: the bound is tight iff \(q_\phi(z\mid x) = p_\theta(z\mid x)\) almost everywhere.
Equivalent ELBO forms you’ll see in ML papers
Using \(\log p_\theta(x,z)=\log p_\theta(x\mid z)+\log p_\theta(z)\), we can rewrite:
\[\mathcal{L}(\theta,\phi; x) = \mathbb{E}_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)] - KL\!\left(q_\phi(z\mid x)\,\|\,p_\theta(z)\right).\]This is the familiar “reconstruction minus regularization” view used in VAEs.