Anthony Kuang

Anthony Kuang

Sitemap

A list of all the posts and pages found on the site. For you robots out there, there is an XML version available for digesting as well.

Pages

Posts

Crow whispering

less than 1 minute read

Published:

Keeping track of the papers I read. Probably mostly generative modeling related.

Coloring the Prokudin-Gorskii photo collection

6 minute read

Published:

Sergei Mikhailovich Prokudin-Gorskii, a Russian chemist and photographer, pioneered color photography in the early 20th century by capturing three exposures of each scene on glass plates through red, green, and blue filters, with the vision of combining them into full-color images. After leaving Russia in 1918, his collection was acquired by the Library of Congress and later digitized. Our goal is to transform these stitched negatives - images containing the three stacked exposures - into aligned, artifact-free color photographs.

ELBO: maximizing a lower bound is minimizing KL divergence

1 minute read

Published:

In latent-variable models, we often want to maximize the marginal likelihood \(p_\theta(x) = \int p_\theta(x, z)\,dz\), but the integral is usually intractable. Variational inference replaces the true posterior \(p_\theta(z \mid x)\) with a tractable approximation \(q_\phi(z \mid x)\), and turns the problem into maximizing a lower bound.

LaTeX preamble

less than 1 minute read

Published:

Attached is a simple \(\LaTeX\) preamble for my course notes, inspired by the Berkeley EECS 127 Reader.

\usepackage{times}
\usepackage{setspace}
\usepackage[left=1in,right=1in,top=0.8in,bottom=1in]{geometry}
\usepackage{tcolorbox}
\usepackage{graphicx}
\usepackage{amsmath,amssymb}

\onehalfspacing

\newtcolorbox{Definition}[1]{
  colback=red!7!white,
  colframe=red!7!white,
  coltitle=red!60!black,
  fonttitle=\bfseries,
  top=0mm,
  arc=0mm,
  title=\vspace{2mm}Definition ({#1}),
}

\newtcolorbox{Theorem}[1]{
  colback=blue!7!white,
  colframe=blue!7!white,
  coltitle=blue!100!white,
  fonttitle=\bfseries,
  top=0mm,
  arc=0mm,
  title=\vspace{2mm}Theorem ({#1}),
}

\newtcolorbox{Example}[1]{
  colback=green!7!white,
  colframe=green!7!white,
  coltitle=green!50!black,
  fonttitle=\bfseries,
  top=0mm,
  arc=0mm,
  title=\vspace{2mm}Example ({#1}),
}

\newtcolorbox{Exercise}[1]{
  colback=black!7!white,
  colframe=black!7!white,
  coltitle=black!50!black,
  fonttitle=\bfseries,
  top=0mm,
  arc=0mm,
  title=\vspace{2mm}Exercise ({#1}),
}

\newenvironment{Solution}
{\vspace{1mm}\par\noindent\textit{Solution.}~}
{\vspace{1mm}}

\newenvironment{Proof}
{\vspace{1mm}\par\noindent\textit{Proof.}~}
{\vspace{1mm}\hfill$\Box$}

\newenvironment{Exposition}
{\vspace{1mm}\par}
{\vspace{1mm}}

\newenvironment{Figure}[1]
{
  \begin{center}
    \includegraphics[\textwidth]{#1}
\end{center}}
{\vspace{1mm}}

Non-negativity and iff condition of KL Divergence

1 minute read

Published:

2 interesting properties of the KL Divergence:

1. Non-negativity: \(KL(p \parallel q) \ge 0\)

This property isn’t immediately obvious, as the \(\log\) terms can be negative if \(q(x) \leq p(x)\). Intuitively, each \(\log\) term is weighted by the probability, so the overall quantity will always be positive. But a rigorous proof involves Jensen’s Inequality (and is called the Gibbs’ Inequality).

Note: from now on, we use \(\ln\) which is fine because all logarithms are related by a constant factor.

Proof:

The definition of KL divergence is \(KL(p \parallel q) = \sum_x p(x) \ln \frac{p(x)}{q(x)}\). We can rewrite this as:

[KL(p \parallel q) = - \sum_x p(x) \ln \frac{q(x)}{p(x)}]

To prove that \(-\sum_x p(x) \ln \frac{q(x)}{p(x)} \ge 0\), it suffices to show that \(\sum_x p(x) \ln \frac{q(x)}{p(x)} \le 0\).

Since \(\ln\) is concave, we can apply Jensen’s Inequality. For a concave function \(f\), Jensen’s Inequality states that \(E[f(X)] \le f(E[X])\). In our case, \(f\) is \(\ln\) and the expectation is over the distribution \(p(x)\).

[\begin{align} \sum_x p(x) \ln \frac{q(x)}{p(x)} &\le \ln \left( \sum_x p(x) \frac{q(x)}{p(x)} \right)
&= \ln \left( \sum_x q(x) \right) \end{align}]

Since \(q(x)\) is a probability distribution, \(\sum_x q(x) = 1\). Therefore:

[\ln \left( \sum_x q(x) \right) = \ln(1) = 0]

This shows that \(\sum_x p(x) \ln \frac{q(x)}{p(x)} \le 0\), which completes the proof that \(KL(p \parallel q) = - \sum_x p(x) \ln \frac{q(x)}{p(x)} \ge 0\).

2. \(KL(p \parallel q) = 0 \iff p=q\)

Proof:

When does equality hold? It turns out there’s an equality condition for Jensen’s Inequality. For a strictly concave function like \(\ln\), equality holds if and only if the random variable is a constant. In our case, this means \(\frac{q(x)}{p(x)}\) must be a constant, \(c\), for all \(x\).

[\frac{q(x)}{p(x)} = c]

So, \(q(x) = c \cdot p(x)\). We can solve for \(c\) by summing over all \(x\):

[\sum_x q(x) = c \sum_x p(x)]

Since both \(p\) and \(q\) are probability distributions, their sums are equal to 1.

[1 = c \cdot 1 \implies c = 1]

Therefore, for the KL divergence to be zero, it must be that \(q(x) = 1 \cdot p(x)\) for all \(x\), which means the distributions must be identical.

Minimizing KL Divergence is Minimizing Cross Entropy is Maximizing Likelihood

4 minute read

Published:

In the derivation of statistical models, we often encounter 3 statistical quantities: the Kullback-Leibler (KL) Divergence, Cross-Entropy, and Maximum Likelihood Estimation (MLE). While they might seem different at first glance, they are deeply intertwined. This post will walk through the proof that demonstrates their equivalence.

From KL Divergence to Cross-Entropy

Let’s start with the definition of KL Divergence. It measures how one probability distribution, \(Q\), diverges from a second, expected probability distribution, \(P\). The formula is as follows:

[D_{KL}(P \parallel Q) = \sum_{x} P(x) \log \frac{P(x)}{Q(x)}]

We can expand this expression:

[\begin{align} D_{KL}(P \parallel Q) &= E_{x \sim P(x)}[\log P(x) - \log Q(x)]
&= E_{x \sim P(x)}[\log P(x)] - E_{x \sim P(x)}[\log Q(x)]
&= E_{x \sim P(x)}[-\log Q(x)] - E_{x \sim P(x)}[-\log P(x)]
&= E_{x \sim P(x)}\left[\log \frac{1}{Q(x)}\right] - E_{x \sim P(x)}\left[\log \frac{1}{P(x)}\right]
&= H(P, Q) - H(P) \end{align}]

Here, \(H(P)\) is the entropy of the distribution \(P\), and \(H(P, Q)\) is the cross-entropy between \(P\) and \(Q\).

This gives us our first key relationship:

[D_{KL}(P \parallel Q) = H(P, Q) - H(P)]

Minimizing KL Divergence is Equivalent to Minimizing Cross-Entropy

In many statistical modeling scenarios, we want to train a model distribution, \(P(x \vert \theta)\), to be as close as possible to the true underlying data distribution, \(P_\text{data}\). In other words we want to minimize \(D_{KL}(P_\text{data} \parallel P(x \vert \theta))\). In practice, however, we don’t know the true underlying data distribution: for one, it may not follow our model distribution, and we can only estimate it with a finite sample. Thus, we write \(\hat{P}_\text{data}\) instead of \(P_\text{data}\) to denote that it is an estimate of the data distribution. We can use something like the Empirical Distribution as the estimator.

If we consider the expression for KL Divergence, we can see that the entropy of the true data distribution, \(H(P)\), is a constant that doesn’t depend on. We can’t change the entropy of the data itself. Therefore, minimizing the KL Divergence between our model and the data is equivalent to minimizing the cross-entropy between them.

[\arg\min_{\theta} D_{KL}(P \parallel Q) = \arg\min_{\theta} [H(P, Q) - H(P)] = \arg\min_{\theta} H(P, Q)]

Minimizing Cross-Entropy is Equivalent to Maximizing Likelihood

Now, let’s connect this to Maximum Likelihood Estimation. In a typical machine learning problem, we have a dataset \(D = \{x_i\}_{i=1}^n\) sampled from the true data distribution, which we’ll call \(P_{\text{data}}\). Our model is a parameterized distribution, \(P(x \vert \theta)\). We want to find the parameters \(\theta\) that make our model best fit the data.

We can frame this as minimizing the KL Divergence between the empirical distribution of our data, \(\hat{P}_{\text{data}}\), and our model, \(P(x \vert \theta)\).

[\arg\min_{\theta} D_{KL}(\hat{P}_{\text{data}} \parallel P(x \vert \theta))]

From our previous derivation, this is equivalent to minimizing the cross-entropy:

[\arg\min_{\theta} H(\hat{P}_{\text{data}}, P(x \vert \theta))]

Let’s write out the definition of cross-entropy:

[H(\hat{P}{\text{data}}, P(x \vert \theta)) = - \sum{i=1}^{n} \hat{P}_{\text{data}}(x_i) \log P(x_i \vert \theta)]

For an empirical distribution from a dataset, \(\hat{P}_{\text{data}}(x_i)\) is simply \(\frac{1}{n}\) for each observed data point \(x_i\). So, we can rewrite the expression as:

[\begin{align} \arg\min_{\theta} H(\hat{P}_{\text{data}}, P(x \vert \theta)) &= \arg\min_{\theta} - \frac{1}{n} \sum_{i=1}^{n} \log P(x_i \vert \theta)
&= \arg\max_{\theta} \frac{1}{n} \sum_{i=1}^{n} \log P(x_i \vert \theta) \end{align}]

Using the properties of logarithms, we can bring the sum inside the log as a product:

[\arg\max_{\theta} \frac{1}{n} \log \prod_{i=1}^{n} P(x_i \vert \theta)]

Since the logarithm is a monotonic function, maximizing the log of a function is the same as maximizing the function itself. The \(\frac{1}{n}\) term is a constant, so we can also drop it. This leaves us with:

[\arg\max_{\theta} \prod_{i=1}^{n} P(x_i \vert \theta)]

This final expression is the Maximum Likelihood Estimate of the parameters \(\theta\). The MLE finds the parameters that maximize the probability of observing our given data.

Therefore, we have shown that minimizing the KL Divergence between our model and the data distribution is equivalent to minimizing the cross-entropy, which in turn is equivalent to maximizing the likelihood of the data.

portfolio

publications

Cognitive Behavioral Modeling with Activation Steering

Published in NeurIPS CogInterp Workshop, 2025

CBMAS is a diagnostic framework designed for the systematic analysis of cognitive behaviors in LLMs, employing continuous activation interventions to assess how the magnitude of manipulation and the depth of network layers influence model biases.

Download Paper

talks

teaching

Teaching experience 1

Undergraduate course, University 1, Department, 2014

This is a description of a teaching experience. You can use markdown like any other post.

Teaching experience 2

Workshop, University 1, Department, 2015

This is a description of a teaching experience. You can use markdown like any other post.