Tutorial on Probabilistic Topic Modeling: Additive Regularization for Stochastic Matrix Factorization

Abstract

Probabilistic topic modeling of text collections is a powerful tool for statistical text analysis. In this tutorial we introduce a novel non-Bayesian approach, called Additive Regularization of Topic Models. ARTM is free of redundant probabilistic assumptions and provides a simple inference for many combined and multi-objective topic models.

Appendices

A The Karush–Kuhn–Tucker (KKT) Conditions

Consider the following nonlinear optimization problem:

$$ f(x) \rightarrow \max _x; \qquad g_i(x) \ge 0, i=1,\dots , m; \qquad h_j(x) = 0, j=1,\dots , k. $$

Suppose that the objective function ${f:\mathbb {R}^n \rightarrow \mathbb {R}}$ and the constraint functions ${g_i:\mathbb {R}^n \rightarrow \mathbb {R}}$ and ${h_j:\mathbb {R}^n \rightarrow \mathbb {R}}$ are continuously differentiable at a point $x^{*}$. If $x^{*}$ is a local maximum that satisfies some regularity conditions (which are always true if $g_i$ and $h_j$ are linear functions), then there exist constants $\mu _i$, ${i = 1,\ldots ,m}$ and $\lambda _j$, ${j = 1,\ldots ,k}$, called KKT multipliers, such that

$$\begin{aligned}&\frac{\partial }{\partial x} \biggl (f(x) + \sum _{i=1}^m \mu _i g_i(x) + \sum _{j=1}^k \lambda _j g_j(x) \biggr )=0;&\text { (stationarity)}\\&g_i(x) \ge 0; h_j(x) = 0;&\text { (primal feasibility)}\\&\mu _i \ge 0;&\text { (dual feasibility)}\\&\mu _ig_i(x) = 0.&\text { (complementary slackness)} \end{aligned}$$

B The Kullback–Leibler Divergence

The Kullback–Leibler divergence or relative entropy is a non-symmetric measure of the difference between probability distributions ${P = (p_i)_{i=1}^n}$ and ${Q = (q_i)_{i=1}^n}$:

$$ \mathop {\text {KL}}\nolimits (P \Vert Q) \equiv \mathop {\text {KL}}\nolimits _i(p_i \Vert q_i) = \sum _{i=1}^n p_i \ln \frac{p_i}{q_i}. $$

From the informational point of view, $\mathop {\text {KL}}\nolimits (P \Vert Q)$ is a measure of the information lost when $Q$ is used to approximate $P$. KL-divergence measures the expected number of extra bits required to code samples from $P$ when using a code based on $Q$, rather than using a code based on $P$. Typically $P$ represents the empirical distribution of data, $Q$ represents a model or approximation of $P$.

The KL-divergence is always non-negative.

${\mathop {\text {KL}}\nolimits (P \Vert Q) = 0}$ if and only if ${P=Q}$.

The KL-divergence minimization is equivalent to the likelihood maximization of a model distribution $Q(\alpha )$ over parameter vector $\alpha$:

$$ \mathop {\text {KL}}\nolimits (P \Vert Q(\alpha )) = \sum _{i=1}^n p_i \ln \frac{p_i}{q_i(\alpha )} \rightarrow \min _\alpha \quad \Longleftrightarrow \quad \sum _{i=1}^n p_i \ln q_i(\alpha ) \rightarrow \max _\alpha . $$