F-divergence

---

title: "F-divergence" type: doc version: 1 created: 2026-02-28 author: "Wikipedia contributors" status: active scope: public tags: ["f-divergences"] description: "Function that measures dissimilarity between two probability distributions" topic_path: "general/f-divergences" source: "https://en.wikipedia.org/wiki/F-divergence" license: "CC BY-SA 4.0" wikipedia_page_id: 0 wikipedia_revision_id: 0

::summary Function that measures dissimilarity between two probability distributions ::

In probability theory, an f**-divergence** is a certain type of function D_f(P| Q) that measures the difference between two probability distributions P and Q. Many common divergences, such as KL-divergence, Hellinger distance, and total variation distance, are special cases of f-divergence.

History

These divergences were introduced by Alfréd Rényi in the same paper where he introduced the well-known Rényi entropy. He proved that these divergences decrease in Markov processes. f-divergences were studied further independently by , and and are sometimes known as Csiszár f-divergences, Csiszár–Morimoto divergences, or Ali–Silvey distances.

Definition

Non-singular case

Let P and Q be two probability distributions over a space \Omega, such that P\ll Q, that is, P is absolutely continuous with respect to Q (meaning Q0 wherever P0). Then, for a convex function f: [0, +\infty)\to(-\infty, +\infty] such that f(x) is finite for all x 0, f(1)=0, and f(0)=\lim_{t\to 0^+} f(t) (which could be infinite), the f-divergence of P from Q is defined as

: D_f(P\parallel Q) \equiv \int_{\Omega} f\left(\frac{dP}{dQ}\right),dQ.

We call f the generator of D_f.

In concrete applications, there is usually a reference distribution \mu on \Omega (for example, when \Omega = \R^n, the reference distribution is the Lebesgue measure), such that P, Q \ll \mu, then we can use Radon–Nikodym theorem to take their probability densities p and q, giving

: D_f(P\parallel Q) = \int_{\Omega} f\left(\frac{p(x)}{q(x)}\right)q(x),d\mu(x).

When there is no such reference distribution ready at hand, we can simply define \mu = P+Q, and proceed as above. This is a useful technique in more abstract proofs.

Extension to [[singular measures]]

The above definition can be extended to cases where P\ll Q is no longer satisfied (Definition 7.1 of ).

Since f is convex, and f(1) = 0 , the function \frac{f(x)}{x-1} must be nondecreasing, so there exists f'(\infty) := \lim_{x\to\infty}f(x)/x, taking value in (-\infty, +\infty].

Since for any p(x)0, we have \lim_{q(x)\to 0} q(x)f \left(\frac{p(x)}{q(x)}\right) = p(x)f'(\infty) , we can extend f-divergence to the P\not\ll Q.

Properties

Basic relations between f-divergences

• Linearity: D_{\sum_i a_i f_i} = \sum_i a_i D_{f_i} given a finite sequence of nonnegative real numbers a_i and generators f_i.

• D_f = D_g iff f(x) = g(x) + c(x-1) for some c\in \R.

If f(x) = g(x) + c(x-1), then D_f = D_g by definition.

Conversely, if D_f - D_g = 0, then let h = f-g. For any two probability measures P, Q on the set {0, 1}, since D_f(P| Q) - D_g(P|Q) = 0, we get h(P_1/Q_1) = -\frac{Q_0}{Q_1}h(P_0/Q_0)

Since each probability measure P, Q has one degree of freedom, we can solve \frac{P_0}{Q_0} = a, \frac{P_1}{Q_1} = x for every choice of 0 .

Linear algebra yields Q_0 = \frac{x-1}{x-a}, Q_1 = \frac{1-a}{x-a}, which is a valid probability measure. Then we obtain h(x) = \frac{h(a)}{a-1}(x-1), h(a) = \frac{h(x)}{x-1}(a-1).

Thus h(x)=\begin{cases} c_1(x-1)\quad\text{if } x1,\ c_0(x-1)\quad\text{if } 0 \end{cases} for some constants c_0, c_1. Plugging the formula into h(x) = \frac{h(a)}{a-1}(x-1) yields c_0 = c_1.

Basic properties of f-divergences

|1= Non-negativity: the ƒ-divergence is always positive; it is zero if the measures P and Q coincide. This follows immediately from Jensen’s inequality: : D_f(P!\parallel!Q) = \int !f\bigg(\frac{dP}{dQ}\bigg)dQ \geq f\bigg( \int\frac{dP}{dQ}dQ\bigg) = f(1) = 0.

|2= Data-processing inequality: if κ is an arbitrary transition probability that transforms measures P and Q into P**κ and Q**κ correspondingly, then : D_f(P!\parallel!Q) \geq D_f(P_\kappa!\parallel!Q_\kappa). The equality here holds if and only if the transition is induced from a sufficient statistic with respect to {P, Q}.

|3= Joint convexity: for any 0 ≤ λ ≤ 1, : D_f\Big(\lambda P_1 + (1-\lambda)P_2 \parallel \lambda Q_1 + (1-\lambda)Q_2\Big) \leq \lambda D_f(P_1!\parallel!Q_1) + (1-\lambda)D_f(P_2!\parallel!Q_2). This follows from the convexity of the mapping (p,q) \mapsto q f(p/q) on \mathbb{R}_+^2.

|4= Reversal by convex inversion: for any function f, its convex inversion is defined as g(t):= t f(1/t). When f satisfies the defining features of a f-divergence generator (f(x) is finite for all x 0, f(1)=0, and f(0)=\lim_{t\to 0^+} f(t)), then g satisfies the same features, and thus defines a f-divergence D_g. This is the "reverse" of D_f, in the sense that D_g(P|Q) = D_f(Q|P) for all P, Q that are absolutely continuous with respect to each other.

In this way, every f-divergence D_f can be turned symmetric by D_{\frac 1 2 (f + g)}. For example, performing this symmetrization turns KL-divergence into Jeffreys divergence.

In particular, the monotonicity implies that if a Markov process has a positive equilibrium probability distribution P^* then D_f(P(t)\parallel P^) is a monotonic (non-increasing) function of time, where the probability distribution P(t) is a solution of the Kolmogorov forward equations (or Master equation), used to describe the time evolution of the probability distribution in the Markov process. This means that all f-divergences D_f(P(t)\parallel P^) are the Lyapunov functions of the Kolmogorov forward equations. The converse statement is also true: If H(P) is a Lyapunov function for all Markov chains with positive equilibrium P^* and is of the trace-form (H(P)=\sum_{i}f(P_{i},P_{i}^{})) then H(P)= D_f(P(t)\parallel P^), for some convex function f. For example, Bregman divergences in general do not have such property and can increase in Markov processes.

Analytic properties

The f-divergences can be expressed using Taylor series and rewritten using a weighted sum of chi-type distances ().

Basic variational representation

Let f^* be the convex conjugate of f. Let \mathrm{effdom}(f^) be the effective domain of f^, that is, \mathrm{effdom}(f^) = {y : f^(y) . Then we have two variational representations of D_f, which we describe below.

Under the above setup, | name = Theorem | math_statement = D_f(P; Q) = \sup_{g: \Omega\to \mathrm{effdom}(f^)} E_P[g] - E_Q[f^ \circ g].

This is Theorem 7.24 in.

Example applications

Using this theorem on total variation distance, with generator f(x)= \frac 1 2 |x-1|, its convex conjugate is f^(x^) = \begin{cases} x^* \text{ on } [-1/2, 1/2],\ +\infty \text{ else.} \end{cases}, and we obtain TV(P| Q) = \sup_{|g|\leq 1/2} E_P[g(X)] - E_Q[g(X)]. For chi-squared divergence, defined by f(x) = (x-1)^2, f^*(y) = y^2/4 + y, we obtain \chi^2(P; Q) = \sup_g E_P[g(X)] - E_Q[g(X)^2/4 + g(X)]. Since the variation term is not affine-invariant in g, even though the domain over which g varies is affine-invariant, we can use up the affine-invariance to obtain a leaner expression.

Replacing g by a g + b and taking the maximum over a, b \in \R, we obtain \chi^2(P; Q) = \sup_g \frac{(E_P[g(X)]-E_Q[g(X)])^2}{Var_Q[g(X)]}, which is just a few steps away from the Hammersley–Chapman–Robbins bound and the Cramér–Rao bound (Theorem 29.1 and its corollary in ).

For \alpha-divergence with \alpha \in (-\infty, 0)\cup(0, 1), we have f_\alpha(x) = \frac{x^\alpha - \alpha x - (1-\alpha)}{\alpha(\alpha-1)}, with range x\in [0, \infty). Its convex conjugate is f_\alpha^*(y)=\frac{1}{\alpha}(x(y)^\alpha - 1) with range y\in(-\infty, (1-\alpha)^{-1}), where x(y) = ((\alpha-1)y + 1)^{\frac{1}{\alpha-1}}.

Applying this theorem yields, after substitution with h = ((\alpha-1)g+1)^{\frac{1}{\alpha-1}}, D_\alpha(P| Q) = \frac{1}{\alpha(1-\alpha)} - \inf_{h: \Omega\to (0,\infty)}\left( E_Q\left[\frac{h^\alpha}{\alpha}\right]

• E_P\left[\frac{h^{\alpha-1}}{1-\alpha}\right] \right), or, releasing the constraint on h, D_\alpha(P| Q) = \frac{1}{\alpha(1-\alpha)} - \inf_{h: \Omega\to \R}\left( E_Q\left[\frac{|h|^\alpha}{\alpha}\right]
• E_P\left[\frac{|h|^{\alpha-1}}{1-\alpha}\right] \right). Setting \alpha=-1 yields the variational representation of \chi^2-divergence obtained above.

The domain over which h varies is not affine-invariant in general, unlike the \chi^2-divergence case. The \chi^2-divergence is special, since in that case, we can remove the |\cdot | from |h|.

For general \alpha \in (-\infty, 0)\cup(0, 1), the domain over which h varies is merely scale invariant. Similar to above, we can replace h by a h, and take minimum over a0 to obtain D_\alpha(P| Q) = \sup_{h 0} \left[\frac{1}{\alpha(1-\alpha)} \left( 1-\frac{E_P[h^{\alpha-1}]^\alpha}{E_Q[h^\alpha]^{\alpha-1}} \right) \right]. Setting \alpha=\frac 1 2, and performing another substitution by g=\sqrt h, yields two variational representations of the squared Hellinger distance: H^2(P|Q) = \frac 1 2 D_{1/2}(P| Q) = 2 - \inf_{h0}\left( E_Q\left[h(X)\right]

• E_P\left[h(X)^{-1}\right] \right), H^2(P|Q) = 2 \sup_{h 0} \left(1-\sqrt{E_P[h^{-1}]E_Q[h]}\right). Applying this theorem to the KL-divergence, defined by f(x) = x\ln x, f^*(y) = e^{y-1}, yields D_{KL}(P; Q) =\sup_g E_P[g(X)] - e^{-1}E_Q[e^{g(X)}]. This is strictly less efficient than the Donsker–Varadhan representation D_{KL}(P; Q) = \sup_g E_P[g(X)]- \ln E_Q[e^{g(X)}]. This defect is fixed by the next theorem.

Improved variational representation

Assume the setup in the beginning of this section ("Variational representations"). | name = Theorem | math_statement = If f(x) = +\infty on x (redefine f if necessary), then

D_{f}(P | Q)=f^{\prime}(\infty) P\left[S^{c}\right]+\sup {g} \mathbb{E}{P}\left[g 1_{S}\right]-\Psi_{Q, P}^{*}(g) ,

where \Psi_{Q, P}^{}(g) := \inf {a \in \mathbb{R}} \mathbb{E}{Q}\left[f^{}(g(X)-a)\right]+a P[S] and S:={q 0} , where q is the probability density function of Q with respect to some underlying measure.

In the special case of f^{\prime}(\infty)=+\infty, we have

D_{f}(P | Q)=\sup {g} \mathbb{E}{P}[g]-\Psi_{Q}^{}(g), \quad \Psi_{Q}^{}(g) := \inf {a \in \mathbb{R}} \mathbb{E}{Q}\left[f^{*}(g(X)-a)\right]+a .

This is Theorem 7.25 in.

Example applications

Applying this theorem to KL-divergence yields the Donsker–Varadhan representation.

Attempting to apply this theorem to the general \alpha-divergence with \alpha \in (-\infty, 0)\cup(0, 1) does not yield a closed-form solution.

Common examples of ''f''-divergences

The following table lists many of the common divergences between probability distributions and the possible generating functions to which they correspond. Notably, except for total variation distance, all others are special cases of \alpha-divergence, or linear sums of \alpha-divergences.

For each f-divergence D_f, its generating function is not uniquely defined, but only up to c\cdot(t-1), where c is any real constant. That is, for any f that generates an f-divergence, we have D_{f(t)} = D_{f(t) + c\cdot(t-1)}. This freedom is not only convenient, but actually necessary.

::data[format=table]

Divergence	Corresponding f(t)	Discrete Form
\chi^{\alpha}-divergence, \alpha \ge 1 ,	\frac12	t - 1
Total variation distance ( \alpha = 1 ,)	\frac12	t - 1
α-divergence	\begin{cases}
KL-divergence (\alpha=1)	t \ln t	\sum_i p_i \ln \frac{p_i}{q_i}
reverse KL-divergence (\alpha=0)	- \ln t	\sum_i q_i \ln \frac{q_i}{p_i}
Jensen–Shannon divergence	\frac{1}{2} \left(t \ln t -(t + 1)\ln \left(\frac{t + 1}{2}\right)\right)	\frac{1}{2} \sum_i \left( p_i \ln \frac{p_i}{(p_i + q_i)/2} + q_i \ln \frac{q_i}{(p_i + q_i)/2} \right)
Jeffreys divergence (KL + reverse KL)	(t - 1)\ln(t)	\sum_i (p_i - q_i) \ln \frac{p_i}{q_i}
squared Hellinger distance (\alpha=\frac 1 2)	\frac{1}{2}(\sqrt{t} - 1)^2,,1-\sqrt{t}	\frac{1}{2}\sum_i (\sqrt{p_i} - \sqrt{q_i})^2; ; 1 - \sum_i \sqrt{p_i q_i}
Neyman \chi^2-divergence	(t - 1)^2	\sum_i \frac{(p_i - q_i)^2}{q_i}
Pearson \chi^2-divergence	\frac{(t-1)^2}{t}	\sum_i \frac{(p_i - q_i)^2}{p_i}
::

::figure[src="https://upload.wikimedia.org/wikipedia/commons/9/91/Alpha-divergence.svg" caption="Comparison between the generators of alpha-divergences, as alpha varies from -1 to 2."] ::

Let f_\alpha be the generator of \alpha-divergence, then f_\alpha and f_{1-\alpha} are convex inversions of each other, so D_{\alpha}(P| Q) = D_{1-\alpha}(Q| P) . In particular, this shows that the squared Hellinger distance and Jensen-Shannon divergence are symmetric.

In the literature, the \alpha-divergences are sometimes parametrized as

\begin{cases} \frac{4}{1-\alpha^2}\big(1 - t^{(1+\alpha)/2}\big), & \text{if}\ \alpha\neq\pm1, \ t \ln t, & \text{if}\ \alpha=1, \

• \ln t, & \text{if}\ \alpha=-1 \end{cases}

which is equivalent to the parametrization in this page by substituting \alpha \leftarrow \frac{\alpha+1}{2}.

Relations to other statistical divergences

Here, we compare f-divergences with other statistical divergences.

Rényi divergence

The Rényi divergences is a family of divergences defined by

R_{\alpha} (P | Q) = \frac{1}{\alpha-1}\log\Bigg( E_Q\left[\left(\frac{dP}{dQ}\right)^\alpha\right] \Bigg) ,

when \alpha \in (0, 1)\cup (1, +\infty). It is extended to the cases of \alpha =0, 1, +\infty by taking the limit.

Simple algebra shows that R_\alpha(P| Q) = \frac{1}{\alpha - 1}\ln (1+\alpha(\alpha-1)D_\alpha(P|Q)), where D_\alpha is the \alpha-divergence defined above.

Bregman divergence

The only f-divergence that is also a Bregman divergence is the KL divergence.

Integral probability metrics

The only f-divergence that is also an integral probability metric is the total variation.

Financial interpretation

A pair of probability distributions can be viewed as a game of chance in which one of the distributions defines the official odds and the other contains the actual probabilities. Knowledge of the actual probabilities allows a player to profit from the game. For a large class of rational players the expected profit rate has the same general form as the ƒ-divergence.

References

• {{cite journal | first = I. | last = Csiszár | year = 1963 | title = Eine informationstheoretische Ungleichung und ihre Anwendung auf den Beweis der Ergodizitat von Markoffschen Ketten | journal = Magyar. Tud. Akad. Mat. Kutato Int. Kozl | volume = 8 | pages = 85–108
• {{cite journal | doi = 10.1143/JPSJ.18.328 | first = T. | last = Morimoto | year = 1963 | title = Markov processes and the H-theorem | journal = J. Phys. Soc. Jpn. | volume = 18 | issue = 3 | pages = 328–331 | bibcode = 1963JPSJ...18..328M
• {{cite journal | first1 = S. M. | last1 = Ali | first2 = S. D. | last2 = Silvey | year = 1966 | title = A general class of coefficients of divergence of one distribution from another | journal = Journal of the Royal Statistical Society, Series B | volume = 28 | issue = 1 | pages = 131–142 | doi = 10.1111/j.2517-6161.1966.tb00626.x | jstor = 2984279 | mr = 0196777
• {{cite journal | first = I. | last = Csiszár | year = 1967 | title = Information-type measures of difference of probability distributions and indirect observation | journal = Studia Scientiarum Mathematicarum Hungarica | volume = 2 | pages = 229–318 | ref = CITEREFCsisz.C3.A1r1967
• {{cite journal | first1 = I. | last1 = Csiszár | author-link1 = Imre Csiszár | first2 = P. | last2 = Shields | year = 2004 | title = Information Theory and Statistics: A Tutorial | journal = Foundations and Trends in Communications and Information Theory | volume = 1 | issue = 4 | pages = 417–528 | doi = 10.1561/0100000004 | url = http://www.renyi.hu/~csiszar/Publications/Information_Theory_and_Statistics:_A_Tutorial.pdf | accessdate = 2009-04-08
• {{cite journal | first1 = F. | last1 = Liese | first2 = I. | last2 = Vajda | year = 2006 | title = On divergences and informations in statistics and information theory | journal = IEEE Transactions on Information Theory | volume = 52 | issue = 10 | pages = 4394–4412 | doi = 10.1109/TIT.2006.881731 | bibcode = 2006ITIT...52.4394L | s2cid = 2720215
• {{cite journal | first1 = F. | last1 = Nielsen | first2 = R. | last2 = Nock | year = 2013 | title = On the Chi square and higher-order Chi distances for approximating f-divergences | arxiv = 1309.3029 | doi=10.1109/LSP.2013.2288355 | volume=21 | journal=IEEE Signal Processing Letters | issue = 1 | pages=10–13 | bibcode=2014ISPL...21...10N| s2cid = 4152365
• {{cite arXiv | first1 = J-F. | last1 = Coeurjolly | first2 = R. | last2 = Drouilhet | year = 2006 | title = Normalized information-based divergences | eprint = math/0604246 | ref = arXiv:math/0604246

References

1. Rényi, Alfréd. (1961). "On measures of entropy and information".
2. (2022). "Information Theory: From Coding to Learning (draft of October 20, 2022)". Cambridge University Press.
3. (15 October 2003). "Monotonically equivalent entropies and solution of additivity equation". Physica A.
4. Amari, Shun'ichi. (2009). "Divergence, Optimization, Geometry". Springer.
5. (29 April 2014). "General H-theorem and Entropies that Violate the Second Law". Entropy.
6. (December 2014). "Information Measures: the Curious Case of the Binary Alphabet". IEEE Transactions on Information Theory.
7. (2009). "On integral probability metrics, φ-divergences and binary classification".
8. (2020). "Economics of Disagreement—Financial Intuition for the Rényi Divergence". Entropy.

::callout[type=info title="Wikipedia Source"] This article was imported from Wikipedia and is available under the Creative Commons Attribution-ShareAlike 4.0 License. Content has been adapted to SurfDoc format. Original contributors can be found on the article history page. ::