---

title: "Stein's lemma" type: doc version: 1 created: 2026-02-28 author: "Wikipedia contributors" status: active scope: public tags: ["theorems-in-statistics", "theorems-in-probability-theory"] description: "Theorem of probability theory" topic_path: "science/mathematics" source: "https://en.wikipedia.org/wiki/Stein's_lemma" license: "CC BY-SA 4.0" wikipedia_page_id: 0 wikipedia_revision_id: 0

::summary Theorem of probability theory ::

Stein's lemma, named in honor of Charles Stein, is a theorem of probability theory that is of interest primarily because of its applications to statistical inference — in particular, to James–Stein estimation and empirical Bayes methods — and its applications to portfolio choice theory. The theorem gives a formula for the covariance of one random variable with the value of a function of another, when the two random variables are jointly normally distributed.

Note that the name "Stein's lemma" is also commonly used to refer to a different result in the area of statistical hypothesis testing, which connects the error exponents in hypothesis testing with the Kullback–Leibler divergence. This result is also known as the Chernoff–Stein lemma and is not related to the lemma discussed in this article.

Statement

Suppose X is a normally distributed random variable with expectation μ and variance σ2. Further suppose g is a differentiable function for which the two expectations \operatorname{E}(g(X) (X - \mu)) and \operatorname{E}(g'(X)) both exist. (The existence of the expectation of any random variable is equivalent to the finiteness of the expectation of its absolute value.) Then

:\operatorname{E}\bigl(g(X)(X-\mu)\bigr)=\sigma^2 \operatorname{E}\bigl(g'(X)\bigr).

Multidimensional

In general, suppose X and Y are jointly normally distributed. Then

:\operatorname{Cov}(g(X),Y)= \operatorname{Cov}(X,Y)\operatorname{E}(g'(X)).

For a general multivariate Gaussian random vector (X_1, ..., X_n) \sim \mathcal{N}(\mu, \Sigma) it follows that :\operatorname{E}\bigl(g(X)(X-\mu)\bigr)=\Sigma\cdot E\bigl(\nabla g(X)\bigr). Similarly, when \mu = 0, \operatorname{E}[\partial_i g(X) ] = \operatorname{E}[g(X) (\Sigma^{-1}X)_i], \quad \operatorname{E}[\partial_i\partial_j g(X) ] = \operatorname{E}[g(X) ((\Sigma^{-1}X)_i(\Sigma^{-1}X)j - \Sigma^{-1}{ij})]

Gradient descent

Stein's lemma can be used to stochastically estimate gradient:\nabla \operatorname{E}_{\epsilon \sim \mathcal{N}(0, I)}\bigl(g(x + \Sigma^{1/2}\epsilon)\bigr)

\Sigma^{-1/2} \operatorname{E}{\epsilon \sim \mathcal{N}(0, I)}\bigl(g(x + \Sigma^{1/2}\epsilon)\epsilon\bigr) \approx \Sigma^{-1/2} \frac{1}{N} \sum{i = 1}^N g(x + \Sigma^{1/2}\epsilon_i )\epsilon_iwhere \epsilon_1, \dots, \epsilon_N are IID samples from the standard normal distribution \mathcal N(0, I). This form has applications in Stein variational gradient descent and Stein variational policy gradient.

Proof

The univariate probability density function for the univariate normal distribution with expectation 0 and variance 1 is

:\varphi(x)={1 \over \sqrt{2\pi}}e^{-x^2/2}

Since \int x \exp(-x^2/2),dx = -\exp(-x^2/2) we get from integration by parts:

:\operatorname{E}[g(X)X] = \frac{1}{\sqrt{2\pi}}\int g(x) x \exp(-x^2/2),dx = \frac{1}{\sqrt{2\pi}}\int g'(x) \exp(-x^2/2),dx = \operatorname{E}[g'(X)].

The case of general variance \sigma^2 follows by substitution.

Generalizations

Isserlis' theorem is equivalently stated as\operatorname{E}(X_1 f(X_1,\ldots,X_n))=\sum_{i=1}^{n} \operatorname{Cov}(X_1,X_i)\operatorname{E}(\partial_{X_i}f(X_1,\ldots,X_n)).where (X_1,\dots X_{n}) is a zero-mean multivariate normal random vector.

Suppose X is in an exponential family, that is, X has the density

:f_\eta(x)=\exp(\eta'T(x) - \Psi(\eta))h(x).

Suppose this density has support (a,b) where a,b could be -\infty ,\infty and as x\rightarrow a\text{ or }b, \exp (\eta'T(x))h(x) g(x) \rightarrow 0 where g is any differentiable function such that E|g'(X)| or \exp (\eta'T(x))h(x) \rightarrow 0 if a,b finite. Then

:E\left[\left(\frac{h'(X)}{h(X)} + \sum \eta_i T_i'(X)\right)\cdot g(X)\right] = -E[g'(X)].

The derivation is same as the special case, namely, integration by parts.

If we only know X has support \mathbb{R} , then it could be the case that E|g(X)| but \lim_{x\rightarrow \infty} f_\eta(x) g(x) \not= 0. To see this, simply put g(x)=1 and f_\eta(x) with infinitely spikes towards infinity but still integrable. One such example could be adapted from f(x) = \begin{cases} 1 & x \in [n, n + 2^{-n}) \ 0 & \text{otherwise} \end{cases} so that f is smooth.

Extensions to elliptically-contoured distributions also exist. |last1=Cellier |first1=Dominique |last2=Fourdrinier |first2=Dominique |last3=Robert |first3=Christian |year=1989 |title=Robust shrinkage estimators of the location parameter for elliptically symmetric distributions |journal=Journal of Multivariate Analysis |volume=29 |issue=1 |pages=39–52 |doi=10.1016/0047-259X(89)90075-4 |doi-access= |last1=Hamada |first1=Mahmoud |last2=Valdez |first2=Emiliano A. |year=2008 |title=CAPM and option pricing with elliptically contoured distributions |journal=The Journal of Risk & Insurance |volume=75 |issue=2 |pages=387–409 |doi=10.1111/j.1539-6975.2008.00265.x |citeseerx=10.1.1.573.4715 |last1=Landsman |first1=Zinoviy |last2=Nešlehová |first2=Johanna|author2-link= Johanna G. Nešlehová |year=2008 |title=Stein's Lemma for elliptical random vectors |journal=Journal of Multivariate Analysis |volume=99 |issue=5 |pages=912––927 |doi=10.1016/j.jmva.2007.05.006 |doi-access=free}}

References

References

1. Ingersoll, J., ''Theory of Financial Decision Making'', Rowman and Littlefield, 1987: 13–14.
2. (2011). "Information Theory: Coding Theorems for Discrete Memoryless Systems". Cambridge University Press.
3. Thomas M. Cover, Joy A. Thomas. (2006). "Elements of Information Theory". John Wiley & Sons, New York.
4. (2019-09-09). "Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm".
5. (2017-04-07). "Stein Variational Policy Gradient".

::callout[type=info title="Wikipedia Source"] This article was imported from Wikipedia and is available under the Creative Commons Attribution-ShareAlike 4.0 License. Content has been adapted to SurfDoc format. Original contributors can be found on the article history page. ::