Skip to content
Surf Wiki
Save to docs
general/normal-distribution

From Surf Wiki (app.surf) — the open knowledge base

Chi-squared distribution

Probability distribution and special case of gamma distribution


Probability distribution and special case of gamma distribution

Note

the mathematics of the chi-squared distribution

In probability theory and statistics, the \chi^2-distribution with k degrees of freedom is the distribution of a sum of the squares of k independent standard normal random variables.

The chi-squared distribution \chi^2_k is a special case of the gamma distribution and the univariate Wishart distribution. Specifically if X \sim \chi^2_k then X \sim \text{Gamma}(\alpha=\frac{k}{2}, \theta=2) (where \alpha is the shape parameter and \theta the scale parameter of the gamma distribution) and X \sim \text{W}_1(1,k) .

The scaled chi-squared distribution s^2 \chi^2_k is a reparametrization of the gamma distribution and the univariate Wishart distribution. Specifically if X \sim s^2 \chi^2_k then X \sim \text{Gamma}(\alpha=\frac{k}{2}, \theta=2 s^2) and X \sim \text{W}_1(s^2,k) .

The chi-squared distribution is one of the most widely used probability distributions in inferential statistics, notably in hypothesis testing and in construction of confidence intervals.{{cite book

The chi-squared distribution is used in the common chi-squared tests for goodness of fit of an observed distribution to a theoretical one, the independence of two criteria of classification of qualitative data, and in finding the confidence interval for estimating the population standard deviation of a normal distribution from a sample standard deviation. Many other statistical tests also use this distribution, such as Friedman's analysis of variance by ranks.

Definitions

If Z1, ..., Z**k are independent, standard normal random variables, then the sum of their squares, X\ = \sum_{i=1}^k Z_i^2, is distributed according to the chi-squared distribution with k degrees of freedom. This is usually denoted as X\ \sim\ \chi^2(k)\ \ \text{or}\ \ X\ \sim\ \chi^2_k.

The chi-squared distribution has one parameter: a positive integer k that specifies the number of degrees of freedom (the number of random variables Z**i being summed).

Introduction

The chi-squared distribution is used primarily in hypothesis testing, and to a lesser extent for confidence intervals for population variance when the underlying distribution is normal. Unlike more widely known distributions such as the normal distribution and the exponential distribution, the chi-squared distribution is not as often applied in the direct modeling of natural phenomena. It arises in the following hypothesis tests, among others:

  • Chi-squared test of independence in contingency tables
  • Chi-squared test of goodness of fit of observed data to hypothetical distributions
  • Likelihood-ratio test for nested models
  • Log-rank test in survival analysis
  • Cochran–Mantel–Haenszel test for stratified contingency tables
  • Wald test
  • Score test

It is also a component of the definition of the t-distribution and the F-distribution used in t-tests, analysis of variance, and regression analysis.

The primary reason for which the chi-squared distribution is extensively used in hypothesis testing is its relationship to the normal distribution. Many hypothesis tests use a test statistic, such as the t-statistic in a t-test. For these hypothesis tests, as the sample size, n, increases, the sampling distribution of the test statistic approaches the normal distribution (central limit theorem). Because the test statistic (such as t) is asymptotically normally distributed, provided the sample size is sufficiently large, the distribution used for hypothesis testing may be approximated by a normal distribution. Testing hypotheses using a normal distribution is well understood and relatively easy. The simplest chi-squared distribution is the square of a standard normal distribution. So wherever a normal distribution could be used for a hypothesis test, a chi-squared distribution could be used.

Suppose that Z is a random variable sampled from the standard normal distribution, where the mean is 0 and the variance is 1: Z \sim N(0,1). Now, consider the random variable X = Z^2. The distribution of the random variable X is an example of a chi-squared distribution: \ X\ \sim\ \chi^2_1. The subscript 1 indicates that this particular chi-squared distribution is constructed from only 1 standard normal distribution. A chi-squared distribution constructed by squaring a single standard normal distribution is said to have 1 degree of freedom. Thus, as the sample size for a hypothesis test increases, the distribution of the test statistic approaches a normal distribution. Just as extreme values of the normal distribution have low probability (and give small p-values), extreme values of the chi-squared distribution have low probability.

An additional reason that the chi-squared distribution is widely used is that it turns up as the large sample distribution of generalized likelihood ratio tests (LRT). LRTs have several desirable properties; in particular, simple LRTs commonly provide the highest power to reject the null hypothesis (Neyman–Pearson lemma) and this leads also to optimality properties of generalised LRTs. However, the normal and chi-squared approximations are only valid asymptotically. For this reason, it is preferable to use the t distribution rather than the normal approximation or the chi-squared approximation for a small sample size. Similarly, in analyses of contingency tables, the chi-squared approximation will be poor for a small sample size, and it is preferable to use Fisher's exact test. Ramsey shows that the exact binomial test is always more powerful than the normal approximation.

Lancaster shows the connections among the binomial, normal, and chi-squared distributions, as follows. De Moivre and Laplace established that a binomial distribution could be approximated by a normal distribution. Specifically they showed the asymptotic normality of the random variable

\chi = \frac{m - Np}{\sqrt{Npq}}

where m is the observed number of successes in N trials, where the probability of success is p, and q = 1 - p.

Squaring both sides of the equation gives

\chi^2 = \frac{\left(m - Np\right)^2}{Npq}

Using N = Np + N(1 - p), N = m + (N - m), and q = 1 - p, this equation can be rewritten as

\chi^2 = \frac{\left(m - Np\right)^2}{Np} + \frac{\left(N - m - Nq\right)^2}{Nq}

The expression on the right is of the form that Karl Pearson would generalize to the form

\chi^2 = \sum_{i=1}^n \frac{\left(O_i - E_i\right)^2}{E_i}

where

  • \chi^2 = Pearson's cumulative test statistic, which asymptotically approaches a \chi^2 distribution;
  • O_i = the number of observations of type i;
  • E_i = N p_i = the expected (theoretical) frequency of type i, asserted by the null hypothesis that the fraction of type i in the population is p_i; and
  • n = the number of cells in the table.

In the case of a binomial outcome (flipping a coin), the binomial distribution may be approximated by a normal distribution (for sufficiently large n). Because the square of a standard normal distribution is the chi-squared distribution with one degree of freedom, the probability of a result such as 1 heads in 10 trials can be approximated either by using the normal distribution directly, or the chi-squared distribution for the normalised, squared difference between observed and expected value. However, many problems involve more than the two possible outcomes of a binomial, and instead require 3 or more categories, which leads to the multinomial distribution. Just as de Moivre and Laplace sought for and found the normal approximation to the binomial, Pearson sought for and found a degenerate multivariate normal approximation to the multinomial distribution (the numbers in each category add up to the total sample size, which is considered fixed). Pearson showed that the chi-squared distribution arose from such a multivariate normal approximation to the multinomial distribution, taking careful account of the statistical dependence (negative correlations) between numbers of observations in different categories.

Probability density function

The probability density function (pdf) of the chi-squared distribution is f(x;,k) = \begin{cases} \dfrac{x^{k/2 -1} e^{-x/2}}{2^{k/2} , \Gamma{\left(\frac k 2 \right)}}, & x 0; \ 0, & \text{otherwise}. \end{cases} where \Gamma(k/2) denotes the gamma function, which has closed-form values for integer k.

For derivations of the pdf in the cases of one, two and k degrees of freedom, see Proofs related to chi-squared distribution.

Cumulative distribution function

Its cumulative distribution function is: F(x;,k) = \frac{\gamma{\left(\frac{k}{2},,\frac{x}{2}\right)}}{\Gamma{\left(\frac{k}{2}\right)}} = P{\left(\frac{k}{2},,\frac{x}{2}\right)}, where \gamma(s,t) is the lower incomplete gamma function and P(s,t) is the regularized gamma function.

In a special case of k = 2 this function has the simple form: F(x;,2) = 1 - e^{-x/2} which can be easily derived by integrating f(x;,2) = \frac{1}{2}e^{-x/2} directly. The integer recurrence of the gamma function makes it easy to compute F(x;,k) for other small, even k.

Tables of the chi-squared cumulative distribution function are widely available and the function is included in many spreadsheets and all statistical packages.

Letting z \equiv x/k, Chernoff bounds on the lower and upper tails of the CDF may be obtained. For the cases when 0 (which include all of the cases when this CDF is less than half): F(z k;,k) \leq (z e^{1-z})^{k/2}.

The tail bound for the cases when z 1, similarly, is 1-F(z k;,k) \leq (z e^{1-z})^{k/2}.

For another approximation for the CDF modeled after the cube of a Gaussian, see under Noncentral chi-squared distribution.

Properties

Cochran's theorem

Main article: Cochran's theorem

The following is a special case of Cochran's theorem.

Theorem. If Z_1,...,Z_n are independent identically distributed (i.i.d.), standard normal random variables, then \sum_{t=1}^n \left(Z_t - \bar Z\right)^2 \sim \chi^2_{n-1} where \bar Z = \frac{1}{n} \sum_{t=1}^n Z_t.

Proof. Let Z\sim\mathcal{N}(\bar 0,1!!1) be a vector of n independent normally distributed random variables, and \bar Z their average. Then \sum_{t=1}^n(Z_t-\bar Z)^2 = \sum_{t=1}^n Z_t^2 -n\bar Z^2 = Z^\top[1!!1 -{\textstyle\frac1n}\bar 1\bar 1^\top]Z =: Z^\top!M Z where 1!!1 is the identity matrix and \bar 1 the all ones vector. M has one eigenvector b_1:={\textstyle\frac{1}{\sqrt{n}}} \bar 1 with eigenvalue 0, and n-1 eigenvectors b_2,...,b_n (all orthogonal to b_1) with eigenvalue 1, which can be chosen so that Q:=(b_1,...,b_n) is an orthogonal matrix. Since also X:=Q^\top!Z\sim\mathcal{N}(\bar 0,Q^\top!1!!1 Q) =\mathcal{N}(\bar 0,1!!1), we have \sum_{t=1}^n(Z_t-\bar Z)^2 = Z^\top!M Z = X^\top!Q^\top!M Q X = X_2^2+...+X_n^2 \sim \chi^2_{n-1}, which proves the claim.

Additivity

It follows from the definition of the chi-squared distribution that the sum of independent chi-squared variables is also chi-squared distributed. Specifically, if X_i,i=\overline{1,n} are independent chi-squared variables with k_i, i=\overline{1,n} degrees of freedom, respectively, then Y = X_1 + \cdots + X_n is chi-squared distributed with k_1 + \cdots + k_n degrees of freedom.

Sample mean

The sample mean of n i.i.d. chi-squared variables of degree k is distributed according to a gamma distribution with shape \alpha and scale \theta parameters: \overline X = \frac{1}{n} \sum_{i=1}^n X_i \sim \operatorname{Gamma}\left(\alpha{=}\tfrac{n k}{2}, ,\theta{=}\tfrac{2}{n} \right) \qquad \text{where } X_i \sim \chi^2(k)

Asymptotically, given that for a shape parameter \alpha going to infinity, a Gamma distribution converges towards a normal distribution with expectation \mu = \alpha \theta and variance the sample mean converges towards:

\overline X \xrightarrow{n \to \infty} N{\left(\mu{=}k, , \sigma^2{=}\tfrac{2k}{n} \right)}

Note that we would have obtained the same result invoking instead the central limit theorem, noting that for each chi-squared variable of degree k the expectation is k , and its variance 2k (and hence the variance of the sample mean \overline{X} being \sigma^2 = \tfrac{2k}{n} ).

Entropy

The differential entropy is given by \begin{align} h &= \int_0^\infty f(x;,k) \ln f(x;,k) , dx \ &= \frac k 2 + \ln \left[2,\Gamma{\left(\frac k 2 \right)}\right] + \left(1-\frac k 2 \right) \psi!\left(\frac k 2 \right), \end{align} where \psi(x) is the Digamma function.

The chi-squared distribution is the maximum entropy probability distribution for a random variate X for which \operatorname{E}(X)=k and \operatorname{E}(\ln(X))=\psi(k/2)+\ln(2) are fixed. Since the chi-squared is in the family of gamma distributions, this can be derived by substituting appropriate values in the Expectation of the log moment of gamma. For derivation from more basic principles, see the derivation in moment-generating function of the sufficient statistic.

Noncentral moments

The noncentral moments (raw moments) of a chi-squared distribution with k degrees of freedom are given by \begin{align} \operatorname{E}(X^m) &= k (k+2) (k+4) \cdots (k+2m-2) \[1ex] &= 2^m \frac{\Gamma{\left(m+\frac{k}{2}\right)}}{\Gamma{\left(\frac{k}{2}\right)}}. \end{align}

Cumulants

The cumulants are readily obtained by a power series expansion of the logarithm of the characteristic function: \kappa_n = 2^{n-1}(n-1)!,k with cumulant generating function \ln \operatorname{E}[e^{tX}] = - \frac{k}{2} \ln(1-2t) .

Concentration

The chi-squared distribution exhibits strong concentration around its mean. The standard Laurent-Massart bounds are: \Pr(X - k \ge 2 \sqrt{k x} + 2x) \le e^{-x} \Pr(k - X \ge 2 \sqrt{k x}) \le e^{-x} One consequence is that, if Z \sim N(0, 1)^k is a Gaussian random vector in \R^k, then as the dimension k grows, the squared length of the vector is concentrated tightly around k with a width k^{1/2 + \alpha}:\Pr\left(\left|Z\right|^2 \in \left[k - 2k^{1/2+\alpha}, ; k + 2k^{1/2+\alpha} + 2k^{\alpha}\right]\right) \geq 1-e^{-k^\alpha}where the exponent \alpha can be chosen as any value in \R.

Since the cumulant generating function for \chi^2(k) is K(t) = -\frac k2 \ln(1-2t) , and its convex dual is K^*(q) = \frac{1}{2} \left(q - k + k\ln\frac{k}{q}\right) , the standard Chernoff bound yields\begin{aligned} \ln \Pr(X \geq (1 + \varepsilon) k) &\leq -\frac{k}{2} \left( \varepsilon - \ln(1+\varepsilon)\right) \ \ln \Pr(X \leq (1 - \varepsilon) k) &\leq -\frac{k}{2} \left(-\varepsilon - \ln(1-\varepsilon)\right) \end{aligned}where 0 . By the union bound,Pr(X \in (1\pm \varepsilon ) k ) \geq 1 - 2 e^{-\frac k2 (\frac{1}{2} \varepsilon^2 - \frac 13 \varepsilon^3)} This result is used in proving the Johnson–Lindenstrauss lemma.

Asymptotic properties

red}}) between numerical quantile and approximate formula (bottom). For the chi-squared distribution, only the positive integer numbers of degrees of freedom (circles) are meaningful.

By the central limit theorem, because the chi-squared distribution is the sum of k independent random variables with finite mean and variance, it converges to a normal distribution for large k. For many practical purposes, for k50 the distribution is sufficiently close to a normal distribution, so the difference is ignorable. Specifically, if X \sim \chi^2(k), then as k tends to infinity, the distribution of (X-k)/\sqrt{2k} tends to a standard normal distribution. However, convergence is slow as the skewness is \sqrt{8/k} and the excess kurtosis is 12/k.

The sampling distribution of \ln(\chi^2) converges to normality much faster than the sampling distribution of \chi^2, as the logarithmic transform removes much of the asymmetry.

Other functions of the chi-squared distribution converge more rapidly to a normal distribution. Some examples are:

  • If X \sim \chi^2(k) then \sqrt{2X} is approximately normally distributed with mean \sqrt{2k-1} and unit variance (1922, by R. A. Fisher, see (18.23), p. 426 of Johnson).
  • If X \sim \chi^2(k) then \sqrt[3]{X/k} is approximately normally distributed with mean 1-\frac{2}{9k} and variance \frac{2}{9k} . This is known as the Wilson–Hilferty transformation, see (18.24), p. 426 of Johnson.
    • This normalizing transformation leads directly to the commonly used median approximation k\bigg(1-\frac{2}{9k}\bigg)^3; by back-transforming from the mean, which is also the median, of the normal distribution.

Occurrence and applications{{anchor|Applications}}

The chi-squared distribution has numerous applications in inferential statistics, for instance in chi-squared tests and in estimating variances. It enters the problem of estimating the mean of a normally distributed population and the problem of estimating the slope of a regression line via its role in Student's t-distribution. It enters all analysis of variance problems via its role in the F-distribution, which is the distribution of the ratio of two independent chi-squared random variables, each divided by their respective degrees of freedom.

Following are some of the most common situations in which the chi-squared distribution arises from a Gaussian-distributed sample.

  • if X_1, ..., X_n are i.i.d. N(\mu, \sigma^2) random variables, then \sum_{i=1}^n \left(X_i - \bar{X}\right)^2 \sim \sigma^2 \chi^2_{n-1} where \bar{X} = \frac{1}{n} \sum_{i=1}^n X_i.
  • The box below shows some statistics based on X_i \sim N(\mu_i, \sigma^2_i), i= 1, \ldots, k independent random variables that have probability distributions related to the chi-squared distribution:
NameStatistic
chi-squared distribution\sum_{i=1}^k \left(\frac{X_i-\mu_i}{\sigma_i}\right)^2
noncentral chi-squared distribution\sum_{i=1}^k \left(\frac{X_i}{\sigma_i}\right)^2
chi distribution\sqrt{\sum_{i=1}^k \left(\frac{X_i-\mu_i}{\sigma_i}\right)^2}
noncentral chi distribution\sqrt{\sum_{i=1}^k \left(\frac{X_i}{\sigma_i}\right)^2}

The chi-squared distribution is also often encountered in magnetic resonance imaging.

Computational methods

Table of {{math|''χ''2}} values vs {{math|''p''}}-values

The p-value is the probability of observing a test statistic at least as extreme in a chi-squared distribution. Accordingly, since the cumulative distribution function (CDF) for the appropriate degrees of freedom (df) gives the probability of having obtained a value less extreme than this point, subtracting the CDF value from 1 gives the p-value. A low p-value, below the chosen significance level, indicates statistical significance, i.e., sufficient evidence to reject the null hypothesis. A significance level of 0.05 is often used as the cutoff between significant and non-significant results.

The table below gives a number of p-values matching to \chi^2 for the first 10 degrees of freedom.

Degrees offreedom (df)\chi^2 value*p*-value(probability)
10.0040.02
20.100.21
30.350.58
40.711.06
51.141.61
61.632.20
72.172.83
82.733.49
93.324.17
103.944.87
0.950.900.80

These values can be calculated evaluating the quantile function (also known as "inverse CDF" or "ICDF") of the chi-squared distribution; e. g., the χ2 ICDF for and yields 2.1673 ≈ 2.17 as in the table above, noticing that 1 – p is the p-value from the table.

History

This distribution was first described by the German geodesist and statistician Friedrich Robert Helmert in papers of 1875–6, where he computed the sampling distribution of the sample variance of a normal population. Thus in German this was traditionally known as the Helmert'sche ("Helmertian") or "Helmert distribution".

The distribution was independently rediscovered by the English mathematician Karl Pearson in the context of goodness of fit, for which he developed his Pearson's chi-squared test, published in 1900, with computed table of values published in , collected in . The name "chi-square" ultimately derives from Pearson's shorthand for the exponent in a multivariate normal distribution with the Greek letter Chi, writing −χ2 for what would appear in modern notation as −xTΣ−1x (Σ being the covariance matrix).R. L. Plackett, Karl Pearson and the Chi-Squared Test, International Statistical Review, 1983, 61f. See also Jeff Miller, Earliest Known Uses of Some of the Words of Mathematics. The idea of a family of "chi-squared distributions", however, is not due to Pearson but arose as a further development due to Fisher in the 1920s.

References

Sources

References

  1. M.A. Sanders. "Characteristic function of the central chi-square distribution".
  2. Weisstein, Eric W.. "Chi-Squared Distribution".
  3. {{Abramowitz Stegun ref. 26. 940
  4. NIST (2006). [http://www.itl.nist.gov/div898/handbook/eda/section3/eda3666.htm Engineering Statistics Handbook – Chi-Squared Distribution]
  5. (1994). "Continuous Univariate Distributions". John Wiley and Sons.
  6. "The Chi-Squared Distribution".
  7. (2013). "Understanding Advanced Statistical Methods". CRC Press.
  8. (1988). "Evaluating the Normal Approximation to the Binomial Test". Journal of Educational Statistics.
  9. Lancaster, H.O.. (1969). "The Chi-squared Distribution". Wiley.
  10. (January 2003). "An Elementary Proof of a Theorem of Johnson and Lindenstrauss". Random Structures and Algorithms.
  11. [http://mathworld.wolfram.com/Chi-SquaredDistribution.html Chi-squared distribution], from [[MathWorld]], retrieved Feb. 11, 2009
  12. M. K. Simon, ''Probability Distributions Involving Gaussian Random Variables'', New York: Springer, 2002, eq. (2.35), {{ISBN. 978-0-387-34657-1
  13. (2000-10-01). "Adaptive estimation of a quadratic functional by model selection". The Annals of Statistics.
  14. [https://ocw.mit.edu/courses/18-s096-topics-in-mathematics-of-data-science-fall-2015/f9261308512f6b90e284599f94055bb4_MIT18_S096F15_Ses15_16.pdf MIT 18.S096 (Fall 2015): Topics in Mathematics of Data Science, Lecture 5, Johnson-Lindenstrauss Lemma and Gordons Theorem]
  15. Box, Hunter and Hunter. (1978). "Statistics for experimenters". Wiley.
  16. (1946). "The Statistical Analysis of Variance-Heterogeneity and the Logarithmic Transformation". Supplement to the Journal of the Royal Statistical Society.
  17. Pillai, Natesh S.. (2016). "An unexpected encounter with Cauchy and Lévy". [[Annals of Statistics]].
  18. (1931). "The distribution of chi-squared". [[Proc. Natl. Acad. Sci. USA]].
  19. Bäckström, T.. (January 2018). "Fast Randomization for Distributed Low-Bitrate Coding of Speech and Audio". IEEE/ACM Transactions on Audio, Speech, and Language Processing.
  20. den Dekker, A. J.. (2014). "Data distributions in magnetic resonance images: A review". Physica Medica.
  21. [http://www2.lv.psu.edu/jxm57/irp/chisquar.html Chi-Squared Test] {{Webarchive. link. (2013-11-18 Table B.2. Dr. Jacqueline S. McLaughlin at The Pennsylvania State University. In turn citing: R. A. Fisher and F. Yates, Statistical Tables for Biological Agricultural and Medical Research, 6th ed., Table IV. Two values have been corrected, 7.82 with 7.81 and 4.60 with 4.61)
  22. "Chi-squared Distribution | R Tutorial".
  23. [[F. R. Helmert]], "[http://gdz.sub.uni-goettingen.de/dms/load/img/?PPN=PPN599415665_0021&DMDID=DMDLOG_0018 Ueber die Wahrscheinlichkeit der Potenzsummen der Beobachtungsfehler und über einige damit im Zusammenhange stehende Fragen]", ''Zeitschrift für Mathematik und Physik'' [http://gdz.sub.uni-goettingen.de/dms/load/toc/?PPN=PPN599415665_0021 21], 1876, pp. 192–219
  24. (22 June 2021). "The Modified-Half-Normal distribution: Properties and an efficient sampling scheme". Communications in Statistics - Theory and Methods.
Info: Wikipedia Source

This article was imported from Wikipedia and is available under the Creative Commons Attribution-ShareAlike 4.0 License. Content has been adapted to SurfDoc format. Original contributors can be found on the article history page.

Want to explore this topic further?

Ask Mako anything about Chi-squared distribution — get instant answers, deeper analysis, and related topics.

Research with Mako

Free with your Surf account

Content sourced from Wikipedia, available under CC BY-SA 4.0.

This content may have been generated or modified by AI. CloudSurf Software LLC is not responsible for the accuracy, completeness, or reliability of AI-generated content. Always verify important information from primary sources.

Report