World Library  
Flag as Inappropriate
Email this Article
 

Wishart distribution

Wishart
Notation X ~ Wp(V, n)
Parameters n > p − 1 degrees of freedom (real)
V > 0 scale matrix (p × p pos. def)
Support X(p × p) positive definite matrix
PDF

\frac{|\mathbf{X}|^{\frac{n-p-1}{2}} e^{-\frac}{2^\frac{np}{2}|{\mathbf V}|^\frac{n}{2}\Gamma_p(\frac{n}{2})}

Mean nV
Mode (np − 1)V for np + 1
Variance \operatorname{Var}(\mathbf{X}_{ij}) = n \left (v_{ij}^2+v_{ii}v_{jj} \right )
Entropy see below
CF \Theta \mapsto \left|{\mathbf I} - 2i\,{\mathbf\Theta}{\mathbf V}\right|^{-\frac{n}{2}}

In statistics, the Wishart distribution is a generalization to multiple dimensions of the chi-squared distribution, or, in the case of non-integer degrees of freedom, of the gamma distribution. It is named in honor of John Wishart, who first formulated the distribution in 1928.[1]

It is a family of probability distributions defined over symmetric, nonnegative-definite matrix-valued random variables (“random matrices”). These distributions are of great importance in the estimation of covariance matrices in multivariate statistics. In Bayesian statistics, the Wishart distribution is the conjugate prior of the inverse covariance-matrix of a multivariate-normal random-vector.

Contents

  • Definition 1
  • Occurrence 2
  • Probability density function 3
  • Use in Bayesian statistics 4
    • Choice of parameters 4.1
  • Properties 5
    • Log-expectation 5.1
    • Entropy 5.2
    • Cross-Entropy 5.3
    • KL-Divergence 5.4
    • Characteristic function 5.5
  • Theorem 6
    • Corollary 1 6.1
    • Corollary 2 6.2
  • Estimator of the multivariate normal distribution 7
  • Bartlett decomposition 8
  • Marginal distribution of matrix elements 9
  • The possible range of the shape parameter 10
  • Relationships to other distributions 11
  • See also 12
  • References 13
  • External links 14

Definition

Suppose X is an n × p matrix, each row of which is independently drawn from a p-variate normal distribution with zero mean:

X_{(i)}{=}(x_i^1,\dots,x_i^p)^T\sim N_p(0,V).

Then the Wishart distribution is the probability distribution of the p × p random matrix S = XT X known as the scatter matrix. One indicates that S has that probability distribution by writing

S\sim W_p(V,n).

The positive integer n is the number of degrees of freedom. Sometimes this is written W(V, p, n). For np the matrix S is invertible with probability 1 if V is invertible.

If p = V = 1 then this distribution is a chi-squared distribution with n degrees of freedom.

Occurrence

The Wishart distribution arises as the distribution of the sample covariance matrix for a sample from a multivariate normal distribution. It occurs frequently in likelihood-ratio tests in multivariate statistical analysis. It also arises in the spectral theory of random matrices and in multidimensional Bayesian analysis.[2] It is also encountered in wireless communications, while analyzing the performance of Rayleigh fading MIMO wireless channels .[3]

Probability density function

The Wishart distribution can be characterized by its probability density function as follows:

Let X be a p × p symmetric matrix of random variables that is positive definite. Let V be a (fixed) positive definite matrix of size p × p.

Then, if np, X has a Wishart distribution with n degrees of freedom if it has a probability density function given by

\frac{1}{2^\frac{np}{2}\left|{\mathbf V}\right|^\frac{n}{2}\Gamma_p(\frac{n}{2})} {\left|\mathbf{X}\right|}^{\frac{n-p-1}{2}} e^{-\frac{1}{2}{\rm tr}({\mathbf V}^{-1}\mathbf{X})}

where \left|{\mathbf X}\right| denotes determinant and Γp(·) is the multivariate gamma function defined as

\Gamma_p \left (\tfrac{n}{2} \right )= \pi^{\frac{p(p-1)}{4}}\Pi_{j=1}^p \Gamma\left ( \tfrac{n}{2} + \tfrac{1-j}{2} \right ).

In fact the above definition can be extended to any real n > p − 1. If np − 1, then the Wishart no longer has a density—instead it represents a singular distribution that takes values in a lower-dimension subspace of the space of p × p matrices.[4]

Use in Bayesian statistics

In Bayesian statistics, in the context of the multivariate normal distribution, the Wishart distribution is the conjugate prior to the precision matrix Ω = Σ−1, where Σ is the covariance matrix.

Choice of parameters

The least informative, proper Wishart prior is obtained by setting n = p.

The prior mean of Wp(V, n) is nV, suggesting that a reasonable choice for V−1 would be nΣ0, where Σ0 is some prior guess for the covariance matrix.

Properties

Log-expectation

Note the following formula:[5]

\operatorname{E}[\ln|\mathbf{X}|] = \psi_p(n/2) + p\ln(2) + \ln|\mathbf{V}|

where \psi_p is the multivariate digamma function (the derivative of the log of the multivariate gamma function).

This plays a role in variational Bayes derivations for Bayes networks involving the Wishart distribution.

Entropy

The information entropy of the distribution has the following formula:[5]

\operatorname{H}[\mathbf{X}] = -\ln \left (B(\mathbf{V},n) \right ) -\frac{n-p-1}{2} \operatorname{E}[\ln|\mathbf{X}|] + \frac{np}{2}

where B(V, n) is the normalizing constant of the distribution:

B(\mathbf{V},n) = \frac{1}{\left|\mathbf{V}\right|^\frac{n}{2} 2^\frac{np}{2}\Gamma_p(\frac{n}{2})}

This can be expanded as follows:

\begin{align} \operatorname{H}[\mathbf{X}] &= \tfrac{n}{2}\ln|\mathbf{V}| +\tfrac{np}{2}\ln(2) + \ln\Gamma_p(\tfrac{n}{2}) -\tfrac{n-p-1}{2} \operatorname{E}[\ln|\mathbf{X}|] + \tfrac{np}{2} \\ &= \tfrac{n}{2}\ln|\mathbf{V}| + \tfrac{np}{2}\ln(2) + \ln\Gamma_p(\tfrac{n}{2}) -\tfrac{n-p-1}{2}\left( \psi_p\left(\tfrac{n}{2}\right) + p\ln(2) + \ln|\mathbf{V}|\right) + \tfrac{np}{2} \\ &= \tfrac{n}{2}\ln|\mathbf{V}| + \tfrac{np}{2}\ln(2) + \ln\Gamma_p(\tfrac{n}{2}) - \tfrac{n-p-1}{2}\psi_p\left(\tfrac{n}{2}\right) - \frac{n-p-1}{2} \left(p\ln(2) +\ln|\mathbf{V}| \right ) + \tfrac{np}{2} \\ &= \tfrac{p+1}{2}\ln|\mathbf{V}| + \tfrac{1}{2}p(p+1)\ln(2) + \ln\Gamma_p(\tfrac{n}{2}) - \tfrac{n-p-1}{2}\psi_p\left(\tfrac{n}{2}\right) + \tfrac{np}{2} \end{align}

Cross-Entropy

The cross entropy of two Wishart distributions p_0 with parameters n_0, V_0 and p_1 with parameters n_1, V_1 is


\begin{align} H(p_0, p_1) &= \operatorname{E}_{p_0}[-\log p_1]\\ &= \operatorname{E}_{p_0}\left[-\log \frac{|\mathbf{X}|^{\frac{n_1 - p - 1}{2}} e^{-\frac{\mathrm{tr}(\mathbf{V}_1^{-1} \mathbf{X})}{2}}}{2^{\frac{n_1 p}{2}} |\mathbf{V}_1|^{\frac{n_1}{2}} \Gamma_p(\tfrac{n_1}{2})}\right]\\ &= \tfrac{n_1 p}{2} \log 2 + \tfrac{n_1}{2} \log |\mathbf{V}_1| + \log \Gamma_p(\tfrac{n_1}{2}) - \tfrac{n_1 - p - 1}{2} \operatorname{E}_{p_0}[\log |\mathbf{X}|] + \tfrac{1}{2}\operatorname{E}_{p_0}[\mathrm{tr}(\mathbf{V}_1^{-1}\mathbf{X})] \\ &= \tfrac{n_1 p}{2} \log 2 + \tfrac{n_1}{2} \log |\mathbf{V}_1| + \log \Gamma_p(\tfrac{n_1}{2}) - \tfrac{n_1 - p - 1}{2} \left( \psi_p(\tfrac{n_0}{2}) + p \log 2 + \log |\mathbf{V}_0|\right)+ \tfrac{1}{2}\mathrm{tr}(\mathbf{V}_1^{-1} n_0 \mathbf{V}_0) \\ &=-\tfrac{n_1}{2} \log |\mathbf{V}_1^{-1} \mathbf{V}_0| + \tfrac{p+1}{2} \log |\mathbf{V}_0| + \tfrac{n_0}{2}\mathrm{tr}(\mathbf{V}_1^{-1} \mathbf{V}_0)+ \log \Gamma_p(\tfrac{n_1}{2}) - \tfrac{n_1 - p - 1}{2} \psi_p(\tfrac{n_0}{2}) + \tfrac{p(p+1)}{2} \log 2 \\ \end{align}

Note that when p_0=p_1 we recover the entropy.

KL-Divergence

The Kullback–Leibler divergence of p_1 from p_0 is


D_{KL}(p_0 \| p_1) = H(p_0, p_1) - H(p_0) =-\tfrac{n_1}{2} \log |\mathbf{V}_1^{-1} \mathbf{V}_0| + \tfrac{n_0}{2}(\mathrm{tr}(\mathbf{V}_1^{-1} \mathbf{V}_0) - p)+ \log \frac{\Gamma_p(\tfrac{n_1}{2})}{\Gamma_p(\tfrac{n_0}{2})} + \tfrac{n_0 - n_1 }{2} \psi_p(\tfrac{n_0}{2})

Characteristic function

The characteristic function of the Wishart distribution is

\Theta \mapsto \left|{\mathbf I} - 2i\,{\mathbf\Theta}{\mathbf V}\right|^{-\frac{n}{2}}.

In other words,

\Theta \mapsto \operatorname{E}\left [ \mathrm{exp}\left (i \mathrm{tr}(\mathbf{X}{\mathbf\Theta})\right )\right ] = \left|{\mathbf I} - 2i{\mathbf\Theta}{\mathbf V}\right|^{-\frac{n}{2}}

where E[⋅] denotes expectation. (Here Θ and I are matrices the same size as V(I is the identity matrix); and i is the square root of −1).[6]

Theorem

If a p × p random matrix X has a Wishart distribution with m degrees of freedom and variance matrix V — write \mathbf{X}\sim\mathcal{W}_p({\mathbf V},m) — and C is a q × p matrix of rank q, then [7]

\mathbf{C}\mathbf{X}{\mathbf C}^T \sim \mathcal{W}_q\left({\mathbf C}{\mathbf V}{\mathbf C}^T,m\right).

Corollary 1

If z is a nonzero p × 1 constant vector, then:[7]

{\mathbf z}^T\mathbf{X}{\mathbf z}\sim\sigma_z^2\chi_m^2.

In this case, \chi_m^2 is the chi-squared distribution and \sigma_z^2={\mathbf z}^T{\mathbf V}{\mathbf z} (note that \sigma_z^2 is a constant; it is positive because V is positive definite).

Corollary 2

Consider the case where zT = (0, ..., 0, 1, 0, ..., 0) (that is, the j-th element is one and all others zero). Then corollary 1 above shows that

w_{jj}\sim\sigma_{jj}\chi^2_m

gives the marginal distribution of each of the elements on the matrix's diagonal.

Noted statistician multivariate for the case when all univariate marginals belong to the same family.[8]

Estimator of the multivariate normal distribution

The Wishart distribution is the sampling distribution of the maximum-likelihood estimator (MLE) of the covariance matrix of a multivariate normal distribution.[9] A derivation of the MLE uses the spectral theorem.

Bartlett decomposition

The Bartlett decomposition of a matrix X from a p-variate Wishart distribution with scale matrix V and n degrees of freedom is the factorization:

\mathbf{X} = {\textbf L}{\textbf A}{\textbf A}^T{\textbf L}^T,

where L is the Cholesky factor of V, and:

\mathbf A = \begin{pmatrix} c_1 & 0 & 0 & \cdots & 0\\ n_{21} & c_2 &0 & \cdots& 0 \\ n_{31} & n_{32} & c_3 & \cdots & 0\\ \vdots & \vdots & \vdots &\ddots & \vdots \\ n_{p1} & n_{p2} & n_{p3} &\cdots & c_p \end{pmatrix}

where c_i^2 \sim \chi^2_{n-i+1} and nij ~ N(0, 1) independently.[10] This provides a useful method for obtaining random samples from a Wishart distribution.[11]

Marginal distribution of matrix elements

Let V be a 2 × 2 variance matrix characterized by correlation coefficient −1 < ρ < 1 and L its lower Cholesky factor:

\mathbf{V} = \begin{pmatrix} \sigma_1^2 & \rho \sigma_1 \sigma_2 \\ \rho \sigma_1 \sigma_2 & \sigma_2^2 \end{pmatrix}, \qquad \mathbf{L} = \begin{pmatrix} \sigma_1 & 0 \\ \rho \sigma_2 & \sqrt{1-\rho^2} \sigma_2 \end{pmatrix}

Multiplying through the Bartlett decomposition above, we find that a random sample from the 2 × 2 Wishart distribution is

\mathbf{X} = \begin{pmatrix} \sigma_1^2 c_1^2 & \sigma_1 \sigma_2 \left (\rho c_1^2 + \sqrt{1-\rho^2} c_1 n_{21} \right ) \\ \sigma_1 \sigma_2 \left (\rho c_1^2 + \sqrt{1-\rho^2} c_1 n_{21} \right ) & \sigma_2^2 \left(\left (1-\rho^2 \right ) c_2^2 + \left (\sqrt{1-\rho^2} n_{21} + \rho c_1 \right )^2 \right) \end{pmatrix}

The diagonal elements, most evidently in the first element, follow the χ2 distribution with n degrees of freedom (scaled by σ2) as expected. The off-diagonal element is less familiar but can be identified as a normal variance-mean mixture where the mixing density is a χ2 distribution. The corresponding marginal probability density for the off-diagonal element is therefore the variance-gamma distribution

f(x_{12}) = \frac{\left | x_{12} \right |^{\frac{n-1}{2}}}{\Gamma\left(\frac{n}{2}\right) \sqrt{2^{n-1} \pi \left (1-\rho^2 \right ) \left (\sigma_1 \sigma_2 \right )^{n+1}}} \cdot K_{\frac{n-1}{2}} \left(\frac{\left |x_{12} \right |}{\sigma_1 \sigma_2 \left (1-\rho^2 \right )}\right) \exp{\left(\frac{\rho x_{12}}{\sigma_1 \sigma_2 (1-\rho^2)}\right)}

where Kν(z) is the modified Bessel function of the second kind.[12] Similar results may be found for higher dimensions, but the interdependence of the off-diagonal correlations becomes increasingly complicated. It is also possible to write down the moment-generating function even in the noncentral case (essentially the nth power of Craig (1936)[13] equation 10) although the probability density becomes an infinite sum of Bessel functions.

The possible range of the shape parameter

It can be shown [14] that the Wishart distribution can be defined if and only if the shape parameter n belongs to the set

\Lambda_p:=\{0,\cdots,p-1\}\cup \left(p-1,\infty\right).

This set is named after Gindikin, who introduced it[15] in the seventies in the context of gamma distributions on homogeneous cones. However, for the new parameters in the discrete spectrum of the Gindikin ensemble, namely,

\Lambda_p^*:=\{0, \cdots, p-1\},

the corresponding Wishart distribution has no Lebesgue density.

Relationships to other distributions

See also

References

  1. ^  
  2. ^ Gelman, A. (2013). Bayesian Data Analysis. Chapman & Hall. p. 582.  
  3. ^ Zanella, A.; Chiani, M.; Win, M.Z. (April 2009). "On the marginal distribution of the eigenvalues of wishart matrices". IEEE Transactions on Communications 57 (4): 1050–1060.  
  4. ^ Uhlig, H. (1994). "On Singular Wishart and Singular Multivariate Beta Distributions". The Annals of Statistics 22: 395.  
  5. ^ a b C.M. Bishop, Pattern Recognition and Machine Learning, Springer 2006, p. 693.
  6. ^  
  7. ^ a b Rao, C. R. (1965). Linear Statistical Inference and its Applications. Wiley. p. 535. 
  8. ^ Seber, George A. F. (2004). Multivariate Observations.  
  9. ^ Chatfield, C.; Collins, A. J. (1980). Introduction to Multivariate Analysis. London: Chapman and Hall. pp. 103–108.  
  10. ^  
  11. ^ Smith, W. B.; Hocking, R. R. (1972). "Algorithm AS 53: Wishart Variate Generator".  
  12. ^  
  13. ^ Craig, Cecil C. (1936). "On the Frequency Function of xy". Ann. Math. Statist. 7: 1–15.  
  14. ^ Peddada and Richards, Shyamal Das; Richards, Donald St. P. (1991). "Proof of a Conjecture of M. L. Eaton on the Characteristic Function of the Wishart Distribution,".  
  15. ^ Gindikin, S.G. (1975). "Invariant generalized functions in homogeneous domains,".  
  16. ^ Dwyer, Paul S. (1967). "Some Applications of Matrix Derivatives in Multivariate Analysis".  
  17. ^ Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. 

External links

  • A C++ library for random matrix generator
This article was sourced from Creative Commons Attribution-ShareAlike License; additional terms may apply. World Heritage Encyclopedia content is assembled from numerous content providers, Open Access Publishing, and in compliance with The Fair Access to Science and Technology Research Act (FASTR), Wikimedia Foundation, Inc., Public Library of Science, The Encyclopedia of Life, Open Book Publishers (OBP), PubMed, U.S. National Library of Medicine, National Center for Biotechnology Information, U.S. National Library of Medicine, National Institutes of Health (NIH), U.S. Department of Health & Human Services, and USA.gov, which sources content from all federal, state, local, tribal, and territorial government publication portals (.gov, .mil, .edu). Funding for USA.gov and content contributors is made possible from the U.S. Congress, E-Government Act of 2002.
 
Crowd sourced content that is contributed to World Heritage Encyclopedia is peer reviewed and edited by our editorial staff to ensure quality scholarly research articles.
 
By using this site, you agree to the Terms of Use and Privacy Policy. World Heritage Encyclopedia™ is a registered trademark of the World Public Library Association, a non-profit organization.
 


Copyright © World Library Foundation. All rights reserved. eBooks from Project Gutenberg are sponsored by the World Library Foundation,
a 501c(4) Member's Support Non-Profit Organization, and is NOT affiliated with any governmental agency or department.