MLE of Gaussian models¶

Gaussian model¶

z \sim N (μ (θ_{0}), Σ (θ_{0})) .

ℓ (θ | z) = - \frac{T}{2} \log (2 π) - \frac{1}{2} \log (| Σ (θ) |) - \frac{1}{2} {(z - μ (θ))}^{'} Σ^{- 1} (θ) (z - μ (θ))

\begin{array}{r} \begin{aligned} {S_{T} (θ)}_{i} & = \frac{1}{2} tr (\frac{\partial Σ^{- 1} (θ)}{\partial θ_{i}} Σ (θ)) + \frac{\partial μ (θ)^{'}}{\partial θ_{i}} Σ^{- 1} (θ) (z - μ (θ)) - \frac{1}{2} (z - μ (θ))^{'} \frac{\partial Σ^{- 1} (θ)}{\partial θ_{i}} (z - μ (θ)) \\ = \frac{\partial μ (θ)^{'}}{\partial θ_{i}} Σ^{- 1} (θ) (z - μ (θ)) + \frac{1}{2} tr (\frac{\partial Σ^{- 1} (θ)}{\partial θ_{i}} (Σ (θ) - (z - μ (θ)) (z - μ (θ))^{'})) \\ = \frac{\partial μ (θ)^{'}}{\partial θ_{i}} Σ^{- 1} (θ) (z - μ (θ)) + \frac{1}{2} vec {(\frac{\partial Σ^{- 1} (θ)}{\partial θ_{i}})}^{'} vec (Σ (θ) - (z - μ (θ)) (z - μ (θ))^{'}) \end{aligned} \end{array}

\begin{array}{r} S_{T} (θ) = 0 \\ \frac{\partial μ (θ)^{'}}{\partial θ_{i}} Σ^{- 1} (θ) (z - μ (θ)) + \frac{1}{2} vec {(\frac{\partial Σ^{- 1} (θ)}{\partial θ_{i}})}^{'} vec (Σ (θ) - (z - μ (θ)) (z - μ (θ))^{'}) = 0 \end{array}

Assume that we know $μ (θ) = 0$ (data is de-meaned). MLE solves

\begin{array}{r} \begin{aligned} \frac{1}{2} vec {(\frac{\partial Σ^{- 1} (θ)}{\partial θ_{i}})}^{'} vec (Σ (θ) - z z^{'}) & = 0 \end{aligned} \end{array}

for all $i$

\begin{array}{r} \begin{aligned} W (θ) vec (Σ (θ) - z z^{'}) & = 0 \end{aligned} \end{array}

MLE picks values of $θ$ that minimize the difference between empirical ( $z z^{'}$ ) and theoretical ( $Σ (θ)$ ) second moments
Optimality means that information about $θ$ is maximized, i.e. estimation uncertainty is minimized
MLE is equivalent to GMM with a weighting matrix which is optimal when the true distribution is Gaussian.
When $μ (θ) \neq 0$ , the same intuition holds: MLE picks $θ$ so as to minimize the difference between empirical and theoretical first and second order moments.

What if the true model is not Gaussian?

Fisher information matrix

\begin{array}{r} \begin{aligned} I_{T, i j} (θ) & = {(\frac{\partial μ (θ)}{\partial θ_{i}})}^{'} Σ^{- 1} (θ) (\frac{\partial μ (θ)}{\partial θ_{j}}) \\ + \frac{1}{2} tr (\frac{\partial Σ^{- 1} (θ)}{\partial θ_{i}} Σ (θ) \frac{\partial Σ^{- 1} (θ)}{\partial θ_{j}} Σ (θ)) \end{aligned} \end{array}

asymptotic variance matrix of MLE $\hat{θ}$

{(\lim_{T \to \infty} \frac{1}{T} I_{T, i j} (θ_{0}))}^{- 1}