.ipynb .pdf

Binder

Contents

Maximum likelihood estimation

Time series model

• a specification of the joint distribution of $\{z_{𝑡}\}$

$$p(\mathbf{z};\mathit{\theta })$$

definition: Likelihood function

$$\mathcal{L}(\mathit{\theta }|\mathbf{z})=p(\mathbf{z};\mathit{\theta })$$

Note

The likelihood function is identical in functional form to the PDF of $\mathbf{z}$, $p(\mathbf{z};\mathit{\theta })$, but is interpreted as a function of $\mathit{\theta }$, for a given value of $\mathbf{z}$, rather than as a function of $\mathbf{z}$ for a given value of $\mathit{\theta }$.

definition: Log-likelihood function

$$\ell (\mathit{\theta }|\mathbf{z})=\log (\mathcal{L}(\mathit{\theta }|\mathbf{z})$$

The maximum likelihood estimator (MLE)

$$\begin{array}{r}\begin{array}{rl}\hat{\mathit{\theta }}& =\underset{\mathit{\theta }}{\mathrm{argmax}}\mathcal{L}(\mathit{\theta }|\mathbf{z})\\ & =\underset{\mathit{\theta }}{\mathrm{argmax}}\ell (\mathit{\theta }|\mathbf{z})\end{array}\end{array}$$

Rationale for MLE

For a given $\mathit{\theta }$, the value of $p(\mathbf{z};\mathit{\theta })d\mathbf{z}$ evaluated at the observed sample $\mathbf{z}$ tells us what is the probability of observing a sample in a small neighborhood around the actual $\mathbf{z}$ for that value of $\mathit{\theta }$. Compared to the MLE $\hat{\mathit{\theta }}$, any other value of $\mathit{\theta }$ is associated with a pdf that assigns a lower probability of observing such a sample. Therefore, $\hat{\mathit{\theta }}$ is the value most supported by the observed sample.

Note

Difference between ML estimator and ML estimate:

• estimator: $\hat{\mathit{\theta }}$ as a function of a generic sample $\mathbf{z}$

• estimate: the value $\hat{\mathit{\theta }}$ at a particular sample $\mathbf{z}$

Score

$${\mathit{S}}_T(\mathit{\theta })=\frac{\partial }{\partial \mathit{\theta }}\ell (\mathit{\theta }|\mathbf{z})$$

• describes the steepness of log-likelihood function

• MLE $\hat{\mathit{\theta }}$ solves

$${\mathit{S}}_T(\hat{\mathit{\theta }})=\mathbf{0}$$

Observed Fisher information

$$\begin{array}{r}\begin{array}{rl}{\mathit{I}}_T(\hat{\mathit{\theta }})& =-\frac{\partial }{\partial \mathit{\theta }}{\mathit{S}}_T(\mathit{\theta }){|}_{\hat{\mathit{\theta }}}\\ & =-\frac{{\partial }^2}{\partial \mathit{\theta }\partial {\mathit{\theta }}^{\prime }}\ell (\mathit{\theta }|\mathbf{z}){|}_{\hat{\mathit{\theta }}}\end{array}\end{array}$$

• describes the curvature of the log-likelihood function at the maximum $\hat{\mathit{\theta }}$

• measures how much information about $\mathit{\theta }$ we have at the MLE.

Expected Fisher information

$${\mathcal{I}}_T(\mathit{\theta })=\mathrm{E}[{\mathit{S}}_T(\mathit{\theta }){\mathit{S}}_T(\mathit{\theta }{)}^{\prime }]$$

$${\mathcal{I}}_T(\mathit{\theta })=-\mathrm{E}[\frac{\partial }{\partial \mathit{\theta }}{\mathit{S}}_T(\mathit{\theta })]=\mathrm{E}[{\mathit{I}}_T(\mathit{\theta })]$$

• expected curvature of the log-likelihood function

• measures how much information about $\mathit{\theta }$ we can expect to have

Consistency and asymptotic normality of MLE

Assumption: $\mathbf{z}$ is a draw from $p(\mathbf{z};{\mathit{\theta }}_0)$, ${\mathit{\theta }}_0$ - true value of $\mathit{\theta }$

• $\hat{\mathit{\theta }}$ is consitent estimator of ${\mathit{\theta }}_0$

$$\hat{\mathit{\theta }}⟶{\mathit{\theta }}_0$$

• $\hat{\mathit{\theta }}$ is asymptotically normally distributed

$$\sqrt{T}(\hat{\mathit{\theta }}-{\mathit{\theta }}_0)⟶\mathcal{N}(\mathbf{0},{\mathcal{I}}_{\mathrm{\infty }}^{-1}({\mathit{\theta }}_0))$$

where

$${\mathcal{I}}_{\mathrm{\infty }}(\mathit{\theta })=\underset{T\to \mathrm{\infty }}{\lim }\frac{1}{T}{\mathcal{I}}_T(\mathit{\theta })$$

$$\hat{\mathit{\theta }}\stackrel{a}{\sim }\mathcal{N}({\mathit{\theta }}_0,\frac{1}{T}{\mathcal{I}}_{\mathrm{\infty }}^{-1}({\mathit{\theta }}_0))$$

previous

Review of maximum likelihood estimation

next

MLE of Gaussian models