.ipynb .pdf

Binder

Contents

Multivariate Normal Distribution

Time series model

• a specification of the joint distribution of $\{z_{𝑡}\}$

Gaussian time series model

$$p(z_1,z_2,\cdots ,z_T)=\mathcal{N}(\mathit{\mu },\mathbf{\Sigma }).$$

• probability density:

$$p(\mathit{z};\mu ,\mathrm{\Sigma })={\left(2\pi \right)}^{-\left(\frac{T}{2}\right)}det{\left(\mathrm{\Sigma }\right)}^{-\frac{1}{2}}\exp (-.5{(\mathit{z}-\mu )}^{\prime }{\mathrm{\Sigma }}^{-1}(\mathit{z}-\mu ))$$

• where $\mu =\mathrm{E}\mathit{z}$ is the mean of the random vector $\mathit{z}$

• and $\mathrm{\Sigma }=\mathrm{E}(\mathit{z}-\mu ){(\mathit{z}-\mu )}^{\prime }=\mathrm{cov}(\mathit{z})$ is the covariance matrix of $\mathit{z}$

Building Gaussian time series models

• Gaussian innovations $\{{\epsilon }_{𝑡}\}$:

$$p({\epsilon }_1,{\epsilon }_2,\cdots ,{\epsilon }_T)=p(\mathit{\epsilon })=\mathcal{N}(\mathbf{0},{\sigma }^2\mathbf{I}).$$

• affine transformation $\mathit{\epsilon }\to \mathit{z}$

$$\mathit{z}=\mathit{\mu }+\mathit{A}\mathit{\epsilon }\sim \mathcal{N}(\mathit{\mu },\mathbf{\Sigma })$$

where

$$\mathbf{\Sigma }={\sigma }^2\mathit{A}{\mathit{A}}^{\prime }$$

General result

$$\mathit{z}\sim \mathcal{N}(\mathit{\mu },\mathbf{\Sigma })$$

• affine transformation $\mathit{z}\to \mathit{y}$

$$\mathit{y}=\mathit{d}+\mathit{B}\mathit{z}\sim \mathcal{N}(\mathit{d}+\mathit{B}\mathit{\mu },\mathit{B}\mathbf{\Sigma }{\mathit{B}}^{\prime })$$

The joint, marginal and conditional distributions

Joint

$$p(\underset{{\mathbf{z}}_1}{\underbrace{z_1,z_2,\cdots ,z_k}},\underset{{\mathbf{z}}_2}{\underbrace{z_{k+1},z_{k+2},\cdots ,z_T}})=p({\mathbf{z}}_1,{\mathbf{z}}_2)\sim \mathcal{N}(\mathit{\mu },\mathbf{\Sigma }).$$

with:

$$\begin{array}{r}\mathit{\mu }=\left[\begin{array}{c}{\mathit{\mu }}_1\\ {\mathit{\mu }}_2\end{array}\right]\text{and}\mathbf{\Sigma }=\left[\begin{array}{cc}{\mathbf{\Sigma }}_{11}& {\mathbf{\Sigma }}_{12}\\ {\mathbf{\Sigma }}_{21}& {\mathbf{\Sigma }}_{22}\end{array}\right],\text{ where}{\mathbf{\Sigma }}_{21}={\mathbf{\Sigma }}_{12}^{{}^{\prime }}\end{array}$$

Marginal

$$\begin{array}{r}p({\mathbf{z}}_1)={\int }_{{\mathbf{z}}_2}p({\mathbf{z}}_1,{\mathbf{z}}_2)\text{d}{\mathbf{z}}_2=\mathcal{N}({\mathit{\mu }}_1,{\mathbf{\Sigma }}_{11})\\ p({\mathbf{z}}_2)={\int }_{{\mathbf{z}}_1}p({\mathbf{z}}_1,{\mathbf{z}}_2)\text{d}{\mathbf{z}}_1=\mathcal{N}({\mathit{\mu }}_2,{\mathbf{\Sigma }}_{22})\end{array}$$

Conditional

$$\begin{array}{r}p({\mathbf{z}}_1|{\mathbf{z}}_2)=\mathcal{N}(\underset{\mathrm{E}({\mathbf{z}}_1|{\mathbf{z}}_2)}{\underbrace{{\mathit{\mu }}_1+{\mathbf{\Sigma }}_{12}{\mathbf{\Sigma }}_{22}^{-1}({\mathbf{z}}_2-{\mathit{\mu }}_2)}},\underset{\mathrm{cov}({\mathbf{z}}_1|{\mathbf{z}}_2)}{\underbrace{{\mathbf{\Sigma }}_{11}-{\mathbf{\Sigma }}_{12}{\mathbf{\Sigma }}_{22}^{-1}{\mathbf{\Sigma }}_{21}}})\end{array}$$

Let

$$\begin{array}{r}\mathbf{Q}={\mathbf{\Sigma }}^{-1}=\left[\begin{array}{cc}{\mathbf{Q}}_{11}& {\mathbf{Q}}_{12}\\ {\mathbf{Q}}_{21}& {\mathbf{Q}}_{22}\end{array}\right]\end{array}$$

Then

$$\begin{array}{r}\begin{array}{rl}\mathrm{E}({\mathbf{z}}_1|{\mathbf{z}}_2)& ={\mathit{\mu }}_1+{\mathbf{Q}}_{11}^{-1}{\mathbf{Q}}_{12}({\mathbf{z}}_2-{\mathit{\mu }}_2)\\ \mathrm{cov}({\mathbf{z}}_1|{\mathbf{z}}_2)& ={\mathbf{Q}}_{11}^{-1}\end{array}\end{array}$$

$\mathbf{Q}$ is called the precision matrix.

Independence

$$\begin{array}{r}\text{if}{\mathbf{\Sigma }}_{12}=\mathbf{0}\\ p({\mathbf{z}}_1|{\mathbf{z}}_2)=\mathcal{N}({\mathit{\mu }}_1,{\mathbf{\Sigma }}_{11})=p({\mathbf{z}}_1)\end{array}$$

If ${\mathbf{z}}_1\sim \mathcal{N}({\mathit{\mu }}_1,{\mathbf{\Sigma }}_{11})$ and ${\mathbf{z}}_2\sim \mathcal{N}({\mathit{\mu }}_2,{\mathbf{\Sigma }}_{22})$ are independent then

$${\mathbf{z}}_1+{\mathbf{z}}_2\sim \mathcal{N}({\mathit{\mu }}_1+{\mu }_2,{\mathbf{\Sigma }}_{11}+{\mathbf{\Sigma }}_{22})$$

and

$$p({\mathbf{z}}_1,A{\mathbf{z}}_1+B{\mathbf{z}}_2)\sim \mathcal{N}(\mathit{\mu },\mathbf{\Sigma }).$$

with:

$$\begin{array}{r}\mathit{\mu }=\left[\begin{array}{c}{\mathit{\mu }}_1\\ A{\mathit{\mu }}_1+B{\mathit{\mu }}_2\end{array}\right]\text{and}\mathbf{\Sigma }=\left[\begin{array}{cc}{\mathbf{\Sigma }}_{11}& {\mathbf{\Sigma }}_{11}{A}^{\prime }\\ A{\mathbf{\Sigma }}_{11}& A{\mathbf{\Sigma }}_{11}{A}^{\prime }+B{\mathbf{\Sigma }}_{22}{B}^{\prime }\end{array}\right]\end{array}$$

Information

$$\mathrm{cov}({\mathbf{z}}_1|{\mathbf{z}}_2)={\mathbf{\Sigma }}_{11}-{\mathbf{\Sigma }}_{12}{\mathbf{\Sigma }}_{22}^{-1}{\mathbf{\Sigma }}_{21}\ge \mathbf{0}$$

• stronger correlation => more information from ${\mathbf{z}}_1$ about ${\mathbf{z}}_2$ (and from ${\mathbf{z}}_2$ about ${\mathbf{z}}_1$)

Bivariate Normal

$$p(z_1,z_2)\sim \mathcal{N}(\mathit{\mu },\mathbf{\Sigma }).$$

where:

$$\begin{array}{r}\mathit{\mu }=\left[\begin{array}{c}{\mu }_1\\ {\mu }_2\end{array}\right]\text{and}\mathbf{\Sigma }=\left[\begin{array}{cc}{\sigma }_1^2& \rho {\sigma }_1{\sigma }_2\\ \rho {\sigma }_1{\sigma }_2& {\sigma }_2^2\end{array}\right]\end{array}$$

$$\begin{array}{r}\begin{array}{rl}p(z_1|z_2)& =\mathcal{N}({\mu }_1+\frac{\rho {\sigma }_1{\sigma }_2}{{\sigma }_2^2}(z_2-{\mu }_2),{\sigma }_1^2-\frac{(\rho {\sigma }_1{\sigma }_2{)}^2}{{\sigma }_2^2})\\ & =\mathcal{N}({\mu }_1+\frac{\rho {\sigma }_1}{{\sigma }_2}(z_2-{\mu }_2),(1-{\rho }^2){\sigma }_1^2)\end{array}\end{array}$$

Example: ${\sigma }_1={\sigma }_2=1$, ${\mu }_1={\mu }_2=0$

case 1 $\rho =.9$

$$p(z_1)=\mathcal{N}(0,1)$$

rho=.9
sigma1 = 1
sigma2 = 1
mu1=0

mu2=0

y2 = 1

(
mu1 + (y2 - mu2)*(rho*sigma1*sigma2)/(sigma2**2), 
1 - (rho*sigma1*sigma2)**2/(sigma2**2)
)

(0.9, 0.18999999999999995)

If $z_2=1$

$$p(z_1|z_2=1)=\mathcal{N}(0.9,0.19)$$

Information about $z_1$ from observing $z_2$

$$\mathrm{var}(z_1)-\mathrm{var}(z_1|z_2)=1-0.19=0.81$$

Reduction of the uncertainty about $z_1$ by 81%

case 2 $\rho =.1$, ${\sigma }_1={\sigma }_2=1$, ${\mu }_1={\mu }_2=0$

$$p(z_1)=\mathcal{N}(0,1)$$

rho=.1
sigma1 = 1
sigma2 = 1
mu1=0
mu2=0

y2 = 1

(
mu1 + (y2 - mu2)*(rho*sigma1*sigma2)/(sigma2**2), 
1-(rho*sigma1*sigma2)**2/(sigma2**2)
)

(0.1, 0.99)

If $z_2=1$

$$p(z_1|z_2=1)=\mathcal{N}(0.1,0.99)$$

Reduction of the uncertainty about $z_1$ by 1%

Let $\mathbf{z}$ be jointly Gaussian, and partitioned as ($z_1$ is a scalar)

$$\mathbf{z}=[z_1,{\mathbf{z}}_2]$$

$$\begin{array}{r}\mathit{\mu }=\left[\begin{array}{c}{\mu }_1\\ {\mathit{\mu }}_2\end{array}\right]\text{and}\mathbf{\Sigma }=\left[\begin{array}{cc}{\mathrm{\Sigma }}_{11}& {\mathbf{\Sigma }}_{12}\\ {\mathbf{\Sigma }}_{21}& {\mathbf{\Sigma }}_{22}\end{array}\right]\end{array}$$

Then

$$\begin{array}{r}p(z_1|{\mathbf{z}}_2)=\mathcal{N}(\underset{{\beta }_0}{\underbrace{{\mu }_1-{\mathbf{\Sigma }}_{12}{\mathbf{\Sigma }}_{22}^{-1}{\mathit{\mu }}_2}}+\underset{{\beta }_1^{\prime }}{\underbrace{{\mathbf{\Sigma }}_{12}{\mathbf{\Sigma }}_{22}^{-1}}}{\mathbf{z}}_2,\underset{{\sigma }^2}{\underbrace{{\mathrm{\Sigma }}_{11}-{\mathbf{\Sigma }}_{12}{\mathbf{\Sigma }}_{22}^{-1}{\mathbf{\Sigma }}_{21}}})\end{array}$$

therefore,

$$z_1={\beta }_0+{\beta }_1^{\prime }{\mathbf{z}}_2+\epsilon \text{where }\epsilon \sim \mathcal{N}(0,{\sigma }^2)$$

Conditional (in)dependence

Two random variables $z_1$ and $z_2$ are conditionally independent given ${\mathbf{z}}_3$ if

$$\begin{array}{}\text{(1)}& p(z_1,z_2|{\mathbf{z}}_3)=p(z_1|{\mathbf{z}}_3)p(z_2|{\mathbf{z}}_3)\end{array}$$

If $z_1$, $z_2$ and ${\mathbf{z}}_3$ are jointly Gaussian, then (1) is true iff

$$\{{\mathbf{\Sigma }}^{-1}{\}}_{1,2}={\mathbf{Q}}_{1,2}=0$$

More generally, $z_i$ and $z_j$ are conditionally independent given the remaining elements of $\mathbf{z}$ (denote them with ${\mathbf{z}}_{-ij}$) iff

$$Q_{i,j}=0$$

And the conditional correlation between $z_i$ and $z_j$ is

$$\mathrm{corr}(z_i,z_j|{\mathbf{z}}_{-ij})=-\frac{Q_{i,j}}{\sqrt{Q_{i,i}{Q}_{jj}}}$$

also known as partial correlation between $z_i$ and $z_j$

Example 1

$$\begin{array}{r}\mathbf{Q}=\left[\begin{array}{ccc}1& -0.4& 0\\ -0.4& 1.16& -0.4\\ 0& -0.4& 1\end{array}\right]\end{array}$$

then

$$\begin{array}{r}\mathbf{\Sigma }=\left[\begin{array}{ccc}1.19047619& 0.47619048& 0.19047619\\ 0.47619048& 1.19047619& 0.47619048\\ 0.19047619& 0.47619048& 1.19047619\end{array}\right]\end{array}$$

therefore, $z_1$ and $z_3$ are unconditionally dependent, but conditionally (given $z_2$) independent

Example 2

$$\begin{array}{r}\mathbf{\Sigma }=\left[\begin{array}{ccc}1.09& 0.3& 0.\\ 0.3& 1.09& 0.3\\ 0& 0.3& 1.09\end{array}\right]\end{array}$$

then

$$\begin{array}{r}\mathbf{Q}=\left[\begin{array}{ccc}1.& -0.3& 0.08\\ -0.3& 1.08& -0.3\\ 0.08& -0.3& 1\end{array}\right]\end{array}$$

therefore, $z_1$ and $z_3$ are unconditionally independent, but conditionally (given $z_2$) dependent

from statsmodels.tsa.arima_process import arma_acovf
from scipy.linalg import toeplitz
import numpy as np

acov1 = arma_acovf(ar=[1, -.4], ma=[1], nobs=10, sigma2=1)
Sigma1 = toeplitz(acov1)
acov2 = arma_acovf(ar=[1,], ma=[1, .3], nobs=10, sigma2=1)
Sigma2 = toeplitz(acov2)


print(np.linalg.inv(Sigma1[:3][:,:3]).round(4))
print(np.linalg.inv(Sigma2[:3][:,:3]).round(4))

[[ 1.   -0.4   0.  ]
 [-0.4   1.16 -0.4 ]
 [ 0.   -0.4   1.  ]]
[[ 0.9993 -0.2976  0.0819]
 [-0.2976  1.0812 -0.2976]
 [ 0.0819 -0.2976  0.9993]]

Missing values

• Suppose we have a Gaussian model for $\mathbf{z}$

$$\mathbf{z}\sim \mathcal{N}(\mathit{\mu },\mathbf{\Sigma })$$

• but some elements of $\mathbf{z}$ are not observed, i.e. we observe a vector ${\mathbf{z}}_1$

• for example

$$\begin{array}{r}\mathbf{z}=\left[\begin{array}{c}{z}_1\\ z_2\\ ⋮\\ z_{k-1}\\ z_k\\ z_{k+1}\\ ⋮\\ z_{T-2}\\ z_{T-1}\\ z_T\end{array}\right],{\mathbf{z}}_1=\left[\begin{array}{c}\ast \\ z_2\\ ⋮\\ z_{k-1}\\ \ast \\ z_{k+1}\\ ⋮\\ z_{T-2}\\ \ast \\ \ast \end{array}\right]\end{array}$$

Some examples

• mixed frequency

  • (univariate) GDP at annual, GDP at quarterly

  • (multivariate) GDP at annual, inflation at monthly

  • etc.

• forecasting

• backcasting

• unobserved (latent) variables

  • state of the economy

  • natural rates (interest, unemployment)

  • economic shocks

observed elements ${\mathbf{z}}_1$, unobserved elements ${\mathbf{z}}_2$

• ${\mathbf{z}}_1,{\mathbf{z}}_2$ - jointly Gaussian

• marginal distribution of ${\mathbf{z}}_1$

$$p({\mathbf{z}}_1)\sim \mathcal{N}({\mathit{\mu }}_1,{\mathbf{\Sigma }}_1)$$

example:

$$\begin{array}{r}\mathbf{z}=\left[\begin{array}{c}{z}_1\\ z_2\\ z_3\end{array}\right],{\mathbf{z}}_1=\left[\begin{array}{c}{z}_1\\ z_3\end{array}\right],{\mathbf{z}}_2=\left[\begin{array}{c}{z}_2\end{array}\right]\end{array}$$

$$\begin{array}{r}{\mathbf{z}}_1=\left[\begin{array}{ccc}1& 0& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{z}_1\\ z_2\\ z_3\end{array}\right]={\mathit{B}}_1\mathbf{z}\end{array}$$

$$\begin{array}{r}{\mathbf{z}}_2=\left[\begin{array}{ccc}0& 1& 0\end{array}\right]\left[\begin{array}{c}{z}_1\\ z_2\\ z_3\end{array}\right]={\mathit{B}}_2\mathbf{z}\end{array}$$

• conditional distribution of ${\mathbf{z}}_2$ given ${\mathbf{z}}_1$

$$p({\mathbf{z}}_2|{\mathbf{z}}_1)=\mathcal{N}(\underset{\mathrm{E}({\mathbf{z}}_2|{\mathbf{z}}_1)}{\underbrace{{\mathit{\mu }}_2+{\mathbf{\Sigma }}_{21}{\mathbf{\Sigma }}_{11}^{-1}({\mathbf{z}}_1-{\mathit{\mu }}_1)}},\underset{\mathrm{cov}({\mathbf{z}}_2|{\mathbf{z}}_1)}{\underbrace{{\mathbf{\Sigma }}_{22}-{\mathbf{\Sigma }}_{21}{\mathbf{\Sigma }}_{11}^{-1}{\mathbf{\Sigma }}_{12}}})$$

if ${\mathbf{z}}_1$ is the past and ${\mathbf{z}}_2$ is the future,

• $\mathrm{E}({\mathbf{z}}_2|{\mathbf{z}}_1)$ - optimal forecast of ${\mathbf{z}}_2$ given ${\mathbf{z}}_1$

• $\mathrm{cov}({\mathbf{z}}_2|{\mathbf{z}}_1)$ - variance of the optimal forecast

where optimality is in the sense of minimizing the MSE

in general, $\mathrm{E}({\mathbf{z}}_2|{\mathbf{z}}_1)$ is our best guess of ${\mathbf{z}}_2$ given ${\mathbf{z}}_1$, and $\mathrm{cov}({\mathbf{z}}_2|{\mathbf{z}}_1)$ is the associated uncertainty

previous

Multivariate normal distribution

next

Review of maximum likelihood estimation