Time series regression

Introduction

• $\beta_d$: parameter for predictor no.d
• $z_{d,n}$: predictors no.d at time n
  • one $z$ can be $y$, another $z$ can be $y^2$
• $x_n$: x value at time n, to be predicted
• $w_n$: white noise at time n
• Basic form
  • $Xt = \beta1z{1,t}+\beta2z{2,t}+…+\betadz{d,t}+Wt$
  • Predict $X$ at time t, based on factors $z1, z2,…,z_d$ at time $t$
  • $W_t$ is gaussian white noise series
• Simple linear regression model
  • $Xt = \beta0+\beta1zt+W_t$
  • ‘simple’ because there is only one predictor
  • ‘linear’ because COEFFICIENTS are linear
• Multiple linear regression model
  • $Xt = \beta1z{1,t}+\beta2z{2,t}+…+\betadz{d,t}+Wt$

Parameter estimation

• Simple linear regression model
  • SSE = $\sum{t=1}^n(xt-{(\beta0+\beta1z_t)})^2$
  • $\hat{\beta}1$ = $\Large\frac{\sum{t=1}^n(zt-\bar{z})(xt-\bar{x})}{\sum{t=1}^n(zt-\bar(z))^2}$
  • $\hat{\beta0} = \bar(x)-\hat{\beta1}\bar{z}$
• Multiple linear regression model
  • model:
  • $x=Z\beta+w$
  • SSE:
  • SSE = $(x-Z\beta)^T(x-Z\beta)$
  • $\hat\beta$:
  • $\hat\beta = (Z^TZ)^{-1}Z^Tx$
  • Where:
  • $x = [x1, x2,…,x_n]^T$
  • $w = [w1, w2,…,w_n]^T$
  • $\beta = [\beta1,\beta2,…,\beta_d]^T$
  • $Z = \begin{pmatrix} z{1,1} & z{2,1} & \cdots & z{d,1} \ z{1,2} & z{2,2} & \cdots & z{d,2} \ \vdots & \vdots & \ddots & \vdots \ z{1,n} & z{2,n} & \cdots & z_{d,n} \end{pmatrix}$

Model selection

• Approaches:
  • Forward selection
  • Backward elimination
  • Cross-validation
  • Divide data into train and validation sets, iteratively train models on training set, select the model with the best performance on validtaion set
  • Regulation methods
• Metrics:
  • Adjusted $R^2$
  • The higher the better
  • AIC
  • AIC = $ \log(\frac{SSE}{n}) + \frac{2d}{n}$
  • Measures the distance between current constructued model with best model you can construct so it’s good and the lower the better
  • BIC