RDP 2015-12: Modelling the Australian Dollar Appendix A: Out-of-sample Forecasting

In assessing the models' out-of-sample performance, actual realised values of the explanatory variables are used to construct forecasts of the RTWI at different horizons. Test statistics are then constructed by comparing these forecasts to the realised values of the RTWI.

Consistent with a large portion of the literature, the structural models are compared to a naïve forecast model, namely a random walk. This is done using the DM (Diebold and Mariano 1995; West 1996) and CW statistics (Clark and West 2006, 2007), both of which assess models based on their mean squared forecast errors (MSFE).

The CW statistic is widely used in assessing the out-of-sample performance of nested models.^[47] The CW statistic compares the MSFE of the two models but, unlike a number of other test statistics, it accounts for a bias in the MSFE that arises when comparing nested models. The intuition behind this adjustment is that, if the true data-generating process is a random walk, the structural model is over-fitted, which can reduce its forecast accuracy and lead to a higher MSFE. This issue would be ameliorated if the sample was sufficiently large, as the estimate of the structural model should approach the true random walk model. However, in most cases the sample will not be sufficiently large and the small-sample bias will remain (Clark and West 2006).

The CW statistic therefore compares the MSFE of the random walk model to the adjusted MFSE of the structural model.^[48] If the forecasts tend not to be biased, the null hypothesis is that the two models have equivalent forecast performance and the test can be considered to be a minimum MSFE test.^[49] A CW statistic of zero would then indicate equivalent forecast performance, while a CW statistic above zero would indicate that the structural model's forecasts are ‘better’ than those from the random walk model (Rogoff and Stavrakeva 2008). However, if the forecasts are biased, the null hypothesis is that the exchange rate is a random walk and the test can no longer be considered a test of minimum MSFE (Rogoff and Stavrakeva 2008).

Comparing the MSFE of the random walk to that of the structural models remains a valid question even if the ‘true’ model is something other than a random walk. For this reason the DM statistic is also considered, as it compares the ‘raw’ – or unadjusted – root MSFE from the structural model directly to that of the naïve random walk model. A DM statistic greater than zero (less than zero) indicates that the structural model has a lower (higher) MSFE and so produces superior (inferior) forecasts, compared to the random walk model.

While both the CW and DM statistics have standard normal asymptotic distributions, bootstrapped distributions are also constructed given the small sample size. Distributions for the CW and DM statistics are constructed using a semi-parametric residual bootstrapping technique. The technique closely follows that of Mark and Sul (2001), which is also employed in a number of other papers, including Rogoff and Stavrakeva (2008). The p-values for both the CW and DM statistics are defined as the proportion of the distribution above the ‘observed’ statistic.

The out-of-sample assessment is conducted on the baseline model, the four decomposed ToT specifications, the two I/GDP models, and the two models that include short- and long-term RIRD.^[50] Rolling windows are used, consistent with much of the literature, with a window length of 70 quarters.^[51] The statistics are calculated for forecast horizons of one, four and sixteen quarters, and are reported in Tables A1 and A2.

One drawback of this bootstrapping technique, and relatedly the method for calculating the forecast statistics, is that it involves estimating the cointegrating relationship over the full sample period. As a result, information from the full sample is used in constructing the forecasts, which could provide the structural models with an ‘unfair’ advantage. Therefore, as a robustness check, bootstrapped distributions are also constructed using a fairly standard residual bootstrap, and carried out under the null hypothesis of no predictability.^[52] This approach involves estimating the cointegrating relationship over rolling windows, rather than over the full sample, when constructing the forecast statistics and their bootstrapped distributions. Alquist and Chinn (2008) contend that estimating the cointegrating relationship over rolling windows should ensure that the forecasts are true ex ante predictions and should make the hypothesis tests more stringent.

At both the one- and four-quarter horizons, the results of this exercise are broadly similar to those obtained when estimating the cointegrating relationship over the full sample (Tables A3 and A4). In contrast, the models' forecast performance at the sixteen-quarter horizon appears slightly better when examined using the rolling-window bootstrap methodology. However, it is difficult to draw any strong conclusions given that the results differ based on both the choice of model and test statistic.

Footnotes

A model is nested in another model if it can be seen as a special case of the more general model (i.e. if it can be obtained by applying restrictions to the parameters of the general model). [47]

Clark and West (2007) extend this to the case where the nested model is not a random walk. [48]

‘Bias’ refers to scale bias, not location bias. See Rogoff and Stavrakeva (2008) for details. [49]

Out-of-sample testing could not be carried out on the FToT model due to the short sample. [50]

As a robustness check, the out-of-sample forecast testing was also carried out using recursive regressions. The results were broadly similar and are therefore not reported. [51]

This method does not impose cointegration. Therefore, it could also be useful in identifying whether the imposition of cointegration as part of the first bootstrapping methodology leads to a bias in the results. [52]