RDP 8609: The Performance of Exchange Rate Forecasts 4. Forecast Performance of the Group Plan and $A/us$ Forecasts

Table 2 reports the mean absolute errors and the mean square errors for the group mean and median forecasts and for the six benchmark forecasting models.

Comparison of the relative forecasting performance of the group mean and the median forecasts shows the mean forecast to be slightly superior to the latter, having both a smaller MAE and MSE. This superiority of the mean forecast over the median is consistent with recent studies of the forecasting accuracy of US forecasters (see Zarnowitz (1982b) and Hafer (1984)).

This superiority stems from the fact that the mean value gives some weight to all pieces of information, while the median may ignore certain pieces of information. For example, suppose a forecaster in the foreign exchange market gains last minute sole access to a piece of information which suggests that the exchange rate is likely to fall significantly over the next week. As a result he predicts a sharper fall in the exchange rate than he was previously predicting. This lower prediction will reduce the value of the mean forecast, but may well leave the median forecast unchanged. Since the mean makes more extensive use of such information, it is not surprising that the mean forecast series proves to be superior to that of the median.

Given the superior performance of the group mean forecast it, rather than the group median, will be used to compare the performance of the AFR survey forecasts with that of the benchmark forecasts.

A comparison of the MAE figures in Table 2 shows the group mean AFR survey forecast to have been more accurate than any of the six benchmark forecasting models. All the benchmark models produced a mean absolute error of greater than one US cent while the group mean forecast produced a mean absolute error of only 0.9838 US cents. Of the six benchmark models the no change model and the restricted time series model produced the most accurate forecasts.

Given that the optimised time series model is less restrictive than the restricted model, its inferior performance is perhaps a little surprising. The explanation, however, lies in the fact that the models were estimated using “out of sample” data. The inferior performance of the more general model suggests that the structure of the time series of exchange rates changed through time. That is, the relationship between the exchange rate and its lagged values was different over the period December 1983 to March 1985 to that over the period March 1985 to December 1985. Had the forecasting equations been estimated using “in sample” data the general model would have out-performed the one lag model (and the no change model). However, as discussed in Section 3 the use of “in sample” data is not appropriate in estimating benchmark forecasting equations, because these data were not available to market participants at the time they prepared their forecasts.

While the group mean forecast series performed better than all the benchmark forecasting equations using the MAE criterion, it proved to be superior to only three out of the six benchmark forecasts using the MSE criterion. The no change model, the restricted time series model and the interest differentials model all produced forecasts with a lower MSE than the mean AFR survey forecast. The optimised time series model and the extrapolative expectations model again proved to be the worst performers.

The third criterion used to evaluate the performance of the various forecasts is the percentage of forecasts predicting the correct direction of movement. Table 3 reports these percentages for the group mean and median forecasts and for the benchmark forecasting models. The table shows that the group mean forecast produced the highest percentage of forecasts in the correct direction (65.9%). Of the benchmark models the extrapolative model was the only one to produce forecasts in the correct direction on more than 50 per cent of the 41 weeks. The poorest performer on this criterion was the optimised time series model.

The fact that approximately two thirds of the group mean forecasts predicted the correct direction of the hedge settlement rate movement suggests that movements in the rate differ from a pure random walk. The result indicates that the market often had some information regarding the direction of movement of the spot rate. This information, however, seems to consist of little more than the previous movement in the spot rate, as the extrapolative model performs almost as well as the AFR survey forecasts. Such a result would be consistent with a predominant use of charting methods in the formation of expectations in the foreign exchange market.

The above comparisons of the performance of the mean AFR survey forecast series with the benchmark forecasts, suggests that on average the mean AFR survey forecast provided a slightly better forecast of the hedge settlement rate on the following Wednesday than did any of the benchmark forecasts. However, the MSE results suggest that on a number of occasions when the exchange rate moved significantly, the mean forecast not only failed to pick the size of the movement but also failed to pick the direction of the movement.

Any conclusion, therefore, regarding the overall performance of the mean of the AFR survey forecasts must rest on an assessment of the costs of making occasional large forecast errors. The occasional large errors are likely to be of major concern if transactions dependent on the forecasts are executed relatively infrequently. If positions are taken or altered continuously on the basis of the forecasts, the effect of the occasional large error may well be offset by the consistently relatively good forecasts. If, however, such transactions are undertaken infrequently, forecasts which exhibit a relatively low mean absolute error but relatively high mean square error may well prove to be inferior to forecasts with a relatively high mean absolute error and low mean square error. This is particularly the case for highly risk averse transactors.