RDP 9708: Measuring Traded Market Risk: Value-at-risk and Backtesting Techniques 4. Conclusion

Despite their weaknesses, the tests employed in this paper provide some useful diagnostics for evaluating VaR model performance. The exceptions-based testing identifies portfolios B, D and E as areas where the VaR model is not adequately covering realised losses. In the case of portfolio D, this assessment is supported by the results of the variance-based test. These weaknesses in the VaR model can, to a large extent, be attributed to the observed non-normality of profit and loss. Given that the risk-tracking performance of the model is not strong, there does appear to be considerable scope for the model to be further refined.

The use of VaR models has spread remarkably rapidly across the financial industry over the past six or so years. As the techniques have become more widespread, the range of methods for calculating VaR estimates has broadened. That said, the use of VaR models are at comparatively early stages of development for many Australian banks. The need for further refinement of newly developed models on the part of banks, and the need for regulators to assess the performance of such models, suggests that backtesting should play an important role in furthering the evolution of VaR techniques.

Given that our sample bank was just starting the process of implementing and refining its VaR model at the time when the data were collected, it would be expected that the model might not perform well. Over the past year, many banks that use VaR measures as part of their risk-management process (including the sample bank discussed here) have significantly upgraded their VaR model; for example, by improving the quality of the historical price data underlying the calculation, more accurately modelling the price sensitivities of individual instruments or by moving from a variance-covariance approach to a Monte-Carlo simulation approach. Hence, the test results shown here should not be taken as describing the performance of Australian banks' models more broadly.

The tests that have been applied in this paper are not sufficiently precise to form the basis of regulatory treatment of banks' VaR estimates in the way that the Basle Committee defines its three zones. Notwithstanding their shortcomings, these tests do appear to provide some useful information to both risk managers within banks and bank supervisors. The tests may aid in identifying those portfolios where VaR performs poorly, and may provide some insight into why a VaR model may not capture the risks of particular portfolios well, thus guiding the process of improving the analysis and modelling of trading risks.