Assessment of ASX Clearing and Settlement Facilities 3. Operational Incident Management

3.1 Background

ASX experienced a number of operational incidents in the assessment period. The most serious of these occurred in the week of 16 November 2020:

• ASX detected that some orders on the ASX market were not being reported correctly following the opening of trading on 16 November. This was the first day of trading following a major upgrade to ASX's equity trading platform, ASX Trade. ASX took the decision to close the market for the remainder of the day while it investigated the problem. The issue was caused by problems associated with Tailor Made Combinations (TMCs), an order type that allows trades across multiple securities and derivatives to be executed simultaneously. The market reopened the next day with TMC functionality disabled, although this functionality was partially re-enabled in December.
• ASX closed its Centre Point order matching service for close to a week after experiencing two issues on 16 and 17 November. The first issue affected ASX's processing of the price feed from Chi-X Australia Pty Ltd (Chi-X) that ASX and Chi-X participants rely on in order to ensure that they are meeting best execution requirements under ASIC's market integrity rules. The second issue resulted in certain Centre Point orders interfering with the ability of participants to cancel ASX orders in bulk.
• On 17 November, the completion of the CHESS settlement batch was delayed from around 1:00 pm to 5:40 pm after an unscheduled database process caused settlement to freeze prior to the completion of the delivery-versus-payment (DvP) settlement process. The CHESS end-of-day processing of trades commenced at 8:00 pm, one hour later than usual.

ASIC and the Bank maintained close engagement with ASX throughout the series of incidents experienced in the week of 16 November and in the weeks that followed. While the most significant disruptions during the week impacted ASX's trading operations rather than the clearing and settlement facilities, the Bank has taken a close interest in all of the operational disruptions. ASX takes a group-wide approach to operational risk and project management, which means operational issues affecting ASX's trading platform can have implications for its clearing and settlement operations. Accordingly, the Bank has engaged with ASIC and ASX to fully understand the underlying causes of these events, as well as whether there are any underlying systematic issues that contributed to the incidents.^[19]

3.2 Review of ASX Trade Refresh project

Following the outage on 16 November, ASX commissioned IBM to undertake an independent expert review of the ASX Trade Refresh project. The review examined:

• if it was reasonable to expect the new trading platform was ready for succesful production implementation and ongoing availability
• whether the project had sufficent resources
• the efficacy of the change control process
• the robustness and rigour of risk and issue management
• whether it was reasonable to expect the project test plan to be effective, commensurate with the risk appetite and criticality of the ASX Trade system
• the implications of the project on stakeholders
• whether ASX took into consideration the lessons learnt from managing a previous trading system incident in 2016
• whether the project met industry standards, frameworks or practices.

Overall, IBM found that ASX met or exceeded leading industry practices in around three-quarters of the capabilities that IBM assessed. ASX was found to have provided the project with sufficient staff, finance, time and technological resources, and communicated appropriately with key stakeholders. IBM also found that ASX's actions in managing the outage on 16 November were appropriate and reflected the lessons learnt from the 2016 incident.

However, IBM identified a number of key shortcomings in the ASX Trade Refresh project, including that:

• there were gaps in the rigour applied to the way that risks and issues were identified and managed and the project would have benefitted from more input from independent parties
• it was not reasonable to expect that the test plan used by ASX would meet its stated near-zero appetite for service disruption for a systemically important national infrastructure as stated in the ASX test policy
• IBM raised concerns about the composition of the primary governance function and that this governance function was also responsible for several other projects, diluting its attention on the ASX Trade Refresh project
• a number of factors suggested the platform was not ready for go live considering ASX's near-zero appetite for service disruption, including that ASX had identified the need for additional testing.

IBM made recommendations in seven key categories: risk, governance, delivery, requirements, vendor management, testing and incident management.

ASIC and the Bank published a summary of the findings from the IBM review in August.^[20] ASX has agreed to address the recommendations from the review and has provided ASIC and the Bank with its high-level response. ASIC and the Bank expect ASX to apply the insights from IBM's findings across the ASX Group to ensure existing and proposed projects, including the CHESS replacement program, are managed and implemented appropriately.

3.3 Impact on settlement from CHESS batch delays

ASX was unable to identify the root cause of the 17 November CHESS settlement batch incident and has not been able to replicate the unscheduled database process that led to the delay. However, ASX has developed a procedure that will allow it to terminate the database process if it were to reoccur, allowing ASX to resolve the issue more quickly. This mitigates the risk posed by the specific events that led to the delay in the CHESS settlement batch on 17 November.

Nonetheless, the incident highlighted circumstances in which an operational incident may disrupt the DvP settlement process that is used to mitigate the risk that securities are delivered without payment being received, or vice versa. The DvP process used in CHESS is designed so that securities are transferred within CHESS if and only if the corresponding cash payments are made in RITS. In order to achieve this, the securities are locked in CHESS until confirmation is received that the cash leg has settled in RITS, at which point the securities are released to the accounts of the purchasing participants. This process typically takes around 15 minutes, but it was during this time that the settlement process froze on 17 November, meaning that the payment leg completed but the movement of securities did not.

If ASX had not been able to resolve the issue on the day, its options would have been to: 1) defer settlement of securities to the following day; or 2) seek to reverse the payments that had already settled and then reschedule settlement of both legs to the following day. However, it is unlikely that the latter option would have been feasible as it relies on the voluntary cooperation of payment providers that settle CHESS-related payments in RITS on behalf of participants.

Under the first option, ASX would utilise emergency powers in the ASX Settlement Operating Rules (ASXSORs) to delay the movement of securities into the accounts of purchasing participants to a subsequent day. During this period, the securities would remain locked in order to guarantee that they are available for settlement. The Bank and ASIC are engaging with ASX to determine whether there are any issues affecting the legal certainty of the DvP process in the event of an operational disruption.

ASX has reflected on lessons learned from its management of the incident and engaged with CHESS users to better understand the effects of service disruptions and to improve communications if they occur. It has provided greater clarity to participants and payment providers on the operational and communication processes for CHESS batch settlement, including actions it may take in the event of a contingency affecting the CHESS batch.

3.4 Conclusions and recommendations

While the CS facilities were not responsible for the incidents affecting ASX trading systems, ASX applies a group-wide approach to operational risk and project management, so there is a risk that the shortcomings identified by IBM in the ASX Trade Refresh project could affect the CS facilities if not addressed. Accordingly, the Bank will work closely with ASIC and ASX to ensure that any relevant findings or recommendations from the IBM review are applied to its clearing and settlement operations, and in particular to the CHESS replacement program.

The Bank and ASIC consider it important for ASX to complete analysis to confirm the legal certainty of arrangements to ensure that settlement of securities in CHESS takes place if and only if the corresponding cash movement has taken place in RITS.

Footnotes

Other operational incidents experienced during the assessment period included problems associated with the launch of a new website in October, and an incident in December which resulted in a small number of equities derivatives positions being incorrectly recorded in ASX's Derivatives Clearing System. [19]

For ASIC's media release see: https://asic.gov.au/about-asic/news-centre/find-a-media-release/2021-releases/21-220mr-update-on-the-independent-expert-review-of-november-s-asx-trade-outage/. For the Bank's media release see: Update on the Independent Expert Review of November's ASX Trade Outage, 23 August 2021. ASX also issued a media release that includes a summary of IBM's recommendations: https://www2.asx.com.au/content/dam/asx/about/media-releases/2021/independent-review-of-asx-trade-outage-november-2020.pdf. [20]