As with any technology system, the SIP has finite capacity. The SIP’s governing committee plans for capacity based on the needs of its member exchanges, adding an appropriate cushion of extra capacity to handle heavier than expected volume. In January 2013, a regularly scheduled systems capacity test showed the SIP system was capable of handling approximately 500,000 messages per second across 50 of the SIP system’s ports, for an average peak of approximately 10,000 messages per-port, per second. The tests are conducted in a controlled environment – performance in real-life scenarios will typically be lower.

On the morning of August 22, however, a sequence of events combined to create an unprecedented volume of message traffic into the SIP; well beyond the system’s tested capacity of 10,000 messages per-port, per second.

On August 22, the SIP received more than 20 connect and disconnect sequences from NYSE Arca, each of which consumed significant resources. Available capacity was further eroded as the SIP received a stream of quotes for inaccurate symbols from NYSE Arca, and generated quote rejects. Both of these actions served to degrade the system below the tested capacity of 10,000 messages per per-port, per second. During this period, NYSE Arca sent multiple bursts with each connect and disconnect, topping more than 26,000 quote updates per-port, per second as it attempted to reconnect. By comparison, a typical August day for NYSE Arca would peak at less than 1000 messages per-port, per second. The events of August 22 were 26 times greater than the average per-port, per-second activity.

The confluence of these events vastly exceeded the SIP’s planned capacity, which caused its failure and then revealed a latent flaw in the SIP’s software code.

This latent flaw prevented the system’s built-in redundancy capabilities from failing over cleanly, and delayed the return of system messages. The combination of large system inputs and delayed outputs ultimately degraded the ability of the SIP system to process quotes to an extent that a shutdown of the system was in the broader public interest, to prevent information asymmetry and ensure fair conditions for all market participants.

Although the problem was quickly identified and data feeds were operational within 30 minutes of the halt, additional time was required for NASDAQ to consult UTP SIP committee members and market participants, and to test and evaluate scenarios to re-open the market to ensure that trading could be resumed in a fair and orderly manner. As soon as that process was completed, connectivity to the SIP was made available to UTP participants. The markets reopened without incident.

via NASDAQ, N.Y. Times Dealbook, Economist


Nanex Image

Digging into market data before the Nasdaq blackout at 12:20 EDT on August 22, 2013, we came across several significant periods of extremely high quote bursts. …[T]he 3 quote bursts at 11:48, 11:50 and 11:54 – which were just minutes before Nasdaq decided to shut down the SIP – were each comprised of a loop of the last 50 minutes worth of quotes. Basically, the previous 50 minutes worth of quotes in hundreds of stocks were blasted, with fresh new timestamps, to millions of subscribers, over a 3 second period. …

via The Big Picture, Nanex