This is a follow-up on the partial disruption of service we experienced on our HSRP net between 08:30-08:50 PST on Friday, 8/26/2016.
The event that caused the disruption in service occurred “upstream” from the ASAP network at one of the core internet service providers. For some background, the ASAP network infrastructure is supported by a redundant HSRP circuit, which is backed by multiple ISPs. The ISPs supporting our Internet connectivity are Level 3, ATT, Verizon, and WV Fiber. Traffic to/from the Internet is load balanced between these 4 providers, and in the event of an catastrophic failure on one of these circuits, our traffic will continue to be supported by the remaining ISPs.
On the morning of the event, another client using the same HSRP net as ASAP was the target of a “very large” DDOS attack. During an attack of this kind, large amounts of traffic are sent to a specific target with the intent of overwhelming the target networking nodes and saturating the Internet pipes to/from the target. The infrastructure at SWITCH (the hosting facility where our servers are located) is fully capable of handling such an attack, but in this case, one of the providers was not. The ISP’s systems were overwhelmed, and the result was a partial disruption in the traffic being passed to SWITCH and ultimately to ASAP.
The reaction on the SWITCH side, once the attack was discovered, was to stop accepting any traffic from the affected provider and to effectively only accept traffic from the remaining 3 vendors. The delta for this process to occur was approximately 20 minutes. During that time access to the ASAP network and applications were partially affected. The internal ASAP infrastructure was not a part of this event.
We know any disruption to service has a major impact on your organization as well as an effect on confidence in ASAP, which is why I wanted to to post a quick note to provide you with details of the event.
Please do not hesitate to reach out if you have any questions.