Recent Outage: Network Changes Coming

**Karl** · 07-20-2008, 10:57 AM

The brief issue yesterday and network issues earlier today has been with a very large DDoS. After yesterday, we had thought the issue was resolved after the IP being attacked was null-routed, but today things resumed again on several different IPs for the same customer and we have now been forced to ask that customer to leave. The attack today was of much greater magnitude and was many many millions of packets per second. The sheer number of packets was simply overloading the routers, causing them to drop BGP sessions, etc.

Part of the problem is that right now the network is a bit overly complex, partially because of the fact we're operating two separate networks, performance and standard, which had forced this large number of packets through the same router multiple times. We are currently in the process of making major network changes to simplify the network, which will then reduce the effects of these types of attacks and also speed up the time it takes to resolve them. These changes involve us going to a single network product and increasing the number of and functionality of our core Cisco switches/routers. This change is to be completed in October, but we will be continually making progress towards that point up until then.

**Justec** · 07-20-2008, 07:36 PM

Yeah, its been on and off thorugh the day today. It was just out for about an hour just now ~7.15 - 8.30PM EST. Luckily it was a Sunday so not really a big deal.

**Karl** · 07-20-2008, 07:43 PM

Yes, this latest event was a DDoS targeted directly at our core networking equipment. The attack profile was similar to the previous attack, but was significantly larger, making working on the core equipment nearly impossible.

We will be stepping up the planned network maintenances/changes and will try to accomplish the changes by the end of the month. There will be further announcements made with scheduled maintenance periods, etc.

As a note, we are also re-vamping our entire out-of-band system, so that hopefully at least basic support services, etc. remain online during such issues.

**phaselight** · 07-20-2008, 08:40 PM

Since you are revamping the support system and creating a status page (according to the post in WHT), may I suggest a section of the forum accessible only to clients for discussions of outages, status notification, and changes planned? Discussing it in WHT not only gives credit to the attackers but also makes it easier for them to know what changes are being made and how to circumvent them. Besides, many of us don't go to WHT first to find out what's going on.

Originally Posted by Karl

Yes, this latest event was a DDoS targeted directly at our core networking equipment. The attack profile was similar to the previous attack, but was significantly larger, making working on the core equipment nearly impossible.

We will be stepping up the planned network maintenances/changes and will try to accomplish the changes by the end of the month. There will be further announcements made with scheduled maintenance periods, etc.

As a note, we are also re-vamping our entire out-of-band system, so that hopefully at least basic support services, etc. remain online during such issues.

**Karl** · 07-20-2008, 11:09 PM

The plan, right now, is to use this forum for such discussion. The basic details, etc. are all available on our site anyway, and we will not go into the exact specifics, for this very reason, even to clients.

Right now, the plan is to have a new status page setup along with various support services being put on an out-of-band configuration this week. We then hope to have both networks combined into a unified network, which will simplify the network and remove the bottleneck we currently have with the Juniper routers around the end of this month. The network changes had been planned to be finished for October 1, but it seems we're being forced to proceed faster than we had initially planned.

**sbeasley** · 07-21-2008, 08:59 AM

So quick question... how did the RioRey DDOS boxes not stop this?

Thanks!
-Steven

**Karl** · 07-21-2008, 09:53 AM

Too many packets per second for the RioReys to handle, which should then give you an idea as to the scope of the attack we were facing. The issue was the number of packets and core processing it required to handle the packets as we were receiving an extremely high number of very small packets.

Part of the plan is to upgrade to 10 GigE RioRey devices, which will be able to handle a higher number of packets per second. That had initially been planned to be implemented in October, but we will now be upgrading as soon as we can get our hands on one.

**Josmul123** · 07-21-2008, 11:02 AM

Last time a major network outage occurred and we were unable to connect to your support system, we were told that your PBX system was being moved onto another network so that it would be unaffected by core network crashes.

Has this not happened, or was your phone system just overloaded with the many people who were probably calling to figure out where their servers went? Even your fax line was down, so that would indicate the former. Is there a plan to move support off your core network? It makes little sense to leave the only two ways of getting a hold of Steadfast on the network that's down.

As for the customer who claimed it wasn't a big deal because it was Sunday night, I'd beg to differ. Sunday night is when many of our clients use their web applications, and it is a big deal when it goes down. It's always mid-afternoon somewhere in the world.