« CTDATA Servers Temporarily Knocked Off-Line by Massive Internet Server Attack | Main | OnLAMP.com Publishes Good Description of SQL Slammer Impact on Some Cisco Routers »

Yesterday's Internet Server Attack Exposes Less Obvious Infrastructure Weaknesses

Dave Aiello wrote, "Yesterday's massive denial of service attack, while aimed at Microsoft SQL Server 2000 servers, exposed a lot of other holes in infrastructure, and lacks of redundancy or robustness. I want to cite a few examples from CTDATA's infrastructure because I think they will be illustrative:"

  1. Lack of meaningful DNS diversity: At the time of the outage, CTDATA's servers had primary and secondary DNS servers located in the same colocation facility. This is a bad idea because yesterday showed that all of the routes from any one facility to the Internet may be overwhelmed with traffic simultaneously, even if they go through different ISPs.

  2. Lack of local mail relays for critical network services: The network monitoring service that we run does not have an SMTP server on the same subnet. This means that we depend upon one of the SMTP servers that we are attempting to monitor to email our outage alerts to us.

    This also became an issue for our firewalls, because they mail their logs to administrators as they fill up. When huge amounts of traffic hit the firewalls, many events were logged, filling up the memory quickly. Those logs could not be emailed because of the network failure. So, we probably lost a good amount of information about the attack as it was occuring.

Dave Aiello continued, "We knew about these infrastructure issues, but haven't been able to deal with them expeditiously because they require more server resources than we have available and can afford at the moment."

"Although our firewalls prevented the attack from reaching our servers, we still experienced total loss of connectivity for about 10 hours. The connectivity loss is attributable to routers at ISPs upstream from our servers. Those routers simply went down when massive amounts of traffic hit them. When CTDATA's servers came back on-line, I received over 700 email messages within an hour, mostly from servers that had the ability to queue their error and alert message in memory until the email servers came back on-line."

"I object to articles like Massive Internet Outage was Preventable from the UPI because it gives people the impression that attacks like these are predictable, easy-to-understand, have straightforward solutions, and only have obvious side effects. Nothing could be further from truth."

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)


CTDATA Venutures (CTDATA) develops Internet and Intranet applications for corporations and non profit organizations. Our services include:

  • Consulting services for Movable Type and TypePad-based publishing systems (visit our Weblog Improvement website for more information),
  • Financial services business process consulting,
  • Content management system and knowledge management system consulting,
  • Apache web server engineering and hosting,
  • MySQL, Sybase, and Microsoft SQL Server architecture and development,
  • SOAP, REST, and XML-RPC system architecture and programming, including Amazon Web Services and
  • Weblog publishing.
For more information, contact Dave Aiello by email at dave [at] daveaiello.com or call him at +1-267-352-4420.
Copyright © 1995-2010, CTDATA Ventures. All Rights Reserved.
Powered by
Movable Type 4.25