" /> CTDATA: January 2003 Archives

« December 2002 | Main | February 2003 »

January 30, 2003

Thousands of NJ Doctors Expected to Participate in Job Action on February 3

The New York Times reports that a job action being organized by the Medical Society of New Jersey to protest soaring malpractice insurance rates is expected to involve 5,000 to 10,000 of the 22,000 physicians practicing in the state. The job action will take place on Monday, February 3, 2003. According to the article, it will involve doctors who practice in offices cancelling their non-emergency appointments.

Ice Breakers on the Hudson

The Star-Ledger reported today that ice breaking tug boats are being employed to keep ferries moving across the Hudson River. This hasn't been necessary in years.

In spite of the use of ice breakers, the only dependable ferry services are running from Weehawken, Hoboken, and Jersey City. The longer ferry services, from the Highlands, Belford, and Bayshore, have not been able to operate for some time.

January 28, 2003

Red Hat Explains New End of Life Policies for Its Mainstream Linux Distributions

Newsforge published a statement from Jeremy Hogan of Red Hat that explains Red Hat's recently announced End of Life Policy for their core Linux distributions. According to the article:

...As we worked on Red Hat Linux 8.0 we realized that Red Hat Linux's lifecycle no longer made much sense. This offering is increasingly aimed at providing easy access to leading edge open source technologies, which by definition evolve extremely rapidly. Our ability to support these rapidly changing projects for long periods of time is quite limited, and we wanted to provide realistic promises for both the level of support and the time period we can offer such support.

We also looked at our customer base and saw how rapidly our older products get upgraded. For example, about two thirds of our Red Hat Linux users currently run Red Hat Linux 7.3 or 8.0, and over 90% run 7.2, 7.3, or 8.0, all of which are less then {sic} a year old. These numbers strongly support our belief that while setting our lifecycle for these products at one year we're inconvienencing some of our users, this is a nonevent for the vast majority of them....

One of the more interesting aspects of this article is the comments that were posted in response. Here are a few examples:

  • "You will probably get a lot of heat over this decision, but it is a good one. RedHat is super easy to upgrade and EOL on RH is not like EOL on Windows...."
  • "People will upgrade willingly if they see substantial value in doing so. They will upgrade grudgingly, or not at all, if told to do so by a vendor (or accept the consequences of lack of support/compatibility)...."
  • "The problem here is that this policy makes RH very unatractive for use in Colo'd machines. I don't want to have to send a tech to visit each machine at remote offices or colo spaces once a year just to throw a CD in the drive, even if the upgrade process is easy...."
  • "So here I am trying to set up clusters and SANs, and the vendors that pay lip service to linux (HP/Compaq), don't even have host bus adapter drivers for 7.3. They only currently support 7.2. They sure as hell don't even mention Advanced Server...."

OnLAMP.com Publishes Good Description of SQL Slammer Impact on Some Cisco Routers

An article by Iljitsch van Beijnum on OnLAMP.com called Network Impact of the MS SQL Worm does a great job of explaining the impact of the SQL Slammer worm on three networks with different Cisco routers.

In van Beijnum's experience, some Cisco routers lost their Border Gateway Protocol (BGP) sessions. When that occurred "the router was unable to advertise the network's IP address ranges to the rest of the world, with the result that these addresses became unreachable." This was compounded by problems with the Cisco Express Forwarding (CEF) algorithm on routers that did not have enough memory. Consequently, the author recommends that packet forwarding algorithms be studied from a worst-case perspective.

Van Beijnum also gives documents some of the steps he took to log and, later, filter the network traffic generated by the worm. So, the article gives insight into how an experienced network manager researches and solves network problems as they occur.

January 26, 2003

Yesterday's Internet Server Attack Exposes Less Obvious Infrastructure Weaknesses

Dave Aiello wrote, "Yesterday's massive denial of service attack, while aimed at Microsoft SQL Server 2000 servers, exposed a lot of other holes in infrastructure, and lacks of redundancy or robustness. I want to cite a few examples from CTDATA's infrastructure because I think they will be illustrative:"


  1. Lack of meaningful DNS diversity: At the time of the outage, CTDATA's servers had primary and secondary DNS servers located in the same colocation facility. This is a bad idea because yesterday showed that all of the routes from any one facility to the Internet may be overwhelmed with traffic simultaneously, even if they go through different ISPs.


  2. Lack of local mail relays for critical network services: The network monitoring service that we run does not have an SMTP server on the same subnet. This means that we depend upon one of the SMTP servers that we are attempting to monitor to email our outage alerts to us.

    This also became an issue for our firewalls, because they mail their logs to administrators as they fill up. When huge amounts of traffic hit the firewalls, many events were logged, filling up the memory quickly. Those logs could not be emailed because of the network failure. So, we probably lost a good amount of information about the attack as it was occuring.


Dave Aiello continued, "We knew about these infrastructure issues, but haven't been able to deal with them expeditiously because they require more server resources than we have available and can afford at the moment."

"Although our firewalls prevented the attack from reaching our servers, we still experienced total loss of connectivity for about 10 hours. The connectivity loss is attributable to routers at ISPs upstream from our servers. Those routers simply went down when massive amounts of traffic hit them. When CTDATA's servers came back on-line, I received over 700 email messages within an hour, mostly from servers that had the ability to queue their error and alert message in memory until the email servers came back on-line."

"I object to articles like Massive Internet Outage was Preventable from the UPI because it gives people the impression that attacks like these are predictable, easy-to-understand, have straightforward solutions, and only have obvious side effects. Nothing could be further from truth."

January 25, 2003

CTDATA Servers Temporarily Knocked Off-Line by Massive Internet Server Attack

Today from 3:00am to 1:00pm Eastern Time, CTDATA's main web and mail servers were off-line due to a routing problem that affected our colocation provider. At this time, we believe that the routing problem was part of the traffic surge associated with the attempted exploit of Microsoft SQL Servers that began at midnight. As Slashdot said:

...UUNet and Internap are being hit very hard. This is the cause of major connectivity problems being experienced worldwide.... This has effectively disabled 5 of the 13 root nameservers.

We apologize for any inconvenience this problem may have caused our customers, and we will update this story in the event that any new information about this incident becomes available.

January 23, 2003

Revision of "Linux Apache Web Server Administration" is Worth Owning

Dave Aiello wrote, "I just picked up the second edition of
Linux Apache Web Server Administration
by Charles Aulds. This is part of the Craig Hunt Linux Library and its a worthy companion to first edition. It basically updates the subject matter of the book so that it is appropriate for Apache 2.0, where the previous version focused on Apache 1.3. It also adds more content about SSL, and new content about management GUIs like Comanche and Webmin."

"It's important to note that the first edition of the book is still a better reference for Apache 1.3 than the new edition. A lot of low level changes have been made in Apache 2.0, and I personally found myself spinning my wheels when I tried to use the new edition while making configuration changes to an Apache 1.3 server."

"I'm setting out to add SSL functionality to an Apache server on my own today. If I'm successful, this will save my client considerable money over commercial alternatives like Apache Stronghold from Red Hat. If I'm successful, the $40 or $50 I spent on the second edition of this book will seem like peanuts."

Signs of Life at LinuxWorld Expo in New York

Dave Aiello wrote, "Derek Vadala notes that business appears to be up at LinuxWorld Expo in New York, taking place this week at The Javits Center. This is very good news."

"In talking to Tony Iams, a scheduled speaker at LinuxWorld, I wondered aloud if it would be worth attending. It turns out that I am too busy to go because one of my clients inserted a project deadline that I am trying to honor. But, based on the lack of pre-show buzz from friends in the industry, and the terrible website that was put together for the show, I concluded that the show was going to be depressing."

"Now it appears I may have come to the wrong conclusion. I'll be looking for more information on attendance and vendor support, and if I see anything else that appears significant, you'll see it here."

XML.com Publishes Great Article on Difficulties of RSS Processing

Mark Pilgrim wrote an excellent article on practical RSS processing for XML.com, called Parsing RSS at All Costs. In it, he gives a sense of the breadth of problems associated with trying to parse headline feeds from many of the weblogs out on the Internet:

...{As} RSS has gained popularity, the quality of RSS feeds has dropped. There are now dozens of versions of hundreds of tools producing RSS feeds. Many have bugs. Few build RSS feeds using XML libraries; most treat it as text, by piecing the feed together with string concatenation, maybe (or maybe not) applying a few manually coded escaping rules, and hoping for the best.

Then he explain how desktop news aggregators are dealing with the situation:

... {Most} desktop news aggregators are now incorporating parse-at-all-costs RSS parsers which they use when XML parsing fails. However, since no one likes tag soup, they are also implementing subtle visual clues, such as smiley and frown icons, to indicate feed quality. Click on the frown face, and the end user can learn that this RSS feed is not well-formed XML. But the program still displays the content of the feed, as best it can, using a parse-at-all-costs parser.

The article goes on to give some code examples of how to deal with these problems using Python.

January 18, 2003

Using REST for Web Services, Instead of SOAP

Amit Asaravala wrote an interesting piece for DevX.com called Giving SOAP a REST. In it, he explains that REST means "Representational State Transfer" and that it is a serious alternative to SOAP ("Simple Object Access Protocol"). Both of these are protocols for web services.

What's the difference between REST and SOAP and why should you care?

REST is more an old philosophy than a new technology. Whereas SOAP looks to jump-start the next phase of Internet development with a host of new specifications, the REST philosophy espouses that the existing principles and protocols of the Web are enough to create robust Web services. This means that developers who understand HTTP and XML can start building Web services right away, without needing any toolkits beyond what they normally use for Internet application development.

The key to the REST methodology is to write Web services using an interface that is already well known and widely used: the URI....

This is a great article that helps to define REST for people who are seeing the term used for the first time. It also provides some high level discussion of the pros and cons of REST versus SOAP, and argues that REST is rapidly becoming a nearly complete alternative to SOAP as a web services protocol.

New York Times Profiles Glenn Reynolds of Instapundit

On Thursday, The New York Times profiled Glenn Reynolds, a law professor at the University of Tennessee at Knoxville who is better known to the world as the author of Instapundit. This article is an interesting look at the life of one of the world's most famous weblog writers.

Is it every webloggers dream to be profiled in The New York Times? Probably not. But, if you are going to spend as much time on your weblog as Glenn Reynolds spends on his, it couldn't hurt.

Why CTDATA Switched from SOAP to XML Over HTTP When Using Amazon Web Services

Dave Aiello wrote, "One of the more interesting aspects of the time I spent with Amazon Web Services was the realization that the SOAP interface provided by Amazon.com doesn't implement as many search methods as the XML over HTTP interface does. I discovered this when I built a Perl script that iterates through Amazon Marketplace offers for a given item in order to get specific details about each offer. This is easily done using REST, but not possible using SOAP."

"Apparently, I am not the only one who discovered this. There is a useful article on CYBAEA called Which interface should I use: XML/HTTP (REST) or SOAP? that comes to the same conclusion:"

SOAP... Does not provide access to the full AWS functionality (specifically the XSLT service is not available through SOAP)....

"The CYBAEA article goes into more detail on the pros and cons of using the AWS SOAP and REST implementations. Even more useful information is found in the CYBAEA Amazon Web Services FAQ, which includes the article I just mentioned. The FAQ is definitely worth exploring if you are using Amazon Web Services for anything more than experimentation."

January 17, 2003

XML::Twig Cleanly Picks Elements Off of XML Documents

Dave Aiello wrote, "I've been working with Amazon Web Services again recently. In the course of doing so, I found that using regular expressions to extract elements (data fields) from XML documents doesn't work reliably. I looked through my copy of
Perl & XML
and saw reference made to a number of XML processing modules. For example: XML::Parser, XML::LibXML, XML::XPath, XML::Writer, XML::SAX, XML::Simple, etc."

"My task was to extract only a few elements from each XML document. Some of the documents contained sets of nodes, so they would have multiple instances of the same node."

"The easiest, most efficient, and most "perlish" way of handling this turned out to be using XML::Twig. I found a really good article about XML::Twig on XML.com. It does the job and provides the kind of TMTOWTDI that experienced Perl users expect from a Perl module."

"Perl and XML is a very good book, but, it's surprising that a module as useful as XML::Twig was left out of it. I'd recommend taking a fresh look at all the Perl XML modules whenever you set out to solve a type of XML problem that's new to you."

January 15, 2003

Analyst Suggests that RIM Should Buy Handspring

Earlier this week, Unstrung reported that Seamus McAteer of the Zelos Group suggested that Research in Motion should acquire Handspring in order to strengthen its presence in the mobile phone-based messaging market. According to the article:

Handspring has both GPRS and CDMA 1xRTT variants of its Treo device. Most of the BlackBerry devices that RIM has sold actually run over pager networks. The company has recently introduced GPRS and iDEN variants of its devices.

{Seamus McAteer said,} "With Handspring, there's a brand there, and it gives RIM a CDMA channel and a relationship with Sprint PCS."

Although RIM has a GSM/GPRS device of its own, McAteer estimated that they had only sold 10,000 such devices.

Keynote: U.S. Mobile Phone Networks Lose 7.5 Percent of SMS Messages

Keynote Systems, a company that analyzes Internet service performance, has come out with its first study of the reliability of Short Message Service messages that are sent and received via mobile telephones. This study attempts to measure the performance of U.S. mobile phone carriers. According to a press release, 7.5 percent of all messages generated for the study never reached their intended destination. This is an astoundingly high percentage, but not outside the realm of possibility in our experience.

CTDATA sends and receives thousands of SMS messages on a monthly basis. We have built an internal application that uses SMS as its primary transport mechanism. We have active users on AT&T Wireless, Cingular, and T-Mobile.

We urge SMS users to read the study and contact their provider if network performance is unacceptable. We will be in touch with T-Mobile about their spotty performance tomorrow.

January 13, 2003

In Memory of Ed O'Donnell

Dave Aiello wrote, "Over on AAHArefs, I posted news that Ed O'Donnell died as a result of a heart attack he sustained while officiating a high school hockey game on Friday in Pennsauken, NJ. He was 45 years old and is survived by a wife and three school-aged children."

"This is the third death of a well known, respected ice hockey official in New Jersey in the past 12 months. Two of the deaths were of men in their mid-40s who were very physically active."

"The article that we posted on AAHArefs was the first, and so far only, article on the web that fully reports the tragedy that befell Ed. We are grateful that a couple of websites that cover New Jersey high school hockey have linked to it. It's important that players, coaches, and fans of the sport learn about this loss."

January 9, 2003

Symantec Develops Tomorrow's Security Monitoring System Today

In one of the more interesting recent articles published in a mainstream U.S. newspaper about the IT industry, The Washington Post has profiled Symantec's managed-security service headquartered in Alexandria, Virginia. According to the article:


The four-year-old operation, which includes special monitoring and "data mining" technology, was created by a local start-up called Riptech. Last year, California-based Symantec paid about $350 million to buy Riptech and three other electronic-security firms (Recourse Technologies, SecurityFocus and Mountain Wave) that had developed proprietary anti-hacker technology. Symantec merged Riptech's operations with its own and now has four similar centers -- in Britain, Japan, Germany and San Antonio.

Custom Printed Stationery Arrives

Dave Aiello wrote, "I picked up the new CTDATA stationery that I ordered from Allegra Printing and Imaging in Lawrenceville. I explained the bid evaluation process that I used in an article I posted last week."

"We haven't had stationery with a correct address and telephone number since June. It's great to finally get it. Maybe we can use it to drum up more business."

January 7, 2003

Norwegian Court Acquits DeCSS Developer of Piracy Charges

The New York Times reports that a three-member panel in Oslo City Court ruled that Jon Johansen had not broken any laws by using or distributing DeCSS and that he is free to view any DVDs he purchased leagally in any way he chooses. This is a major setback for the Entertainment Industry, which argued that the mere existance of software to decrypt DVDs was an open invitation to digital piracy.

However, the court found that Norwegian law treats a DVD purchased at retail as the purchaser's property, and not merely a license to view the content of the DVD on a player certified by the Motion Picture Association of America and similar industry trade groups. As a result, according to Aftenposten, "Johansen and his defense attorney Halvor Manshaus won on all counts, with the Oslo court ruling that Johansen did nothing wrong when he helped cracked the code on a DVD that was his own personal property."

January 6, 2003

NY Times Points Out Little-Known Value of On-Line Booksellers

There's a great article in The New York Times today called Online Retailers Try to Flourish Year-Round. Deep in the article is some great information about research that Erik Brynjolfsson from the Sloan School of Management at MIT did, comparing Amazon.com with local bookstores and superstores like Barnes and Noble and Borders:

Judging by what consumers spent in 2000 online for books they could not buy offline, Professor Brynjolfsson said the value of the Internet's product selection in this category alone was between $731 million and $1 billion. While consumers often enjoy lower prices online, he said, "the big benefit is getting access to goods you wouldn't otherwise have."

Professor Brynjolfsson takes his point one step further, arguing that the value of greater product selection over the past decade or so — which the Internet has hastened with its nearly endless product offerings — has gone unnoticed by statisticians.

This point is particularly salient regarding venues like Amazon Marketplace. This is the area of Amazon.com where third party sellers offer both new and used versions of books, CDs, DVDs, and other things consumers want. Quite often, this is the place to find an out-of-print book that came out five years ago. Many of these books are more valuable now than they were when they were in print, due to their scarcity. This is exactly what Dr. Brynjolfsson is getting at.

Karlgaard: Can Software Startups Succeed?

Dave Aiello wrote, "My subscription to Forbes resumed Saturday. I found out that the reason I had not received a magazine in about two months was that my address had never been updated since I moved in June."

"One of the more interesting articles I saw in that first issue was a column by Rich Karlgaard, Forbes' publisher, called Can Software Startups Succeed? This is really interesting because I have been discussing the same thing with some of my friends. Some of his suggestions to the leadership of small software companies also seems quite valuable at this point in the market:"

  • Forget trying to be mission critical. No CIO in America is going to bet his company on a little-known startup.
  • Avoid like the plague the phrase "total solutions" when describing your product.
  • Don't compete on price; compete on speed.

Business Software Alliance to Challenge Hollywood on Digital Rights Management

The San Jose Mercury-News reported on Friday that The Business Software Alliance and the Computer Systems Policy Project intend to take on lobbying groups representing the Entertainment industry over the issue of Digital Rights Management. The recording and motion picture industries have relentlessly pursued the introduction of very strong copy protection at the hardware and operating system levels of all sorts of digital devices, including PCs aimed at consumers.

According to the article, the lobbying groups for the computer and electronics industries, "hope to convince Congress that strict copy-protection legislation that sets technological mandates would stifle innovation, harm consumers and threaten an already suffering tech industry."

It will be interesting to see how much influence the Hollywood Establishment loses in Washington, now that both houses of Congress are under Republican control.