Begin forwarded message:

From: Brett Glass <brett@lariat.net>
Date: May 4, 2009 11:07:50 AM EDT
To: dave@farber.net, "Ip ip" <ip@v2.listbox.com>
Subject: An unusual denial of service attack

Dave, and everyone:

This weekend, my ISP suffered an unusual sort of denial of service
attack.

Starting on Saturday morning, users were reporting that their Web
browsing had slowed to a crawl, though other services were working
properly. I investigated, and saw that our upstream connection to the
Internet backbone was being saturated -- but not by any one customer.
So, I looked at the statistics on our Web cache (an activity, by the
way, which I'm sure that certain privacy advocates would find
tantamount to "snooping," even though it was for the purpose of
managing the network). After awhile, I was able to figure out what was
wrong.

We were facing a distributed denial of service attack from the world's
largest "botnet:" Microsoft's "Windows Update."

Virtually every Windows machine on our network -- and most of our
customers's machines are running Windows XP or Windows Vista -- was
individually downloading many large updates. (See

http://www.computerworld.com/action/article.do?command=printArticleBasic&taxonomyName=Security&articleId=9131573&taxonomyId=17

for a list of some of the many security holes that were being patched.)

Fixing holes in Windows is a good thing, but to command more than 90%
of all of the computers around the globe to "phone home" at the same
time is, obviously, not. It's doubly bad when the updates are
explicitly marked as non-cacheable, making our Web cache of no use to
stem the flood.

What's worse -- at least for our small ISP -- is that the updates are
distributed for Microsoft by a company called Akamai. Akamai, as many
of you know, places caches at the hubs of many ISPs' networks -- but,
alas, only those of larger ones. Our smaller ISP, which has never been
able to convince Akamai to place a cache at our location despite many
years of requests, therefore must use backbone bandwidth to service
all of these redundant requests. When I checked -- and it was not at
the peak -- the traffic was consuming about half of our main DS-3 line
to the Internet, leaving only half of its capacity available to carry
all other traffic (including VoIP and bandwidth-intensive streaming
video). Our cache's CPU utilization was above 95%, slowing response
times still further.

I solved the problem by telling the cache to throttle traffic to and
from Akamai's upstream caches, which were serving up the updates.
Instantly, the load dropped off and normal service was restored.

As Spider-Man creator Stan Lee once noted, "with great power comes
great responsibility." Microsoft, by virtue of its control over
Windows-based PCs, has the ability to shut down the entire Internet at
will -- and must be careful not to do it, inadvertently, by turning
90% of the world's PCs into a "zombie army."

Furthermore, content delivery networks such as Akamai, which
distributes Microsoft's updates, must not be allowed to discriminate
against smaller providers by making updates uncacheable (at least by a
standards-conforming Web cache) and then denying smaller ISPs access
to a cache that WILL cache them. (Google, too, is also placing caches
at the hubs of larger ISPs, thus giving them an edge when it comes to
delivering Google services and video.) Small and competitive ISPs
already have a tough row to hoe when competing with the telcos and
cable companies. If they are further disadvantaged by prejudicial
business practices of content providers and content delivery networks,
Internet service will -- devastatingly for consumers -- become a
duopoly.

--Brett Glass