Fight DDOS with DGOS?

Glenn class=”textlink”>reports that Jeff Goldstein is suffering another DDOS attack, limiting access to the Protein Wisdom we all crave.
I agree with the Instafellow: this is indeed getting out of hand.
So I have a thought. It seems that what we bloggers need is a way to combat a Distributed Denial Of Service (DDOS) attack which leverages the same principals as the attack itself — most particularly, the Distributed part. Call it a Distributed Guarantee Of Service.
The challenge is this: how could we establish a system so that a blogger suffering a DDOS attack (or simple system downtime, even) could be guaranteed a way to post during their outage.
The key part would be setting up a way for member blogs to ‘host’ a downed blogger’s posts. It seems to me that there are two categories of bloggers that matter here: those that are on limited / controlled hosts such as Blogspot (who therefore can’t run server-side scripts, but can generally include Javascript code) and those who have full hosts (who can run PHP or other server-side scripts).
So what I’m picturing is a PHP script that would provide the actual ‘hosting’ which would run on the full hosts, and actually act as a temporary guest home for a downed blogger. And then perhaps a Javascript applet for the limited hosts which could at least serve as a notifying beacon that there is a blogger in ‘down’ status, and link a reader to the full hosts to actually see that blogger’s posts.
There’s lots of design details to be done here. How could the blogger post? E-mail, or via a simple web-form hosted by the full members? How can the post, once entered on one full member’s site, be replicated automatically to all other members? (That’s the magic: it has to be replicated so that the DDOS attacker can’t just re-target a single backup site).
I’ll noodle on this more and post further thoughts, but I’d like to open the discussion and get some other smart minds working on this problem. Comments are open — let’s get to work!
-N.Z.
Update: OK, we’ve got some good discussion rolling in the comments. So here’s the deal: I’ve got ideas, and I can contribute support & a bit of thought bandwidth to this effort. But there’s no way I can be the primary driver of this, what with everything else I’ve piled on my plate. So we need some volunteers who do have some bandwidth to form a working group to further flesh out this problem and potential solutions, and then go ahead and actually do it.
So: if you’re interested in being part of such an effort, speak up in the comments, and/or e-mail me directly. If necessary, I can set up a Wiki or a mailing list to facilitate the discussion — but if someone else can do that, go ahead and do it! I won’t be offended.
With that said, a few more ideas on the substance of the problem:
I believe our goal is not strictly “fault tolerance” for a given blog or set of blogs. I think accomplishing that is impractical, and would involve some kind of mirroring solution that would be overkill for what we’re trying to accomplish. In my mind, our goal should be to ensure that when a blogger’s site is down:

  • a) They have a place to post new blog posts
  • b) There is an established system so that their readers can find those new blog posts
  • c) The new posts are hosted in a distibuted manner so that they are mirrored on many different sites and are therefore protected from a secondary DDOS attack.

Note that what this essentially means is that we wouldn’t be constantly mirroring every participating blog’s site — we’d simply be mirroring new posts by a downed blogger once the system is activated. This strikes me as a simpler, and more realistic approach, although I’m open to thoughts about some crude level of mirroring for recent, pre-DDOS attack posts. Terry proposed using RSS feeds below, which is a good first thought, but I can say from my experience with TTLB that the main problem there is many bloggers don’t include full content in their RSS feeds. I suspect a better solution might be brute force: just have a way to copy the full HTML of each blog’s front page to a distributed archive. The cleverest way would be to somehow have each blog copied to a small number of mirror-blogs (let’s say 10) — if we have a solution spanning hundreds or thousands of blogs, it obviously doesn’t make sense to have every blog mirrored at every other blog’s site.
Finally, I’d suggest that we approach this problem in several phases:

  • Phase 1: Quick, Dirty, and Manual: With only a little bit of coordination, we could set up a mostly-manual system virtually immediately which would allow a downed blogger to have a place to go. This could be as simple as identifying several volunteers with MovableType or other full-hosted blogs who are willing to create a special “DGOS blog” within their installlation that, in the event of an attack, a downed blogger would be given access to for posting. I’m sure there are other ways to approach the problem manually too — let’s start there!
  • Phase 2: Automated and Distributed: With a manual solution in place, we can focus on implementing the whiz-bang approaches I’ve started outlining above, or alternatives.
  • Phase 3: Nirvana: With any complex implementation, I find that the first release is never really the full solution you wanted. We’ll probably find that we’ve got a medium-term Phase 2 solution that will work, but isn’t perfect, and a long-term Phase 3 solution that is really everything we want it to be.

OK, that’s enough from me for now. Like I said, please speak up if you’re willing to join a working group and get cracking on this, and even if you are not, please spread the word on this idea. Thanks!
-N.Z.
Update 7/11: I’m pleased to report that Tim at Aardvark Salad has joined the effort, and his initial thoughts on the problem can be found here. Tim has requested a SourceForge project site for the effort, which should hopefully be available later today. More to come…

41 thoughts on “Fight DDOS with DGOS?”

  1. How do you guarantee that the DGOS system isn’t spammed? Once the system is documented and in public, how do you keep spammers from abusing it?
    Cool idea, otherwise. Just thinking out loud.

  2. This is a 2 part solution, one backup (albeit temporary) and the other is the ability to post at an alternative site).
    I would think that the best promise is to leverage RSS feeds and use them to archive and backup a bloggers posts. I have found that FeedBurner is the most comprehensive for me because it links images (which will no doubt be unavailable during a DOS, but who cares, it’s the content that matters most). Ex http://feeds.feedburner.com/Soundtrip/
    This usage is limited such that it can only be used to archive post contents, generally text. Multimedia such as podcasts are temporarily lost IF they are hosted on the same server that is undergoing a DOS. This argues for distributed hosting, especially for podcasters; blog/text on one site, mp3 content on another host. Pros and Cons are many.
    The next task is to find an alternative site to host new posts and possibly allow comments while the main site is down.
    Alternatives such as having a backup blog at a clearinghouse such as WordPress.com is not a bad idea. It would be nice if there was a wordpress plugin to feed another WordPress site with the rss feed so that posts are automatically mirrored.
    You might want to team up with some sort of blog alliance that would allow an alternative posting solution as a backup. I would certainly provide logins and temporary hosting for Webloggin members and any needy people who are under attack (wordpress is virtually a 5 minute setup).
    Users with access to their logs and root file system can try and shut down a DOS as it occurs but it is often better to let the host deal with it because they have bigger and better tools. That is part of a different discussion as the main goal is to try and provide a backup posting mechanism while down.
    Fault Tolerance is a very complex and very costly. Smart routers and big cisco units can redirect to distributed sites to reduce loads but that is a hosting solution and not a blogger solution. Bigger hosts should be able to provide more stability during a DOS.
    Terry

  3. You need to cluster — and set up SRV records in the DNS (control your own records directly). Have the blogsite run across distributed servers, just like ecommerce sites do, use DNS to provide load-balancing and failover. A single DB backend that is hidden from outside access or just a cluster of DB-backends that hotsync each other will do the job, all supported by MySQL and PHP (or Perl).
    In fact, an interesting approach could be for a group of blogging sites to form a syndicate and distribute their blogsites across major hosting sites with strong resistance to attacks. Many of these support root-control servers with MySQL/PHP. Best to avoid tinker toy resources, like Fedora, which all due respect, is not a production environment. (I recommend FreeBSD, the system of champions — Google, Yahoo, HotMail, FreeBSD Mall, that site that tracks 70,000 simultaneous flights and draws maps on demand.) One could do this without breaking the bank, since one can buy modest disk space and root-controlled VM for as little as $25/month/site on decent bandwidth providers; with pooling of $$$ one could probably do better — and tune the selection of sites to get the best mix of service for the $$$. One may want to look at a collection of COLO sites as a possibility too — that cost has gone way down in the last 5 years. That would allow huge redundant disk array(s) for the DB backend(s) and allow the distribution of front-end for DDOS resisitance. It’s mainly flow constriction from the poster to the front-end, as the back-end can be buffered and be kept away from malicious activity.

  4. The problem is actually pretty easy to solve by creating a counter and sharing program. If you have more than, let’s say, 1000 hits per day on the counter average the past week, you gain access to a distributed posting method and your posts are tossed into a tarball containing only the text. The tarball is hosted as a p2p file among the participants. This would mean that to take down Protein Wisdom, you’d have to take out Instapundit, Atrios, and the Daily Kos among others. The prospect is ideologically unsatisfying. This would leave external attacks on the blogosphere in general by the MSM or other forces that want to take down the whole sector but if you’ve reduced the problem to being DDOS attacked by Knight Ridder, it’s essentially a solved problem (unleash the lawyers!).
    In the end, you’d have a “protect free speech” link on each blog linking to both the most recent tarball of the past week’s blog output of the busiest bloggers out there along with a place for people who are victims of DDOS and other such nonsense to add posts by logging in to participant sites. Ideally, blogger software, when it gets back up, would pull the intervening posts back into the original blog but you could make it a manual process in order to safeguard against the tarball itself becoming a target for fake entries and corruption.

  5. Call me silly, but I’d harden the infrastructure. You’d need a four-part solution. I’d run a software firewall on the server itself, and front it with an integrated hardware security solution consisting of a managed switch, a hardware firewall, and an intrusion detection system. Cisco makes a well-integrated hardware set.
    The downside to this is that you’d have a startup cost of about $15K to $18K and a monthly recurring cost of roughly $1.5K – but you’d be pretty rock solid.

  6. A simple solution which already exists is to create “mirror” sites. Many mirror sites exist if only to “spread the bandwidth” over many servers and increase accessibility. An example of a model might look like this:
    Instapundit is hosted on one server, Truth Laid Bear on another. If each traded off space for mirrors of each others sites using canonical domain names, each would benefit from having a dedicated backup such as instapundit.truthlaidbear.com, or – truthlaidbear.instapundit.com. Muliply that concept over 50 or more bloggers, each using different hosts with at least 5 bloggers employing mirrors for 5 other bloggers and you have a “near” fail-safe system which would deny the DDOS attacks. There are bandwidth issues to consider, but that is really pay-for-play money matter.

  7. I wonder if Pajamas Media would be willing to bankroll some research and provide infrastructure for a proof-of-concept?

  8. This is a great dialogue by thoughtful experts. I would raise one related point: This is all defense, what about offense? Is there a way to determine who is responsible for the attacks, and having done so, is there a legal deterrant?
    Also Bill Nye it is good to note that there is an effective “cadillac” solution out there, but the cost does seem prohibitive for all but the largest, most developed sites (Malkin, LGF, etc)

  9. You need to talk to the hosting ISP and make sure they have DDoS protection. There are many companys out that that make DDoS protection devices that stop the attacks as they happen. Here are a couple examples and there are a lot of others products out there also so read some of the whitepapers on these websites.
    http://www.cisco.com/en/US/products/ps5894/index.html
    http://www.radware.com/content/products/dp/default.asp
    You get what you pay for with hosting companies, so if you can find a ISP that has these types of devices in place you won’t get hit with a DDoS attack. If you are running your own network, then getting end point security in place be the solution. Talk to a GOOD network security person if you have your own site about how to protect it.

  10. all the university user services i know of have excellent network monitoring and forensics, state of the art mecha, also. the govvies have this too, but university help might be easier to access.
    perhaps universities could be involved in tracing DDoS attacks? and a bonus for them might be uncovering DDoS attacks originating from their campuses.

  11. I sense the approach – to try to keep a site going during a DDOS attack – is the wrong one. All efforts should be made to:
    1. Prevent the attack in the first place.
    2. Stop one that gets thru quickly
    3. Trace back to the source of the attack and prosecute vigorously. Large blogs can run up significant financial losses in a hurry during a DDOS attack. If you could track down the perp – sue him into bankruptcy. That, more than anything else I can think of, would bring an end to these attacks.

  12. Wouldn’t we need a scheme to re-point the URL to a new IP address, instantly? If PW.com is on server 123.123 and the hosting service has various levels of backup, that’s good.
    Otherwise, someone’s gotta inform the internets, “for PW.com, dont go to 123.123, but use 456.456.”
    Sounds tricky.
    I suppose we could have an informal, spit-and-baling wire approach, where Instrapundit et al announce, “Jeff G. will be posting at TTLB.com for the next few days.” Very unsatisfying.

  13. Meanwhile, she’s lying again:
    http://www.insidehighered.com/news/2006/07/10/frisch
    >for the record
    Jeff Goldstein’s web site was down for two days after the incident. While I take responsibility for most of the horrid, inappropriate, mean-spirited, unprofessional comments attributed to me, I never wrote anything about French-kissing. That was added and falsely attributed to me.
    Deborah Frisch, at 9:25 am EDT on July 10, 2006<
    Maybe there’s a libel suit in here, too.

  14. You need something like the peer to peer music sharing software so there are copies of the blog in many places and the peer to peer software finds the blog entries for you. Each blog reader could be blog server. This massive distribution would make dos attacks impossible.
    It might be useful for getting around the great firewall of china too.

  15. I believe a conservative blogging web service is needed. That way only valid IP’s can blog through that web site. Web services can be hosted on multiple domains that point at a central database. The Web Service could then return XML for use in RSS feeds or for transforming to HTML on blog sites using AJAX.

  16. I concure with nobody_special’s suggestion. Get control of the DNS servers and cluster them. aka not on the same network. You can instanly change a DNS record to point to any Mirror site that you have waiting as just an IP address. Keep your published files also local so you can update any mirror at any time. It would only take a few minutes to edit a DNS record. Stay away from PHP as much as possible. Start using JSP or J2EE. Their are sooo many exploits in PHP produced apps. Sites like Spry.com or eApps.com are good serivces providers and are good at defending against DDOS attacks.

  17. The total volume of responses to bloggers can’t be so large that all responses to bloggers could be routed through a network of blogger firewall servers that check packets for DDoS attacks.

  18. Egfrow / nobody_special:
    I have to differ with you in fundamental approach, here. I think aiming for a solution at the DNS/infrastructure level is going to be infeasible: it would require way too much coordination with every single hosting provider that any blogger is using. That’s why I’m aiming at a solution at the “application” layer: a mechanism that involves only coordination and participation by bloggers themselves.
    That doesn’t mean we can’t (or shouldn’t) try to enlist the help of hosting providers, but I don’t like going down a path that mandates that if a blogger’s host won’t change x, y, or z about the way they do their business, that blogger can’t participate.
    In simpler terms: Grabbing control of and clustering DNS servers is a great technical solution, but seems utterly impractical in a real-world sense…

  19. How about, just to begin things at their simplest level… a blog catalogue that lists alternate sites for bloggers? I could easily host something at my operation called, say, ‘altblogs.com’ (haven’t checked to see if it’s available) where if you type in a search box ‘protein wisdom’ (or it’s URL) you’d receive a response where alternate URLs show up (or, if you allow cookies and have a login for the site, have a preference for the browser to go directly to the alternate blogsite) -or a response in which the server responds that the blog isn’t registered with an alternate.
    Then, if in case of DDOS of THIS site, then this site is the only one to need mirroring.
    Thoughts?

  20. Blogs Under Attack: Whaddya Do?

    NZ Bear is asking how smaller operators harden their sites against attack or engineer a blog redundancy system. Really interesting tech talk going on here.

  21. I’ve just registered ‘alternateblogs.com’ in case it’s a useable idea. (altblogs.com was unavailable).

  22. We solved this problem for the Internet Haganah site three years ago.
    Ingredients:
    1. DNS service separate from web hosting, with a DNS provider that allows round-robin DNS. Round-robin means you have multiple A records – for the lay people that means you have a domain name assigned multiple IP addresses.
    2. Web hosting separate from content management. This means you have multiple web hosting accounts on severs at different datacenters, and a separate account where you actually prepare your content for publication. When you publish you copy your content to the other locations.
    That is the system in a nutshell, and it works. Now we don’t get DDoS’d. Instead we get death threats:-)
    Supporting comments complicates this. You’d want a database server separate from both the content management site and the content distribution sites, so that comments for all those instances of your site are stored on a single database. That single database becomes a point of failure, but then at least only the comment system goes down.
    If you want to turn this in to a commercial solution you’ll want a minimum of 6 dedicated servers all at different datacenters around the US. If I were doing it I would have at least 6 ‘live’ servers and another 6 ‘spares’ (all at separate locations from the first 6) so that I could pull servers out that were under attack and replace them with others.
    It’s not perfect, but it’s good enough. Current DDoS defense solutions typically involve rapidly isolating the victim to save the rest of the network. What you want is to defend the victim.

  23. Real DDOS attack prevention would require ISP support and an RFC for DDOS prevention.
    The DDOS packets (I assume) could be matched by packet template or IP. The packets could be matched and silently discarded at the edge routers of the ISP.
    When a DDOS attack starts the offending packet template/originating IPs could be reported to the ISP.
    The RFC would define the protocol/packet format for reporting the attack to the ISP.
    A software module, that conforms to the RFC, to detect and report DDOS attacks would be added to customer servers.
    A DDOS would have to overwhem the Peer-Peer ISP connections to be successful. Since this bandwidth is hundreds (if not thousands) of times greater than the connnection to the server(s), DDOS attacks would require ISP or government involvement to be successful. If the plan was extended to the US international/satellite connections only US enties subject to US laws (and lawsuits) could launch a DDOS attack and it would have to originate within the customer’s ISP network.
    Toughing the laws on DDOS attacks and requiring compliance by ISPs with the RFC would complete the picture
    The distributed server – multihomed server approach combined with customer-premise filtering would mitigate smaller attacks. I doubt that most blog sites are hosted on dedicated servers – so it is a matter of selecting a service provider who provides this configuration.

  24. Splendid idea!
    Of course, there is an added problem with regards to how our respective hosts would respond to the added bandwidth usage when we’re “guest-hosting”, but that’s just me brainstorming here. I LOVE the idea.
    Another note: I love the idea of a “beacon” thingy that could alert others to which sites are down when they go under. We see it a lot of times: A site goes down and there’s no way of knowing why it’s down or where to go to find out. Obviously, the affected blogger can’t publish an announcement on a site that’s gone dark and, equally obviously, there’s not much use for him or her to have a link to a “backup blog” on a site like Blogger when you can’t access the site where the link is posted.
    So, still brainstorming here, would it be possible to get a small button you could host on your website linking to a current list of sites down with the reason why and where, if at all, you could find a backup? That way you wouldn’t have to depend on somebody knowing somebody who knows somebody who heard from a fourth party etc. etc.
    I’d slap that button up on my site today if it were available.

  25. I host my blog on Typepad, but I have another site that right now is sitting dormant as I work on a phpNuke site that is not ready for launch. I haven’t had the time to get this site up and running. I’m self-taught and cannot claim any knowledge on how to create the system you all are talking about, but I would be more than willing to make my alternate site or service available if someone more technicially knowledgeable wanted to help me make it available as a backup. I know that the unlaunched project I’m working on allows for guest journals by registered users and I think those could be set up as mirror sites. Anyway, if anyone is interested, contact me and I’ll cooperate in any way I can.

  26. Misha: I was envisoning exactly that kind of ‘button’ when I mentioned a Javascript thingy in my original post. That’s how I think we solve the “how do readers find the attacked bloggers posts” problem.
    Every member blog could have that Javascript on their sidebar, and while all is well, it just sits there saying “Member of the DGOS network” or something unobtrusive. But if an attack starts, it lights up and says “Blogger Jeff Goldstein is under attack, and his posts can be found here” with a link.
    There is technical trickiness to be figured out on how to implement that, but I think it’s possible…

  27. A simple solution would be to use rss to aggregate the blogs and automatically add them to a list. There wouldn’t be comments, for space concerns. Setup a login for blog owners to login and make add new posts. then this site would get syndicated through rss and people would get the feed through a central service. I’m going to look into this. I have a spare domain name available and will make anything I come up with available here: http://www.leechgraphics.com

  28. Fighting Back

    NZ Bear, the proprietor of The Truth Laid Bear is making a modest proposal. Given the continuing DDOS attacks on certain blogs, such as what is happening to Jeff Goldstein is it possible to design and implement a DGOS to combat the attacks? I…

  29. N.Z.,
    I don’t have the technical expertise for this kind of thing, but I applaud that you’ve taken it on.
    I emailed you with my contribution to the effort.

  30. The cisco solution discussed by JD seems along the right path. CISCO and other network infrastructure providers have many solutions to combat these problems. The hardware based solution requires a ton of money so one should look for a webhost or ISP that has implemented the infrastructure to handle these kind of attacks. .
    CISCO even provides routers that handle redirects automatically in the case of a dedicated server dropping off the cluster. This is helpful in the multi-server environment that will allow for load balancing and fault tolerant solutions and is better than round robin style DNS mechanisms.
    These units are all hardware centric designed to stop the attacks as they occur and provide maximum uptime in the face of a loss on the server side.
    From there however you may need to revisit the server side caching mechanisms for blogs. For instance it is easy to move requests from from server to server but you may need to have a write through cache on the backend. This is nespecially ecessary for e-commerce sites so it would probably be best to have the e-commerce servers on a different server, perhaps even host in that matter to take them out of the picture. The blogs may not need such a sophisticated write through cache.
    I think the best solution must be handled at the ISP/webhost level from the perspective of preventing an attack. Any solution must be cost effective as well as unobtrusive for the bloggers who want to do nothing more than write.

  31. When is there going to be an InstaCon to work out these details and share ideas face to face?

  32. My first thought is that hardware solutions are going to require too much buy-in from ISPs to be useful.
    Tracking the IP address of the packets isn’t going to work; it depends on the type of DDOS. If it’s a ping flood, they may be coming from a poorly configured DHCP server, where the first IP address in a range (like 10.10.0.1) will be pinged and every computer in the range responds. If the attacker fakes (or “spoofs”) the victim’s address, this unsuspecting dupe amplifies the number of packets the attacker sends by, possibly, hundreds, all targeting the victim.
    However, if it is a port 80, TCP flood, you may be in luck. These don’t really expect a response and the attacker can send more packets if they don’t send real requests. These establish half of a connection and tie up memory while the server times out. Does anyone know if the user can send packets during this? Or if the website appears down when queried locally? There may be a daemon that can be established in JSP containers and .NET containers that can periodically query the hosted site and, if the site is down, establish a connection to a “sister” site (or sites) to upload the RSS feed. Because the client of this connection is the victim server, the port is unpredictable by the attacker and it isn’t listening, so the attacker can’t attack it. Of course, the “sister” site may be attacked, unless it is a particularly robust site that has invested in the hardware described above. The “button” on other blogger sites be initialized by a web service on the “sister” site or a central repository that specializes in this purpose.

  33. Fighting DDOS Attacks

    After the latest big DDOS attack–and especially with the frequent ones on mu.nu–N.Z. Bear is working on a way to fight these idiots.
    The challenge is this: how could we establish a system so that a blogger suffering a DDOS attack (or simpl…

  34. Hi NZB!
    By a curious coincidence, last month I was hired by a VC group to build a fully-distributed blogging system. (It’s not a pure blogging app, but it’s a full blogging app among its other features.)
    We’ll be going into early beta in September unless something comes unstuck.
    The problem is, you’d have to use our app rather than your preferred one. But it’s a very nice system. 😉

  35. Since I am not in any way a programmer, maybe this thought has already been addressed in language that I didn’t understand (just thinking out loud here):
    There are many website where, when you reach the URL, have a simple page (I’ll call it “Welcome Mat”) that may only say “Enter **** site” and a button to click to enter.
    Let’s say Protein Wisdom had a “Welcome Mat”, and the WM and the blog are on different servers. When a visitor clicks the Enter button, the WM sends him/her to the URL of the blog, which might be slightly different. Say, proteinwisdom.com gets you to the WM, and the WM sends you to jeffgoldstein.net.
    A DDOS attacker will target jeffgoldstein.net. If/when the Welcome Mat finds the site unavailable , it will automagically redirect the visitor to a mirror site or, say, to steingoldjeff.us, which would, somehow (don’t ask me how), be set up to post the same exact content that the “real” site contains. (Perhaps the “clone” site could used a google-like crawl every few hours to update it’s content.)
    Did that make any sense whatsoever?
    Also, one more idea that’s probably not at all possible: Would it be possible to develop a kind of “reverse anonymizer” that keeps DDOS attackers from ever tracking the blog’s source from the git go?

Comments are closed.