Tracking the Crisis: Open News Aggregation

After a whole day of testing, the Middle East conflict tracking page has gotten yet another overall. The ‘page’ is now an entire section, consisting of:
The Blogs View : the original page aggregating blog posts
The Map View: the map view showing local blogs and recent news events
The News View: a summary page showing all recent news events with the option for readers to rate them
The main enhancements today have been around the ‘News’ capability. The system is now open wide so that anyone can submit a news item they have found on the conflict. The item will then be placed on the unapproved items page, where other readers can rate it. Those items that achive a sufficiently positive response will then automatically migrate onto the main news page and the map page.
I’m currently working out additional methods of categorizing and presenting news items — particularly in visual form on the map — but for now, you’ll see that there are several categories from which to select when submitting an item.
My hope is that with this system, and a sufficiently interested community of bloggers and blog readers focusing on it, we can create a single location which sums up the very best reporting, analysis, and breaking news from both the traditional media and the bloggers on the ground. I invite everyone to join me in this effort, and as always, suggestions are welcome.


Mapping Mideast Bloggers

After a rather hurried afternoon and evening of a) learning Microsoft’s Virtual Earth API, and b) Figuring out how to twist and bend it into the shapes I wanted, I’ve managed to integrate TTLB’s Middle East blog tracking with Virtual Earth’s rather impressive mapping capabilities.
The new map-based tracking page is here.
I’ve made a first pass through the regional blogs that TTLB has been tracking thus far, and have done my best to assign them locations appropriately. I didn’t get through every single last one, however, and for some blogs, it is a bit difficult to determine exactly where their authors are located.
So: if you are a blogger actually blogging from the conflict zone, by all means, please e-mail me and let me know your location. Ultimately, I need to plug your location in as a latitude and longitude (i.e., 31.7800 35.2300 for Tel Aviv), but if you live in a major city that is shown on the maps, just tell me that and I can look up the lat/long.
Even if you are not a blogger in the Middle East, however, I need help!. There are many things I could do with the existing map integration and blog tracking to vastly improve their usefulness in tracking the conflict — but there are only so many hours in a day, and I’ve only got these ten fingers pounding on the keys.
I am looking, therefore, for volunteers who would be willing to help me with online research and general gofering. Tasks would include things such as hunting down the latitude and logitude for bloggers in the region and other ‘net searches. Requirements are that you be a blogger with an established weblog and be willing to put up with me — other than that, anyone may apply! In exchange for such assistance, I’ll list you & your blog on the tracking page as an Editorial Lackey, or somesuch. If you’re interested — you guessed it: e-mail me.
That’s enough for one day. Goodnight to all, and to those in the conflict zone: may you and yours find safety, peace, and freedom….

Tracking Page: Middle Eastern Bloggers on the Crisis

I’ve set up a brand new tracking page to monitor the unfolding crisis in the Middle East. The page monitors Israeli and Lebanese blogs and aggregates their latest & most linked posts continually throughout the day, and additionally shows the most popular posts on the conflict from non-local bloggers.
The code is working well now, but I’d welcome suggestions on blogs that I may be missing. If you have suggestions for blogs that should be added (or if you think I’ve included a blog inappropriately), please let me know via e-mail.
One note on the layout of the page: if you’re wondering why sometimes the page shows Israeli bloggers at the top and sometimes it shows the Lebanese — it’s on purpose. I’ve deliberately coded the page to randomly select the layout each time it is generated so that each community of bloggers gets the top-of-the-page exposure equally…

— N.Z.

Update: After prompting by Mickey Kaus, I did some searching for Palestinian blogs, and found a bunch. There is now a section devoted to Palestinian bloggers on the page, so check it out. As for why it wasn’t there in the first place: I simply was following links from bloggers I know to find ‘local’ blogs, and for whatever reason, none of them led to the cluster of Palestinian blogs I’ve now found. Still, it was fairly dense of me not to think of searching harder, so sorry about that, and thanks to Mickey for nudging me.
On a related note, a word on how I’ve selected the blogs for listing. My criteria has been simple: for each section, I’m looking for blogs that:

  • Are written by a resident or expatriate of the area
  • Have been updated within the past several days
  • Appear to be posting regularly on the conflict

I’ve no interest in filtering based on content or views, but I have filtered out local bloggers who choose to focus their blogging on poetry or art rather than war. Quite possibly a wise choice for them, I suppose, but not terribly useful for the purposes of the tracking page…

International Talk Like Apu Day!

Given the date, and in honor of Joe Biden*, I declare that today, everyone must talk like Apu.
Consider it an Indian version of Talk-Like-A-Pirate-Day.
Now, to the Kwik-E-Mart I must go!

* “”You CANNOT go to a 7-11 … unless you have a slight Indian accent … I’m not joking.” — Sen. Joe Biden (D-DE)

Update: Obviously, with the events of today in Mumbai, this turns out to be a bad day to make a joke that some might interpret (incorrectly) as being at Indian expense (hopefully it was obvious that my target was Mr. Biden). For the record, however: my thoughts and hopes go out to all those affected by today’s bombings…

Fight DDOS with DGOS?

Glenn reports that Jeff Goldstein is suffering another DDOS attack, limiting access to the Protein Wisdom we all crave.
I agree with the Instafellow: this is indeed getting out of hand.
So I have a thought. It seems that what we bloggers need is a way to combat a Distributed Denial Of Service (DDOS) attack which leverages the same principals as the attack itself — most particularly, the Distributed part. Call it a Distributed Guarantee Of Service.
The challenge is this: how could we establish a system so that a blogger suffering a DDOS attack (or simple system downtime, even) could be guaranteed a way to post during their outage.
The key part would be setting up a way for member blogs to ‘host’ a downed blogger’s posts. It seems to me that there are two categories of bloggers that matter here: those that are on limited / controlled hosts such as Blogspot (who therefore can’t run server-side scripts, but can generally include Javascript code) and those who have full hosts (who can run PHP or other server-side scripts).
So what I’m picturing is a PHP script that would provide the actual ‘hosting’ which would run on the full hosts, and actually act as a temporary guest home for a downed blogger. And then perhaps a Javascript applet for the limited hosts which could at least serve as a notifying beacon that there is a blogger in ‘down’ status, and link a reader to the full hosts to actually see that blogger’s posts.
There’s lots of design details to be done here. How could the blogger post? E-mail, or via a simple web-form hosted by the full members? How can the post, once entered on one full member’s site, be replicated automatically to all other members? (That’s the magic: it has to be replicated so that the DDOS attacker can’t just re-target a single backup site).
I’ll noodle on this more and post further thoughts, but I’d like to open the discussion and get some other smart minds working on this problem. Comments are open — let’s get to work!
Update: OK, we’ve got some good discussion rolling in the comments. So here’s the deal: I’ve got ideas, and I can contribute support & a bit of thought bandwidth to this effort. But there’s no way I can be the primary driver of this, what with everything else I’ve piled on my plate. So we need some volunteers who do have some bandwidth to form a working group to further flesh out this problem and potential solutions, and then go ahead and actually do it.
So: if you’re interested in being part of such an effort, speak up in the comments, and/or e-mail me directly. If necessary, I can set up a Wiki or a mailing list to facilitate the discussion — but if someone else can do that, go ahead and do it! I won’t be offended.
With that said, a few more ideas on the substance of the problem:
I believe our goal is not strictly “fault tolerance” for a given blog or set of blogs. I think accomplishing that is impractical, and would involve some kind of mirroring solution that would be overkill for what we’re trying to accomplish. In my mind, our goal should be to ensure that when a blogger’s site is down:

  • a) They have a place to post new blog posts
  • b) There is an established system so that their readers can find those new blog posts
  • c) The new posts are hosted in a distibuted manner so that they are mirrored on many different sites and are therefore protected from a secondary DDOS attack.

Note that what this essentially means is that we wouldn’t be constantly mirroring every participating blog’s site — we’d simply be mirroring new posts by a downed blogger once the system is activated. This strikes me as a simpler, and more realistic approach, although I’m open to thoughts about some crude level of mirroring for recent, pre-DDOS attack posts. Terry proposed using RSS feeds below, which is a good first thought, but I can say from my experience with TTLB that the main problem there is many bloggers don’t include full content in their RSS feeds. I suspect a better solution might be brute force: just have a way to copy the full HTML of each blog’s front page to a distributed archive. The cleverest way would be to somehow have each blog copied to a small number of mirror-blogs (let’s say 10) — if we have a solution spanning hundreds or thousands of blogs, it obviously doesn’t make sense to have every blog mirrored at every other blog’s site.
Finally, I’d suggest that we approach this problem in several phases:

  • Phase 1: Quick, Dirty, and Manual: With only a little bit of coordination, we could set up a mostly-manual system virtually immediately which would allow a downed blogger to have a place to go. This could be as simple as identifying several volunteers with MovableType or other full-hosted blogs who are willing to create a special “DGOS blog” within their installlation that, in the event of an attack, a downed blogger would be given access to for posting. I’m sure there are other ways to approach the problem manually too — let’s start there!
  • Phase 2: Automated and Distributed: With a manual solution in place, we can focus on implementing the whiz-bang approaches I’ve started outlining above, or alternatives.
  • Phase 3: Nirvana: With any complex implementation, I find that the first release is never really the full solution you wanted. We’ll probably find that we’ve got a medium-term Phase 2 solution that will work, but isn’t perfect, and a long-term Phase 3 solution that is really everything we want it to be.

OK, that’s enough from me for now. Like I said, please speak up if you’re willing to join a working group and get cracking on this, and even if you are not, please spread the word on this idea. Thanks!
Update 7/11: I’m pleased to report that Tim at Aardvark Salad has joined the effort, and his initial thoughts on the problem can be found here. Tim has requested a SourceForge project site for the effort, which should hopefully be available later today. More to come…

two percent solution?

Glenn points out a fellow who notes the potential of solar energy:
“If 2 percent of the continental United States were covered with photovoltaic systems with a net efficiency of 10 percent, we would be able to supply all the U.S. energy needs,” said Bulovic, the KDD Associate Professor of Communications and Technology in MIT’s Department of Electrical Engineering and Computer Science.
Um, ok. Glenn notes “Two percent is a LOT of land”, but that’s understating the case a bit I think.
Let’s do some math! (I was told there would be no math. Shhh!)
According to this page, the total land area of the U.S. is 3,537,379 square miles. Take away Alaska and Hawaii to get the continental U.S., and you are left with 2,959,005 square miles. Two percent of that is…
Fifty-nine thousand, one hundred and eighty square miles. That’s 59,180.
For perspective: Over half of the fifty states are smaller in area than 59,180 square miles. The closest in size to that number are Iowa (55,869), Michigan (56,804), and Georgia (57,906).
So: who’s for paving over Georgia?
Because unless I’m missing something, that is what we’re talking about: literally paving over that much area, and I have to assume utterly destroying any flora, fauna, or other living things that are unlucky enough to have been previously occupying it. Unless they happen to, you know, not require sunlight.
Now that would be one hell of an Environmental Impact Study.
Professor Bulovic seems to have thrown out this statistic as a positive for solar energy, but he’s obviously never negotiated with a local zoning board. If that’s the best future we can hope for with solar as our primary energy source, I suspect even the most strident environmentalists will cry out, “Bring on the coal!”

Continue reading “two percent solution?”