Month: January 2005

When I telephoned a woman named Boxer in New York last week, I wondered who might answer. A DNC flack? A hack posing as a journalist? Someone paid by The New York Times to craft hatchet-jobs on Iraqis who dare to express thanks to America for deposing Saddam? Or simply a lazy writer with some confused ideas about fact-checking and objectivity? Until she picked up the phone, she was just a ghost on the page.
The mystery began last month when I went online to see what the mainstream media considered quality journalism. I stumbled into an ideological snake pit. Out of a list of three major networks (Fox doesn’t count: get over it) and about as many major dailies, I picked The New York Times because I keep hearing about that damned crossword puzzle of theirs.
It delivered more than that. The newspaper, which is quite upbeat about reporting all the bad Iraq news that’s fit to print, had provoked a deluge of intrigue and vitriol. People posting messages on “web” “logs” accused the paper of irresponsibility, laziness, and — common theme, here — irresponsibility again.
Abruptly, the Times had managed in a single story to crystalize everything sordid and depressing about American journalism today. I was determined to get to the bottom of the story, and determined to continue using the first-person voice, no matter how irritating it was and how hackneyed it made my piece sound. And so I set off to Times Square in search of the real story.
Turns out that the Times has decent security at that front door of theirs, though, so I wandered down the street a bit and found that over on 8th Avenue, a conspiracy theory had emerged about the Times on the corner of 8th and 42nd.
One of the guys who “owns” that corner, a colorful fellow named “Mouth” who makes up in keen insight what he lacks in teeth, had some questions for the Times. He wanted to know whether someone in the Zombie Leprechaun Conclave (or someone close to it) was funding it. And what about those newfangled “color” photos on the front page? Did the Zombie Leprechauns “have a shadow role in promoting them”?
Odd.
The questions boiled down to whether The New York Times was “incompetent”. Incompetence occurs when a supposedly professional operation actually doesn’t bother checking facts, shuns anything resembling fairness or objectivity, and publishes lazy pieces that advance their internal editorial biases while flaunting a blissful disregard for reality and truth.
Sarah Boxer, inexplicably still at the Times, tried to quell some of the doubts: “Hi, I would be happy to answer your questions, as you do raise some valid questions.” To the question of the Zombie Leprachaun connection, she responded: “All I remember is that we get a pot of gold from a creepy looking little green guy every month.”
That did not quiet the suspicions on 8th Avenue. A man answering to the name “Queen Britney The Second” reported that his “mental harmonic energy transmissions were always returned rather rudely” by The New York Times. His conclusion? The newspaper is “a refuge for Leprachaun Zombies, Leprachaun Zombie sympathizers, and associated people who look more than a little bit like Leprachaun Zombies.” He added, “I hope some serious attention will be brought to bear on this ‘New York Times’ and reveal it as a fraud.”
What kind of fraud? One guy who offered to let me taste his shoelaces suggested that the newspaper is actually published by humans who are coached by the Leprachaun Zombies on what to write. Another, in support of that theory, noted the Times’ suspiciously enthusiastic support for New York’s annual St. Patties Day Parade. A third man (or woman; not sure, really) observed that coaching wasn’t necessary. All the Leprachaun Zombies would need to do to influence the Times was to get the endorsement of the Democratic party and the Valentine editorials praising flesh-eating, shamrock-loving little people would come rolling in.
The 8th Avenue group pointed out that the Times was getting lots of attention, while media organizations that actually, you know, have a concept of decency and fairness, have gone unsung. Surely the Times did not represent the mainstream of American thinking?
Using a phone number written on a cocktail napkin that a past-his-prime waiter at Tavern on the Green provided me, I got in touch with Sarah to see what in the world was going on. And last week I finally go to talk on the telephone to Sarah Boxer, a tired hack who once learned how to be a journalist with integrity but said, “I don’t look at myself as one now.”
Why did she write such utter crap? When was she going to expose the Zombie Leprachauns that control the Times?
She was surprisingly frank. The Times had changed her. When her time with the Times began, she said, “People surprised me with their warmth and how much they cared about actually getting the truth from their news.” But as time passed, she said, “I felt that hey, they’re not the ones paying my salary: the Zombie Leprachauns are. So screw ’em.”
“Me and my journalistic colleagues”, she said, “we generally agree on the important things about reporting: spelling; grammar; having a killer lede.” But there is one important difference: “Other journalists think checking facts and not floating unsubstantiated rumors that might get people killed are an important part of journalism. I have my questions.”
Now that seems genuine.

Late yesterday I re-enabled everything on the site, including status display code, so everyone should be seeing their actual Ecosystem level on their blogs again. At this point, I still have not been able to isolate why I see Apache child processes exploding in memory size.
But, I have a cron job running that checks every minute to see if one has, and restarts Apache if needed. So the end result is the problem is masked, but not solved. So, assistance is still welcome; see this post for the full details of the problem…

Folks:
Here’s the latest on the performance problems I’ve been tracking. It turns out that it doesn’t appear to be directly related to load of the status display. The following is the technical details of what’s going on: I’d very much welcome any Apache or PHP gurus’ input on how I might proceed to debug the problem.
The basic issue is that a few times a day, an Apache httpd child processes suddenly explodes in memory size and consumes all available memory. The behavior is sudden, not gradual: within a few seconds or a minute the process swells to several orders of magnitude larger than its usual size.
Listed below is a ps aux; the first httpd is a ‘normal’ one, the second is the problem child:
nobody 23166 0.0 0.1 17572 2148 ? S 16:06 0:00 /usr/local/apache/bin/httpd
nobody 23167 1.4 63.7 2245624 1315584 ? D 16:06 1:00 /usr/local/apache/bin/httpd
Generally only a small number of child processes display this behavior: often just one at a time.
The system setup is Linux with Apache 1.3.33, PHP 4.3.10 and MySQL 4.0.22-standard. I am not running mod_perl.
The vast majority of traffic on the site is PHP scripts, some of which access the MySQL backend. I strongly suspect a problem in one of the scripts, but have not been able to identify specifically which one. I added the ‘%P’ variable to my Apache log file and am now able to identify the specific requests that the Apache child process which explodes was handling prior to the error, but thus far no pattern has emerged (different scripts appear last each time). I suspect that the last entry I see in the log is the last successful request, not the one that causes the problem. (I am aware of log_forensic, which I learned would provide log output of a request *before* processing, but my skills are not sufficient to make me feel comfortable rebuilding my Apache server to include it at this time).
I am working around the issue with a cron script that checks if a child process has exploded and then restarts Apache if needed; this helps to mask the issue but is obviously not a fix. I have considered setting max_requests_per_child to a non-zero value but based on my understanding I doubt it would help (given that this is not a gradual leak but a case where the process goes wild in the middle of a request).
I recognize that it is unlikely that this problem can be diagnosed-at-a-distance, but would welcome suggestions on debugging tools and techniques which might help me narrow down the problem area. In particular: other than log_forensic, is there a way to truly see what that child process was doing when it went rogue?
Any and all suggestions are appreciated…
Update 1/16: Thanks again to all who have suggested additional debugging techniques. I tried a few of them this morning and have gathered additional data. None of it has provided an “aha!” moment to me as yet, but I will post it here so more skilled gurus than I can examine.
Output of lsof for rogue httpd process pid 10990
Output of ‘strace -q -f’ for rogue httpd process pid 10990
Output of ‘cat /proc/PID/maps’ for rogue httpd process pid 10990

Folks:
I’ve been attempting to chase down some performance issues around these parts for quite some time now. I am coming to believe that the Ecosystem status code which I provide for folks to put on their blogs may be the root cause of the troubles.
The problem is that, if you have the status Javascript code on your blog, every single time you get a hit on your site, I get a hit on the database here. And these days, it is becoming difficult to convince my server to stand up under the strain.
So: for the moment, I have altered the Javascript status code to simply return a message such as the following:
Profile
in the
TTLB Ecosystem
The ‘My Profile’ link will go to your details page. This still gives you a direct link to your details page, but avoids a hit on the TTLB database, which was the big problem (I think).
I’d welcome feedback on this change: is this a major disappointment? I need to make a decision as to how much it is worth investing to save that one feature, so I’d appreciate responses/thoughts.
Thanks all…
-NZB
Update 1/13: Thanks for the feedback, all, and please keep it coming. A bit more information: I seem to have tracked the problem down to a sudden loss of memory on the server, which seems to be caused by a single Apache thread suddenly going wild and grabbing a huge amount of available memory. Anybody with thoughts on why that might happen, please chime in!

My Tech Central Station column is now up, and provides some hard data examining traffic statistics at top political weblogs leading up to and after the election.
On a related note, if your weblog is registered with the Ecosystem and you have a publicly-visible SiteMeter counter, check your details page and you will now see a second graph, which shows your average daily visits over the past month.

Rainfall.gif”>
Wow. Recognizing that what we’re experiencing is nothing compared to a genuine natural disaster like South Asia’s tsunami, I still have to say, it’s raining like a sumbitch out here in Southern California.
After making the dubious decision to go out this afternoon on some errands, I quickly realized that it was pouring rain harder than I’d ever seen in down here in the sunny OC. And we take our sunshine seriously in these parts: it’s like, mandated by statute.
Anyway, being a geek, I wanted to know just how much rain we’ve gotten so far in 2005, and how that compares to our annual averages. As it turns out, we’re at 4.29 inches as of Sunday night (and climbing) according to Weather Underground. Based on historical averages, it would take until February 13th to accumulate that much rainfall — and we’ve done it in nine days. (See chart).
And, our total average annual rainfall is only 12.81 inches. So we’ve received over a third (33.5%) of our total rain for the year in the first week and a half.
Yikes…
Update 1/10/05 am: Revised to reflect the final total rainfall for January 9th and added percentage of annual average…

As a part of interest in promoting Sumatran coffee as a method for helping the long-term recovery of the tsunami-affected region, I reached out to several coffee-focused websites and forums and asked for their input and suggestions.
As it turns out, Kenneth Davids of Coffee Review was just finishing a roundup of Sumatran coffee, including reviews of 12 coffees (2 of which are Fair Trade offerings from Aceh province). Drop by Coffee Review and find yourself your next brew!

Kevin McCullough notes that Army Maj. Michael Cohen, a doctor who blogged the aftermath of the Dec. 21 mess-hall bombing, been ordered to shut down his blog.
Cohen’s site now contains the following message:
I have some very unfortunate news. Levels above me have ordered, yes ORDERED, me to shut down this website. They cite that the information contained in these pages violates several Army Regulations. I certainly disagree with this. However, I have made a decision to turn off the site pending further investigation as to whether or not I have violated these Army Regulations.
I’d welcome thoughts on this, particularly from those in or from the military…

Thanks to all who have submitted requests for Ecosystem cleanup items. Just wanted to let everyone know that progress is being made, even if it isn’t immediately visible. Over the past week I’ve devoted quite a bit of effort to improving a lot of the behind-the-scenes validation and tools that I use to keep things running smoothly, and will soon start crunching through the list of cleanup requests. One step at a time…
NZB