Oddly Together

You Can’t Trust Most Polls or Surveys

Internet polls are fun but rubbish. Formal surveys conducted by so-called experts aren’t much better. If you disagree, consider this: A poll I conducted for Betanews asking “How would you identify yourself as a computer user?” puts more than 25 percent of respondents as Linux PC users and less than 61 percent as Windows PC users. Do you believe that? I don’t. But I do believe, as early results indicated, that there are more Betanews readers identifying themselves as Linux PC users than Macheads. But more than one-quarter are Linux users? Perhaps in some alternative universe, but not this one./p>

The polling started innocently enough. On November 12th, I blogged: “This film is rated PC: No Macs were used in the making of this video”, praising a Microsoft marketing video. I also inserted the aforementioned poll. Respondents had four choices: Windows PC, Macintosh, Linux PC and Other. Three days later, I awoke to 682 votes, with 507 for Windows PC, 77 for Linux PC, 70 for Macintosh and 19 for Other. That worked out to about 76 percent Windows PC, nearly 12 percent Linux PC and more than 11 percent Macintosh. The Windows PC response was a little lower than I expected, but not by much.

An Unbelievable Result
Linux PC surprised me, so I embedded the same poll in new post: “Do more Betanews readers use Linux PCs than Macs?” I predicted: “This post may marshall the fanboys and skew further results”. That same day, November 15th, a poll commenter simply identified as Jesse responded (comment grammatically corrected):

I have a feeling this poll has massive amounts of bullshit in it, provided by Betanews’s core audience of 13 year-old Xbox Live kids. More people browse the Web on iOS than Linux, so honestly it’s not even a question. Would someone would mark Linux when they’re on Windows just to make Apple look worse? Probably. You people are that sad.

I don’t agree with the “sad” dig at my readers, but Jesse and I are otherwise in agreement about the results being skewed—well, with a caveat I’ll explain in a few paragraphs. As I write there are 1,961 votes:

• 1,191 for Windows PC (60.73 percent)
• 496 for Linux PC (25.29 percent)
• 236 for Macintosh (12.03 percent)
• 38 for Other (1.94 percent)

That’s a fairly good size sampling, but it’s unqualified. I don’t know who the people responding are. I also don’t have handy information on which Websites or forums link to the poll. Could there be a rallying among Linux blogs and forums? The poll uses cookies to prevent repeat voters, but it wouldn’t take much tech savvy to get around that. PollDaddy provides just basic tools, even with my $200/year Pro account, for analyzing data. IP filtering is revealing. There are 1,807 IP addresses, with the largest number of votes (22) coming from bellsouth.net string. Microsoft.com accounts for another 7 IPs.

Geographic analysis is surprisingly useful. Only 958 responses are from the United States. Nearly 64 percent for Windows PC, about 19 percent for Linux PC and 14.5 percent for Macintosh. Canada: 158 responses. United Kingdom: 132 responses. If you believe the Netherlands’ 16 votes, then an equal number—43.75 percent—of people identify themselves as Windows PC users and Linux PC users. Bulgaria’s 59 votes come out to 83 percent Linux PC users (I just might believe that). To my surprise—and I see this as good finding—people from 94 countries responded to the poll.

Questioning Polls and Surveys
Now for that caveat: I didn’t ask which PC operating system people use but how they identify themselves as personal computer users. The poll is specifically meant to measure sentiment, which is about the only value I see coming from any poll or survey. The results reflect respondents’ attitudes rather than what they actually use. But sentiments of whom? That’s the data problem with this poll and many others like it.

Internet polls are suddenly the rage, and the results are easily shared on social networks (yeah, yeah, Facebook). Just click “Like”. The results are too easily believed as they spread. But the data isn’t necessarily representative of anything, and results can be manipulated. Take this post. Based on the data I could have written here or at Betanews something sensational about the surprisingly large number of Linux users or dwindling number of Windows users. I’ve got the data, backed up nearly 2,000 respondents from more than 90 countries. That makes the poll seemingly credible. But it’s not. I know from my everyday dealings, where I often either observe or ask what operating systems businesses or consumers use, the majority run Windows. Virtually no one uses Linux. If by some strangeness, I’m wrong, I just passed up one of the biggest tech stories of the decade. But I’m not wrong, because the data is incomplete and respondents haven’t been properly vetted.

I see poll or survey data being manipulated or misreported nearly every day. Sometimes the fault is the interpretation (by the pollster or people reporting/blogging about it) or the actual poll or survey (Web metrics data is among the most problematic). For online polls, people self-select to take them. Good pollsters weight the data to compensate for how self-selection skews the data, but if the data is relatively clean why should math massaging be necessary? Phone polls/surveys can be fairly random in their representation of the target populace, unless the respondents have been prequalified. Sentiment is another problem, because it can change, sometimes dramatically. Imagine a poll taken about Americans’ attitudes towards muslims on Sept. 10, 2001 and one taken two days later.

CBS News’ Airport Scanner Poll
On November 15th, CBS news posted story “Poll: 4 in 5 Support Full-Body Airport Scanners”. The headline actually misstates the data. CBS News asked 1,137 U.S. adults by telephone: “Should airports use full-body airport scanners?” Eighty-one percent said yes. But agreeing that airports should use the scanners isn’t the same thing as supporting them. It’s this kind of nuance that a comprehensive survey would seek to reveal. For example, I may believe that Barack Obama should be president but not support all of his agenda. You might answer a poll saying the government should use whatever means necessary to fight terrorism, but that wouldn’t necessarily mean supporting surveillance of you or your neighbors.

Timing is important, too. CBS News conducted the poll November 7-10, just as full-body scanners were coming to major airports—San Diego’s scanner(s) arrived in August, among 11 airports planned for this year. Presumably, most respondents haven’t been through a full-body scanner. How would they answer if asked November 29th, right after the busy Thanksgiving travel weekend?

CBS News’ poll result is startling. For most of November there have been news stories every day regarding conflict and controversy about full-body scanners (Then there was the high-profile incident here in San Diego just last week). The U.S. Senate held hearings on airport scanners just two days ago. Has CBS News uncovered some media conspiracy? The news stories suggest people are pissed about the scanners. In contrast, the poll indicates most Americans believe that airports should use the security devices. A broader, well-crafted survey might sniff out the differences and truly be newsworthy. So there remains uncertainty between the poll and the news about the extent of Americans’ outrage or acceptance of full-body scanners. That the poll raises the question but offers no real answer makes the findings unreliable.

Can you really trust polls or surveys? My answer is no for the majority of them. In a future post, I’ll offer tips on how to conduct reasonably reliable polls or surveys.

Photo Credit: Greg Grieco/Penn State University

[Editor’s Note: This post was moved from joewilcox.com to Oddly Together on May 21, 2011.]

Do you have a story about polls, pollsters and survyes that you’d like told? Please email Joe Wilcox: oddlytogether at gmail dot com.

The Comments, No-Comments Debate gets Noisy

People love catfights, which perhaps explains some of the interest in the comments/no-comments debate I’ve been having with Mac blogger John Gruber. It’s a pseudo-debate, really, since the only engagement is blog posting. John and I haven’t directly communicated.

I started it all, by calling out John for not having comments on his blog. I told him to “Be a man,” which I actually meant with some backslapping good nature. But some people are morally offended. Stacia Van Doll reblogged the post as: “QUIT being a douchebag Joe Wilcox.”

She writes:

Does this guy Joe Wilcox think he’s some sort of ancient barbarian man or something?Give me a break. What’s wrong with the world today is everything that stems from shallow, unintelligent macho bullshit like Wilcox’s.

No one has ever accused me of being macho before. I can’t bring myself to be insulted. Whoa, is this irony or what? Look what popped up on Twitter seconds after I wrote the first sentences of this paragraph—from Mark Dagon Hughes: “What I want to know is, what has @joewilcox ever done that was so manly he can make a mortal insult to someone over fucking blog comments?” 

Overnight, John answered my post with “I’ll Tell You What’s Fair,” and I responded with “Blogging: Is Curation or Comments Better?” With my post, I removed Disqus commenting from Oddly Together, as a 14-day experiment trying it John’s way. John described comments as “cacophonous shouting matches” and his Daring Fireball blog as a “curated conversation.” I want to see how right John is and to stir up some meaningful discussion about comments and whether they add or detract from the storytelling. I don’t expect a pat answer, but it’s only Day 1.

Other folks are piping in about the value of comments. I tweeted these earlier today:

• Marco Arment: “Comments”
• Ian Betteridge: “John Gruber, Joe Wilcox, and why comments are anti-Web”
• Matthew Ingram (GigaOM): “Yes, Comments Can Get Noisy, But We Like Them Anyway”
• Doreen Marchionni: “Blogs, Commenting and the Art of Interaction”
• Ravi (Arhen Code): “Comments isn’t free: Gruber vs Wilcox”

A Refreshing Start
My first experiences with curation are surprisingly positive. For the six days Disqus was active on the “Be a Man” post, the majority of comments were rude or condescending. With comments off, I started getting responses by email. Nearly all are civil and well articulated, regardless of their position about commenting or reaction to the “Be a Man” post. In a future post, after asking permission, I will share some of the emails. I treat all emails as private conservations unless the other party has given permission for the contents to be revealed.

Today, I also extended the interaction, by tweeting more often than usual—and predominately about the comments/no-comments topic. I see Twitter as a potentially good alternative to commenting for three reasons:

1. The 140-character limit restricts just how much people can say. They have to be concise.
2. Tweets are more visible than comments tucked away in some blog post, which should make most people more restrained.
3. Twitter easily leads to conversations that many people can join, which extends the reach of the original narrative and its storytelling.

That leads to a confession about comments. Much changed in my thinking about them between my June 10 “Be a Man” post and John’s June 16 “What’s Fair” response; I had already been rethinking comments’ value. It’s a major reason why I so suddenly changed direction and turned on the turn-off comments experiment. Catalyst: Nicholas Carr’s new book The Shallows: What the Internet is Doing to Our Brains. Nicholas’ discourse led to my writing three related blog posts since June 10:

• “Internet Attention Deficit Disorder”
• “Of Course, Technology Changes You”
• “I Was Wrong About Apple iPad”

Nicholas’ book is having an unintended affect on me. I’m a writer, a journalist, storyteller—and it is in these roles I am interpreting The Shallows. Nicholas’ research raises questions about the Web as a training place for distraction. But I see media consumption’s purpose being immersion. The consumer immerses in a good book or movie. In that context, I’m troubled by comments as distractions to the narrative, which admittedly contradicts what I wrote in the “Be a Man” post about comments adding to the narrative. To be clear: Comments would be just one of many distracting elements someone would see on most any Website.

I’m not retreating my position, certainly not on Day 1 of the no-comments experiment. I have long opposed anonymous commenting. If you’ve got something to say, then don’t shout it from the back of the room. Stand in front. Let others see who you are. Don’t cowardly hide behind anonymous comments, which often are the most obnoxious ones. I am now questioning registered-user or moderated comments for many blogs, but not all.

Comments and Process Journalism
One place where comments do make sense, at least as I see them today, is when the Website encourages readers to participate in the storytelling process. TechCrunch practices what founder Michael Arrington calls “Process Journalism,” a concept he picked up from Jeff Jarvis. In summer 2009 post “Process Journalism and Original Reporting,” I explained that many TechCrunch news stories are posted incomplete—later to be extended and expanded through subsequent posts, as more information is available. Readers help shape the story by their comments, other interactions and tips. I further explained:

For TechCrunch, the process can be surprisingly good reading. I haven’t done an official count, but I’d guess that TechCrunch posts more unfinished stories than the more complete kind published by, say, the New York Times. TechCrunch stories evolve—and, I must assert, too often from unsubstantiated rumors. It’s a process I must grudgingly acknowledge that TechCrunch can be quite transparent about. The process, of the story unfolding over time, produces original content that often is interesting reading…[TechCrunch] readers participate in that process, through comments and other social media tools…Social media—reader participation in the evolving story—is crucial to process journalism.

This evening I asked Doreen Marchionni over Twitter: ”Do you see Conversational Journalism value to comments as part of ongoing stories; e.g. TechCrunch’s Process Journalism.” She responded: “You bet there’s value. Whether readers interact via comments or Twitter or whatever works for me.”

Doreen is sharing her “dissertation on journalism as a conversation,” and she has lots of good advice for bloggers or reporters. The distinction I made in my question is process—an ongoing story, as opposed to one that is finished. Comments part of an ongoing story can sharpen the reporting, make the narrative better reading and keep readers coming back for the next update, whether the main post or additional comments. Using TechCrunch as example, the process builds community and audience of loyal, participatory readers.

That’s a wrap. It has been a long day writing.

[Note, June 17, 2010: Original version stated that Doreen is writing her dissertation. She kindly sent a note informing that she completed the dissertation last summer.]

Do you have a story about Web comments that you’d like told? Please email Joe Wilcox: oddlytogether at gmail dot com.

Be a Man, John Gruber

If John Gruber allowed comments on his blog, I wouldn’t need to write this post, and it has been long-time coming. I considered writing it every time I read something outrageous at Daring Fireball but couldn’t directly respond because John doesn’t allow comments. Finally, this morning, I had enough.

John’s post “John Battelle on Apple’s Banning Google From iOS App Ads” is what set me off. I wanted to comment—to correct the competitive marketing lies asserted by Apple CEO Steve Jobs that John repeated as fact. Since I can’t respond at Daring Fireball, I do so here. If I’m going to respond here, I might as well express something held back for some time: My disdain for the brutal effectiveness of John’s storyline—where he hits and jabs others with sarcasm and spit but isn’t man enough to receive jabs back. The story is incomplete, because only one side is presented. 

I thought to write this post at Betanews but didn’t want to be accused of gaming the pageviews. Also, because of the more personal nature of this post, I felt it inappropriate for BN. It’s only marginally appropriate for Oddly Together, which, at this point, is a storytelling site in the making. I want to shift away from technology being the story to technology telling the story. But John’s writing is storytelling (that’s a compliment), and I have made blogging and journalism ongoing topics here at Oddly Together. So there it is.

Which Bullshit is More BS?
Over the past two days, there has been some controversy about the newest revisions to the iPhone OS (now called iOS) developer agreement. It must be a lawyers died-and-gone-to-heaven-contract. How often is a contract between two parties changed, without notice, every few months (and it sure feels more often than that) by one side but not the other? On June 8, All Things Digital’s Peter Kafka observed that Section 3.3.9 of the developer agreement seems to block AdMob from Apple’s platform because Google has a competing operating system. A day later, John Battelle wrote: “It’s Official—Apple Kicking Google out of iWorld”; John Gruber linked to the post and responded:

Bullshit. Google started this. It was Google that turned its sights on the iPhone. If AdMob had remained independent, they could still sell in-app ads on iOS. If AdMob had sold itself to Apple instead of Google, they could still sell in-app ads on iOS. If Google hadn’t declared war against the iPhone, AdMob could still see in-app ads on iOS. They made their bed, now they have to sleep in it.

There’s no question it’s a dick move on Apple’s part. But what’s the argument against it? That Google gets a pass for being dicks to Apple, and Apple ought to just sit there and take it?

If Daring Fireball accepted comments, I would have responded to John’s fantasy land assertions about Apple and Google there. John mimics Steve Jobs, who in March told employees:  ”We did not enter the search business. They entered the phone business.” Steve has since made similar assertion—that somehow Google encroached on Apple’s phone business. That’s simply not true.

Google bought Android in August 2005, about 18 months before Apple announced iPhone and nearly two years before the device shipped. Google effectively announced its intention to go into the mobile phone business when buying Android. Maybe Google’s intentions spoiled secret Apple plans, but there was nothing unclear about them. Responding to Google’s Android acquisition, in August 2005, IDC analyst Scott Ellison predicted: “Wireless is the next frontier in search.”

I was working as a JupiterResearch analyst at the time of the Android acquisition; there I repeatedly blogged about “anytime, anywhere, on-anything” computing. Computing and informational relevance was shifting from the PC to the cloud, with mobiles looking to eventually become the dominant devices. Mobile phones are personal and always with you—and, unlike personal computers, Microsoft doesn’t own the market (even in 2005, when Windows Mobile had more users and bigger marketshare). Google executives would have been hugely negligent by not pursuing wireless devices as the next opportunity for search. Google’s wireless device push started long before Steve stepped onto the Macworld stage in January 2007 and announced iPhone.

John’s Way or the Highway
It’s easy for John to revise history when there is no easy place to respond to him. Daring Fireball is his blog. It’s his voice. He is under no compulsion to offer anyone an easy mechanism for dialog or response. But his no-comments approach is out of place in an era when so many Websites or services provide discussion tools and encourage readers/viewers to use them.

I have to wonder why. Some reasons I see:

• From a storytelling perspective, John maintains a single voice, a single point of view, which helps keep the narrative cleaner and freer from distraction.
• As a writing style, John is freer to jab and punch pretty much anyone he chooses, which makes his wit and sarcasm more effective and even more enjoyable reading.
• The approach also means there is no direct way for those jabbed or punched to respond. Other folks can blog elsewhere as I am doing, but that’s separate from the narrative.
• John’s writing, beret of direct discussion or response, picks up more authority, establishing what some editors call the “God point of view” or “all-knowing point of view.”

John Gruber is not a journalist. He is a blogger and Mac enthusiast with a computing science degree. His posts are often biased in favor of Apple, and he makes no apology for this (that I’ve seen). He needn’t make one. Good writing is first and foremost about audience. John has identified an audience of like-minded Apple lovers, and he writes for them. There’s no rule all writing must be objective. He blogs about other topics, too, but clearly those that interest him. Good for him. John is a successful, self-supporting blogger, and there are too few success stories like his.

But there’s something about the no-commenting approach that irks me. John has whacked me and my writing a few times at Daring Fireball, but I couldn’t respond there. It was a one-sided argument with his supporter minons adding to the noise. (John has loyal followers; some of them are more than just readers, in part because they share the same passion. With no disrespect intended, I’ll call him mini Steve Jobs. Both men are cults of personality.) Then there are the many more posts like the one I respond to here, where I would want to add something to the discussion, whether agreeing or disagreeing with John. 

Something else: John denies his readers opportunity to better benefit from his writing. John is a good practitioner of stock-and-flow blogging. The flow are the many short posts, often with funny or biting commentary, linking to someone else. The stock is insightful analysis—John’s longer posts, and these would be good venues for discussion. Surely readers would like to react and to discuss. In July 2009, Missouri School of Journalism’s Doreen Marchionni wrote about the importance of “conversational journalism.” She offers good advice for John or anyone else writing professionally on the Web. By the way, Missouri’s J school requires Macs.

I allow comments at this blog, which in response to this post opens me to attack by John’s followers. The discussion battle will occur here, if there is one, but Daring Fireball will be free of any scorched earth. That’s the other advantage for John: His followers fan out in attack where others do allow comments. Should a comment battle occur here, for a time, the responses really will stand out. I recently migrated blogging platforms, changing links site wide. So most of the older comments are missing. Thirty days ago, I contacted Disqus and received e-mail: “We’re currently finishing up a tool to make this migration easily and I’ll make sure to let you know when it’s ready.” So hopefully sometime soon I can restore missing comments.

I’ve never met John Gruber. I only know him from his writing. I grew up in Northern Maine, where a man’s worth is his mettle. A man pushes out only as much as he can receive back. By comparison, I see John attacking from a fortified position. He can attack but not easily be assaulted, and, yes, many of his posts are attacks on others. Sarcasm and witticism are the ammunition. Maybe John has different values of what is a man. My values are clear. A man—hell, a good writer—doesn’t hide behind his assertions. He stands by them. Discussion and response test his assertions and expose him to more points of view. So I close with this challenge: Be a man, John Gruber, and allow comments at Daring Fireball.

[Update: John responded to this post on June 16: “I’ll Tell You What’s Fair.” I liked John’s response so much, I’m going to try blogging his way and turn off comments here for the next two weeks. John was a good man to respond. I’ve removed Disqus commenting and will see how the extended narrative works. I’ll report back on the experiment in two weeks.]

Photo Credit: Alex Hong

Do you have a John Gruber story that you’d like told? Please email Joe Wilcox: oddlytogether at gmail dot com.