Gallup Poll: Great Example of How to Bias a Social Science Study

Imagine you receive a phone call from a pollster. When you answer, the person on the other end asks, "For whom do you plan to vote in November: The honest and trustworthy Mr. Smith, or his lying, cheating, and disgusting opponent Mr. Jones?" Quite obviously, the pollster is trying to elicit a particular response.

This is a technique called "push polling," and it's actually far more of an advertising campaign than an attempt to discern the will of the voters. Nobody would take the results of such a poll seriously.

Gallup, a well-respected pollster, recently released the results of a poll on race relations that also asked African-Americans a loaded question: Can you think of any occasion in the last 30 days when you felt you were treated unfairly in the following places because you were black? (They then asked specifically about stores, restaurants, the workplace, etc.)

Just under half (46%) of African-Americans responded "yes" to being treated unfairly in at least one place during the last 30 days. However, because of the way the survey was designed, this result is nearly meaningless.

First, the question, by specifically focusing on race, is designed to elicit a particular response. A much better question would be: "Can you think of any occasion in the last 30 days when you felt you were treated unfairly?" If the question leaves the race factor out, it will likely yield a more accurate response.

Second, and far more importantly, there is no "control" group. The survey has no baseline data with which to compare the responses of African-Americans. It does not ask whites, Asians, Hispanics, or any other ethnic group if they have been treated unfairly. This is a serious oversight. People can be treated unfairly for many reasons, not just racism. Bad experiences with bosses, customer service, or the police are a universal phenomenon.

Consider if Gallup had asked white people the same question. And for the sake of argument, imagine that 20% of white people reported being treated unfairly in the past 30 days. That would mean that of the 46% of African-Americans who reported being treated unfairly, nearly half shared a similar experience with white people. Therefore, the fraction of African-Americans who were treated poorly for reasons possibly attributable to racism was actually 26%, not 46%.

One of the keys to successful social science is to ask unbiased survey questions*. The lack of a control group is just sloppiness. Gallup should know better.

---

*For your edification, here are examples of other biased survey questions:

(For Seattleites) Are the Seahawks the best team in the world or the universe?

(For Chicagoans) Is deep-dish pizza better than that thin-crust crap they eat in New York?

(For New Yorkers) On a scale from 1 to Chris Christie, how much do you hate New Jersey?