Bad Research example: Trump’s Media Survey

This is an example of how to analyze research as related to my blog post Recognizing Bad Social Research or News Articles.

I swear I’ll analyze good examples too! But the bad ones are just such fun. 🙂

The other day I read a survey on Trump’s website that hurt my soul. It is leading, biased, confusing, and faulty.

Let’s start with the good (it won’t take very long).

 

The Good

Some of the questions have these choices:

• Yes
• No
• No opinion

Having no opinion as an option is good (but many of the questions needed a more rigorous scale, this is so black and white they don’t allow for a spectrum of emotions. Darn it I said I would do the good first!)

• Which television source do you primarily get your news from?

A simple data gathering question. It needed more options though.

• “Do you use a source not listed above?”

Hey! An other box. So that people can indicate other  news sources. (Pity it doesn’t let you answer the same questions about the news sources you added. It should also be written as “If any, what other news sources do you get your news from?” I’m a stickler for thoroughness.)

 

The Ugly

We’re skipping the bad on this one folks.

The biggest problem with this survey is black and white absolutism. It features a strong “Us vs. Them” and “If you’re not with us, you’re against us” mentality that is quite alarming. People’s opinions exist in a broad spectrum. Research methods should account for that. They often allow the “neutral” (no opinion) but rarely allow the “other.” This is a breaking of my research rule #2:

2. Did they ask non-biased and unleading questions which allow for neutral and other opinions? Did they observe the subjects in a way that minimally effected their behavior? (Research Methods)

The questions also group the media (a broad range of people, organizations, and opinions) into one single entity. Making answering the majority of the questions quite awkward.

 

Recruiting

Another problem is recruiting. They are not following my research rule #1:

1. Did they recruit enough people from all appropriate groups to represent the population? (Recruiting)

If the black and white doesn’t turn people off, then the length will. How is this getting distributed? A link on a site? A select email list? They aren’t offering any money to finish so only those with an extreme vested interest (for or against) will take it. The sample of people who eventually finish this survey is not representative of the American population.

 

Questions

Now into a little nitty gritty, I tried to find the worst offenders. I could write a page on each question, but I shall refrain.

• Do you trust [Insert news agency] to report fairly on Trump’s presidency?

This should be a matrix table, “How much do you trust the following news sources?” (Mentioning Trump is biasing it, if they trust the news source, they trust them to report accurately on Trump’s Presidency).

Fairness is often used instead of accuracy. They are not concerned about the news agency’s accuracy, just if they talk nice.

• On which issues does the mainstream media do the worst job of representing Republicans? (Select as many that apply.)

Oh, let me count the ways. It should be “How well does the mainstream media represent republicans in the following areas:” Even that I don’t like, because it lumps republicans in all together as if they are carbon copies of each other. It doesn’t allow for people to think the media is doing well. Their data will only tell them how many people were negative, not how many were neutral or positive or no opinion!

• Do you trust the mainstream media to tell the truth about the Republican Party’s positions and actions?

BTW, who the heck is this mainstream media? Is that Facebook? Twitter? Instagram? The TV? What? They start asking these questions without making sure that the respondent understands exactly who they mean by “mainstream media.”

My hurt brains so much right now.

• Do you believe that the mainstream media does not do their due diligence fact-checking before publishing stories on the Trump administration?

Ah the double-negative triple pirouette. Double negatives, does selecting no mean “No, I do not believe that the mainstream media does not do…” Oh sheesh. The poor respondent is getting a headache!

• Do you believe that the media unfairly reported on President Trump’s executive order temporarily restricting people entering our country from nations compromised by radical Islamic terrorism?

That question is very biased. It is attempting to make the respondent feel like if they agree with the media in any way, they want to endanger our nation from the radical islamic terrorists. It should have been an open-ended written this way:

• Do you believe that the media inaccurately reported on President Trump’s executive order temporarily restricting people entering our country from (names of countries)?

It should also link to the text of the order.

• Were you aware that a poll was released revealing that a majority of Americans actually supported President Trump’s temporary restriction executive order?

Wow. This question is serving as a report of prior research. Can I get a link to that research? And that poll revealed that a majority of Americans POLLED supported etc. I’d want to see their recruiting method. This mini-report breaks my rule #3 of research:

3. Did they report on the data itself or extrapolate it to say more than it said and ramble on with their opinions? (Reporting)

• Do you believe that political correctness has created biased news coverage on both illegal immigration and radical Islamic terrorism?

They throw words around with very subjective meanings (like political correctness). Should be asking separate questions about illegal immigration and radical Islamic terrorism. Also, by putting them in the same question, they imply a connection (some say there is a connection, some not, that’s some other research I personally need to look into).

• Do you believe that contrary to what the media says, raising taxes does not create jobs?

Ok, they’re asking about belief now (good). But it should be written as such:

Do you believe that raising taxes does not create jobs?

Also, huh? Darn. It’s still biased. We’ve got the double-negatives again too.

• Do you believe that the media wrongly attributes gun violence to Second Amendment rights?

Because if you don’t you’re wrong! That’s what this is implying. I’m done rewriting this survey because if they were my client I’d be going back to them to ask what are the high-level research questions they have, what do they really want to know? How much does the public believe the media?

This survey, if it were meant for understanding rather than generating ammunition for political debates, should be written to understand the opinions the public has about the news, their demographics, and it should be recruited in such a way to get the best representation.

BTW, what the heck does “Our Movement” mean? Republicans? Trump? Conservatives?

Redundancy is a huge problem with this survey. The next series of questions essentially asks “Do you believe the media does evil things to try to stop us holy warriors of truth and justice?” in one way or another.

• Do you believe that the mainstream media has been too eager to jump to conclusions about rumored stories?

This is actually decent, it should also be a how much though.

• Do you believe that if Republicans were obstructing Obama like Democrats are doing to President Trump, the mainstream media would attack Republicans?

More us vs. them.

• Do you believe that the media uses slurs rather than facts to attack conservative stances on issues like border control, religious liberties, and ObamaCare?

I also don’t appreciate them tying religious liberties to things like border control. Another example of absolutism. Let every person be an individual. Not a cookie cutter. This survey was designed for extremist Republicans to back up extremist Republicans.

 

End Questions

A few questions at the end are shifted toward how they feel about Trump’s method of communicating with the people, but does it in a way that makes him seem like a holy warrior.

• Do you agree with President Trump’s media strategy to cut through the media’s noise and deliver our message straight to the people?

A good thing to try and found out, but questions should be written more objectively. “The almighty Thor cuts through the media noise to rescue the people!” Oh dear. Flair and drama do not belong in surveys.

• Do you agree with the President’s decision to break with tradition by giving lesser known reporters and bloggers the chance to ask the White House Press Secretary questions?

How were these selected? Links please. Orient your respondent before you ask them questions.

At the end, it asks for Name, Email, and Zip code. They really should be asking questions about age, gender, ethnicity (location is still good), and socioeconomic status. And if at the end not all populations are represented, conduct more research.

 

Reporting

Can’t really write on reporting yet, but you can tell that from the way this survey was worded, the report is going to be quite biased in a lot of ways.

 

Conclusion

I’m very concerned that political research is biased all around. They should be hiring outside agencies to write their surveys, analyze the data, and report back to them. The objective should be to get an accurate view of the people, not have a blunt weapon to argue with (such and such biased research said such and such!).

Hope you found this analysis enlightening. Remember, bad data is worse than no data at all!

BTW, if you have concerns about this being a republican-focused survey, please send me a democrat-focused survey. I will gladly analyze that as well. Both parties are guilty of biased research methods.

Booyah.

-Thomas Fawkes