Recognizing Bad Social Research or News Articles

If you don’t have the time to read the whole article, ask yourself these questions every time you read some research or a news article:

  1. Did they recruit enough people from all appropriate groups to represent the population? (Recruiting)

  2. Did they ask non-biased and unleading questions which allow for neutral and other opinions? Did they observe the subjects in a way that minimally effected their behavior? (Research Methods)

  3. Did they report on the data itself or extrapolate it to say more than it said and ramble on with their opinions? (Reporting)

 

Now, for the meat:

We often use research (surveys, interviews, news articles) to evaluate our beliefs (or to prove a point). Much research out there in the world is faulty, biased, or lacking in very vital ways. Not a great way to educate yourself!

Since I’m a researcher by trade, I thought I’d put together a basic guide to analyzing the value of research. This is mainly with practical or social research, rather than scientific research (some principles still apply).

The first thing to remember is that research only says what it says and nothing more. When a report says “40% of Americans hate puppies” it really means “40% of Americans surveyed hate puppies.” Stark contrast there. Think about it.

The next important things to remember are:

  • Just because you agree with the research, doesn’t mean they had good research methods
  • Just because you disagree with the research, doesn’t mean they had poor research methods.
  • Just because they had poor research methods, doesn’t mean the result is wrong (it does mean the research is not a good reason to inform your beliefs).

The rest of this article is divided into 3 sections which reflect the three major phases of social research:

  1. Sources
  2. Research Methods
  3. Reporting

 

1. Sources-Getting the people

How many

When researching a certain population, one needs to get a good representation of all target audience groups (ages, socioeconomic status, political affiliation, region. Census data and other pre-research helps you know who these groups are).

The people that you choose to research are your sample. There’s a lot of math out there as to how many you need in a sample, but the important thing to note is that you need a good number of randomly selected people from each audience group.

How recruited or chosen

Every recruitment method has its drawbacks.

There are people who won’t take a phone survey, an email survey, or a pop-up survey. Ever. If these people are excluded, you’re skewing your results. People who are likely to ignore a survey are also likely to have certain opinions that needs to be in the research.

Money can get more of those people to participate. So, beware of research that is conducted for free, they only get the opinions of the really nice or the really opinionated.

Also, beware of research that uses only one recruiting method. If they only do phone invites, then they only get people who have phone #’s the researchers have access to (or people who have phones at all if the phone-less are potentially one of their target groups!).

 

2. Research Methods-Gathering the data

There are two main types of data you can gather: behavioral data and opinion data. Beware of research that tries to learn people’s behavior through questioning rather than observation. People don’t usually have an accurate picture of what they actually do or what they would do (we have distorted memories and high opinions of our hypothetical selves).

People will often say one thing and do another.

Behavioral research is best done through observation. This can be done in-person, through video footage, or even website analytics data (the measure that they know they’re being watched will change their behavior, beware!).

Opinion based research is best done through questioning. And OH BOY do people go wrong here! I’ve seen surveys with the equivalent of “Here is this really great thing, how great do you think it is?” with the following three options:

  • Pretty great!
  • Super great!
  • Extremely great!

I’m exaggerating. Slighty. But sheesh! My job day in and day out is to take the biased, leading, confusing, and faulty questions from my clients and translate them into actually usable questions.

If the research doesn’t let you in on the wording of their questions when presenting results, beware.

Questions might be leading or choices for answers do not represent all possibilities. It’s important that surveys have a neutral option in order to allow for those people who really don’t give a darn to not give a darn!

I was talking to Howard Tayler on Twitter about survey question writing when he offered this as an example:

c3gshvyucaaol-l

Do you see the problems in it? It’s cramming multiple responses into one choice. A terribly worded on at that. He offered this as his fix as a funny but equally biased alternative:

c3gsn17uoai578g

The sad thing is, I could expect an extremist democrat to actually put this wording in their version of this survey (Howard Tayler is certainly not an extremist, but a writer of science fiction and comedy).

One-on-one interviews (often used by journalists and news sources) answer questions deeply but not broadly, they get into the nitty gritty of a certain thought, but not into how many people think such a thing.

 

3. Reporting-Interpreting and publishing the data

This is likely the most important, since this is what most of us see when investigating a certain body of research. The important thing to remember (and beware if the reporting doesn’t say this) is that:

“Research only says what it says and nothing more.”

Feel free to repeat that a few times. 😛

How much is taken directly from the data, and how much is opinion?

When a report is filled with X number of participants indicated Y, you know that’s at least decent data. But, beware when only a few lines of the report are associated with the findings of the data and the rest is paragraphs and paragraphs of opinion, speculation, and broad exaggeration.

 

EXAMPLES

The biggest example in recent memory is the polls on who people were voting for last election season. The people who were likely to fill out or answer the polls (mainly Clinton voters) were not representative of those who showed up to vote (a lot more Trump voters than the polls).

Here are three examples of types of research I’ve encountered recently:

The news article

Mwahaha! The worst of the worst! A journalist will often handpick a few select individuals, often to prove a point or that they anticipate will agree with what they want to say (not representative of the population), they will then asked biased and leading questions, then will choose what they want to report (in or out of context), and will fill their article with paragraphs of speculation and opinion.

Not a good way to get data! Not all journalists are like this, and most aren’t trying to be faulty, but when accuracy is traded for speed, you have a holey target with nothing in the bullseye.

The phone survey

Ah, the phone survey. Have any of you gotten a call for a political survey? How many of your took it? How did you feel on the subject if you took it? Many don’t like taking these surveys, and many don’t offer any compensation, greatly decreasing the likelihood of good representation.

The intercept survey

The pop-up appears and through analytics we know how many see it and how many click x. Those who are likely to click on it are (like I mentioned before) either the really nice or the really angry. And having people at both extremes does NOT average out to an accurate representation.

 

No data is perfect

Every piece of research should have a list of cautions, exceptions, and caveats. No research is perfect. And one survey or set of interviews is never sufficient to fully understand a problem. Beware those who use one chart to describe the state of a nation. The data is always more complicated than that, and people are more complicated than that.

If you learn one thing from this, it is to take every bit of research with a grain of salt, and sometimes a gallon. Because bad data can be worse than no data at all.

Bad data can mislead you into a false sense of confidence in something that might not be true.

Booyah!

 

-Thomas Fawkes

Advertisements

Just keep learning the craft

The third in my just keep series is in some ways an extension of “just keep reading” because part of “just keep learning the craft” is “just keep reading about writing” (lots of grammar, editing, and how-to books). It also involves experimentation and getting feedback.

This step is how you transition from just a writer to a writer that keeps getting better. This separates the on-their-ways from the wannabes. 

The easiest part of this is to incorporate a steady diet of “how to write” content. You don’t want to inhale it, and you don’t have to accept everything you read as gospel, but as you read about writing your brain will start to fill with good techniques and you will think more critically about your writing.

I’ve read a LOT of how-to content. A few writers and how-to’s that are great to start with are:

That doesn’t even scratch the surface (my gosh I’ve read so many!). You definitely want to learn the tense and point of view  (POV) you’ve picked to write in (if “third-person limited past” doesn’t make sense to you, you better get reading). The point is not to read 100 how to write books and/or make a checklist of things you must read/learn before you can write, the point is to develop a consist habit of learning the craft. Be journey minded, not destination minded (that advice is useful everywhere).

One other great thing is that I reach a point when I’m sick of reading about writing and I scramble to my computer to get out my own words. It builds up until I can’t wait any longer. Others might not get this effect, but I do.

On degrees and classes… I say beware spending money on something that you could get for free. You can learn online for free. And practice is more important than study and that’s free too. Practice is one of the greatest tools to learning, and you can’t throw money at writing and expect to get good without practice. And much that you could learn in college about writing you could learn online.

Degrees are great for creating an external force to make you write, but sheesh that’s some expensive motivation! Maybe you could just work on building a good habit, or join a writing group where you have to turn something in weekly (writing groups are great for this, and I’ll do a post on those soon).

My college experience was not the greatest. Much of what I learned I could have taught myself (I have an English degree). I had one class on grammar that was absolutely revolutionary, but apart from that, yuck.

The most important thing to get from this post is this: consistently read how-to content, experiment with new things often, and read your fiction with a critical eye. Over years, you will find the principles and methods which help you to be a greater writer.

Go forth and have no fear!

Booyah.

Just keep reading

Although this is the second entry in my “just keep… “ series, this may be the first rule you should keep and the first rule you have probably kept.

(There are people who don’t read much who decide to write novels, maybe because they love TV or video games, but if you don’t have a passion to consume the art form you’re trying to create, I don’t think you’ll be very good.)

One of the most important reasons to just keep reading is to learn through osmosis. Passive consumption works as a form of unconscious learning. You begin to absorb the rhythms of story and prose from the writers that you read. Things begin to get embedded into your instincts and intuition that will come out in your writing. (One reason it’s important to read good fiction.)

You can also do what I call “active reading.” This is when you read fiction with a more critical eye, focusing to recognize the patterns on the page, sitting and analyzing a phrase that had a particularly strong effect on you and to trying to understand why it had that effect. If you come to learn why and how, you will then be able to use those tools and components in your own storytelling.

I find that it is very helpful to look back at a story that I’ve read and analyze it. What made me care? What made me feel? How did the writer achieve  getting me through that emotional journey? What worked for me? What didn’t? And, perhaps most importantly, what almost or only partially worked and why or why didn’t it?

As I ponder these things (and write them down), I am more and more able to use the things I’ve learned from my reading in my own writing. I let the fiction I read fill me until I just burst.

I won’t give you a random number of books a year or pages or hours a day, just that you should do it consistently.

Go forth and fill your mind with awesomeness!

Booyah.

Why “just keep writing” is the best writing advice ever

The most common writing advice I’ve heard is to “just keep writing.” When I was very new to writing I thought that such advice told me nothing. I had a million questions about writing, so why was that what I heard the most? Just keep trying? Just keep writing?

The key is that practice makes… better. The more you attempt a certain goal, the more able you become at achieving it. Each time you fail and try again, you get better. It took me years of trying and failing to build enough self control to get to the point where I was writing every single day. Not writing well, just writing at all. I wasn’t ready for more answers before then.

The natural man doesn’t have the drive to “just keep writing,” he looks for every excuse not to write. Every failure is actually a success, for each time you fail at writing consistently, you had also tried again. So keep trying, keep failing, and watch the failures become more and more spaced out. The days where I didn’t have the drive to write became further and further apart (habit tracking apps are great for helping with this).

5 years ago I would  have never dreamed that I’d be willing to wake at 5:00 AM every single day (including weekends) to write for 2 hours to meet my writing goal.

It’s what transitions you from someone who wants to write to a real writer. Wannabes talk or complain about writing. Real writers, well, they actually write. Once you have mastered the principle of “just keep writing” and you have a consistent writing habit, then you will start to discover your own answers to your other writing questions.

The next piece of advice I would offer is to “Just keep studying the craft.” you don’t have to wait ’til you master  “just keep writing” to start working on this skill, but “just keep writing” is more important. I’ll talk about studying craft in a future blog post.

So, here is you first and most important piece of advice:

Just keep writing!

What that really means is “if you fail, try again. This rule does not change.  No matter the number of failures.”

Booyah!

 

How to overcome getting overwhelmed and work efficiently

I sometimes have a problem with getting overwhelmed and feeling anxious about all that I have to do. Over time, I’ve come up with these ludicrously simple (and ludicrously difficult for someone who has a hard time focusing) rules and tools.

Here are my rules of efficiency:

1: Focus on one task at a time

Focus on one task at a time, work on it for at least 30 minutes (use a timer, it helps) or when you need someone else’s input, then make the contact (in person, set up a meeting, make a call, or send an email).

2: Focus on a task for at least fifteen minutes to get momentum

I find that if I focus on one task for at least fifteen minutes, I get into the groove and the rest of the time comes much easier.

3: List all projects and subtasks, put aside and focus

List all projects and sub-tasks needed to get those projects done. Put this list aside when focusing on 1 task (this helps keep you from feeling confused and overwhelmed because you won’t have to remember it all, there is a list!).

4: Multiply expected time by 2 or 3

Assume that everything will take 2-3 times as long as your first impression. Don’t commit the planning fallacy!

5: Listen to non-distracting music

Listen to wordless, calm music like this meditation track (it must be music that will help drown out distractions but won’t be a distraction itself).

5: Wait to check texts or emails

Do not check texts or emails until you are between tasks.

6: Take breaks

Take short breaks (5-10 minutes) every hour or two and long breaks (15-30 minutes) every three or four  hours.

7: Spend 5 minutes pre-task getting pumped up

Spend 5 minutes before each working session to plan, visualize, brainstorm, understand and get pumped up for what you are going to work on next, write this in a paragraph or two on paper. It helps you to get in the mindset.  (Thanks to Rachel Aaron for this glorious tip!

8: Be patient with yourself

Some days you can’t do it all. Forgive yourself. You’ll have to shorten your list and simplify your life as you go along. But if you stress about everything you have to do, the guilt will make you explode!

There you go! Quit lying to yourself about your ability to multitask. You can’t. Become a monotasker (someone who can actually focus). Quit stressing out about all that you have to do, and do one thing at a time. You’ll get a lot more done.

You can do it!

Booyah!

-Thomas

Note: Article Updated Mon Dec 5 2016

When the worst thing that CAN happen will ALWAYS happen (in fiction)

I hate fiction where the worst thing that can happen, will always happen.

A lot of writing books tell you “Make it worse! Did you? Now make it worse, again!” This is done as a technique to increase tension in a story.

But when the worst thing and the worst thing alone happens consistently, it starts to erode my suspension of disbelief (or, the believability of the story goes down). No matter how bad someone’s life is, no one has the single worst possible thing happen to them in every and any situation.

(Unless you’re Job from the Bible, but thank heavens that was only one story and not a trilogy!)

It doesn’t make logical sense for a character’s parents to die, then his mentor, his dog, his best friend, then his girlfriend turns evil, his leg is chopped off, (actually, this is starting to sound like an interesting story and a bad analogy). I’m trying to say it doesn’t make sense to look at all the possible outcomes of a scene and to ALWAYS pick the single WORST potential outcome. There are times, many times, where you want to pick the worst, or one that’s pretty bad. But to feel a little more REAL, you need to sometimes pick the good outcome, or, more often, the OK outcome.

This problem, however, usually isn’t enough for me to shut a book. Though my suspension of disbelief and my trust in the validity of the story are damaged, I usually see that as a plot problem. If the writer has done his most important job, I still care about the characters enough to want to know what happens to them.

The true problem comes from thinking “What’s the worst?” to thinking “What’s the next logical thing to happen with everything here?” Part of the problem is foreshadowing, but sometimes, you wonder why the enemy arrow always hit’s the throat. Or the character makes the wrong choice every time. Or the bad guy wins every time. Mix it up. Make it make sense.

And most of all, don’t feel you have to always make it “THE WORST.” Also, don’t feel you have to listen to my rules either.

 

Booyah!

How to discipline your creative brain

In short, rid yourself of the mindset that you have a Creative Brain and an Analytical Brain. Create a new mindset where you have 1 brain: A well-trained, well-practiced, and highly-disciplined Creative Brain.


In talking about Heinlein’s rules, Dean Wesley Smith says that writing is a creative activity, that we shouldn’t grind all personality out of our work by rewriting. He also says that we must read constantly; we must analyze the writing of others; we must study craft, prose, and, most importantly, storytelling techniques; we must actually write; and we must finish what we write.

Heinlein’s Rules:

  1. You must write
  2. Finish what you start
  3. You must refrain from rewriting, except to editorial order
  4. You must put your story on the market
  5. You must keep it on the market until it has sold
  6. You must study the craft (I added this, from comments Dean made)

In reading his book, a metaphor took form in my head. Currently, most people view their brains as being the creative brain and the critical brain.

The creative brain is like a hippy, wild and free, running around with no forethought, no discipline, making scattered notes on the piano or splattering paint all over the walls, all wild and random and instinctual expression.

The analytical brain is like a librarian, strict and stern, concerned with order and organization and cleaning up the messes of others, following behind the hippy and trying to find what it considers “good” music or an “artistic” paint job based off of what others have said before. Whatever fits the librarians prescribed beliefs survives, everything is muted or painted over.

This is a BAD mental model. It makes it so that the hippy bears no responsibility, and the librarian has all authority but is often following the trend so much that whatever “art” is left is drab, boring, and so much like everything else.

A new mental model is needed. Instead of two halves, there is one whole. An artist. A musician, a composer, a dancer. The artist has all the creativity of the the hippy but doesn’t lack foresight. The artist has all of the discipline of the librarian and none of the generalizing prescribed notions.

The artist finds the balance between expression and creation, discipline and focus. It is to take the hippy and offer focused practiced and study. Not to see what is commonly accepted by other professionals (like the librarian), but to learn how to create an experience for the consumer. A powerful emotional experience, tested and learned step by step, with focused practice. Not wanton splattering of random ideas nor the arbitrary slaughter of all but the most statistically accepted by values that seem to exist “just because.”

Dean Wesley Smith says:

“Rewriting is when you do a sloppy first draft with the intent of “letting it sit” (dumbest thing I have ever heard) and then “fix it” later.

This lazy attitude is the attitude of the hippy and the librarian (which actually ends up being a LOT MORE work). Have the attitude of the Artist. Dean says:

“If you tell your creative voice to do it right the first time, the story won’t be broken.”

Train your creative voice. Read a lot. Consume stories. Study how stories are written. Use that in your writing. Practice. Practice. Practice. Become the writer you were meant to be.

 

p.s. Read everything by Dean Wesley Smith. His words about the business of writing have transformed the way I approach the writing career.