Maybe you’ve heard of or lived with a roommate who never washed the dishes, who talked loudly on the phone late into the night or who stiffed you on rent. Not fun. Bias in our research isn’t fun either. It distorts the nature of the data we collect, analyze and share.
Our bad bias roommate skews our work towards a protected and anxious state rather than elevating our perception and ability to think critically towards an informed future. As we cultivate an awareness of research bias and work to limit its presence in our qualitative studies we have the chance to identify and reduce our personal and organizational prejudice as well.
Whether you’re leading or participating in a research project here are a few common forms of bias that I’ve noticed in my recent work along with some useful strategies to help mitigate the scope and impact in your work.
Cultural Bias “We the [Library] people...”
This subtle and prevalent type of bias represents the tendency to judge behaviors and intentions by the standards and norms within our own experiences. An example of this kind of bias occurs when I feel the impulse to create text labels that are familiar to library staff but are not widely known among a larger population of students and faculty.
Help:
We can overcome cultural bias by critically selecting a variety of represented and underrepresented populations at the periphery of our day to day library experience. We can also work to root out assumptions in our analysis by collaborating with another person or a group to identify common themes, errors and bugs.
Sampling Bias
This bias occurs when communication strategies, participant recruitment or survey distribution yields too narrow a slice of the target population for a study. For example, a series of tabling activities held in the same library location at the same time of day would only reveal the feelings of a small set of available patrons.
Help:
Varying recruitment pools, study dates/times, locations and activities can help reveal a broader range of feedback. Our Design & Discovery team just conducted several public tabling activities at a variety of locations on campus to get feedback on our new search designs. Over a 2 hour period we spoke with a random sample of 7-10 students and we also collected emails of students who were interested in participating in future usability studies. Double win!
Hawthorne Effect
When conducting focus groups, contextual inquiry, user interviews and usability sessions participants frequently alter their behavior when they are being observed or recorded. In a recent remote usability study a participant stated, “If I wasn't being recorded, I would just move on. I wouldn't use this website because it's really just too complicated.”
Help:
Be a good host or take a field trip and meet your participant in their home turf. Field studies conducted in a participant’s office, lab or residence are the best way to reveal their assumptions and behaviors that may keep them from engaging fully with our collections and services. If in-person observations aren’t possible, create a quiet environment free of distractions where you can speak with and watch a user.
Confirmation Bias
The granddaddy of the bias family, confirmation bias is one of the most prevalent and ubiquitous ways we distort inductive research. It is the tendency to interpret or overestimate evidence that supports a pre existing hypothesis in our research. Confirmation bias often occurs when collecting and analyzing data and leads to larger systematic errors in research.
Help:
To overcome this tendency, first stop trying so hard! Allow findings from your research to speak for themselves. Be inclusive, transparent and generous with your stakeholders throughout your research process by inviting differing perspectives to inform your research plan and analysis. Sit with uncomfortable findings and feedback and analyze without ego.
Halo/pitchfork Effect
This cognitive bias occurs when a researcher allows an overall impression of a person, product or service to filter how they record and interpret their actions. The duration and intensity of a user interview or contextual inquiry can be particularly susceptible to this form of bias. We may be charmed or repelled by a participant but the tipping point occurs when we overemphasize or ignore comments that relate to our primary research questions.
During usability studies, asking participants to describe rather than show their behavior can also create distortion. Participants’ memory of their actions is often incomplete and inaccurate.
Help:
Start an interview or usability study with observational questions with the phrase “show me…”. Ask users to verbalize their behaviors and intentions throughout a usability study.
Referring to a list of study questions and focusing on open ended follow-up questions help keep you centered on the research topic rather than getting swept up in a participant's aura.
Focus on and record what participants do or don’t do. Capture unprompted quotes that reveal intention and expectation.
Inference and Assumption
Bias is built on illogical, inaccurate or unjustified inferences (conclusions that something is true in relation to something else being true, or seeming to be true). Inference is supported by assumption (something we take for granted or presuppose, something we believe to be true.)
Help:
In Paul Hackett’s book, “Qualitative Research Methods in Consumer Psychology: Ethnography and Culture, he states, “...in the role of guiding the exploration of a study, hypotheses serve to refine both the nature of what the methodology of a study seeks to explore and describe, and, consequently, the nature and scope of what can be inferred from the results of the study.” (Hackett, 2015, p. 67)
Conclusion
While the bias roommate won’t be moving out anytime soon, knowing how it affects qualitative research will better unify our library’s work. Improving the quality of research methods and findings will help us to co-create user oriented solutions in our physical/online spaces, our services and our collections.
Further Reading
Hackett, P. (2015). Qualitative Research Methods in Consumer Psychology: Ethnography and Culture. London: Taylor and Francis.
Ho, Anna. (2015, June 29) Qualitative Research, Bias & The Hunt For Truth [Blog post]. Retrieved from https://medium.com/@smashingideas/qualitative-research-bias-the-hunt-for-truth-3a11599bb1ea
Kate, Crawford. (2013, April 1) The Hidden Biases in Big Data [Blog post] Retrieved from https://hbr.org/2013/04/the-hidden-biases-in-big-data
Paul,Richard and Elder, Linda. Distinguishing between inferences and assumptions. [Webpage] http://www.criticalthinking.org/pages/distinguishing-between-inferences-and-assumptions/484
Vanscha, Karina. (2015, November 20) Qualitative Research is Always Biased [Blog post]. Retrieved from https://medium.com/@kvanscha/qualitative-research-is-always-biased-c77fe349b39b