When user research leads design astray
or, proceedings from the HFES 64th International Annual Meeting
In October 2020, I participated in a panel discussion at the HFES 64th International Annual Meeting. Invited by Keith Karn, who led with this provocative title, I was more than happy to represent design in a group of human factors and research professionals from both consulting and corporate industry. Though my contributions focused primarily on design, I also spoke from my experiences at Nemera/ Insight where I have worked across our dedicated design research, human factors, and industrial design teams, in addition to interacting with client teams.
Now that the conference proceedings have been published, you can read the full paper here. An introduction and my perspective are also written below:
Introduction
As product development typically requires significant investment of resources, it is important to get it right. User research can play a vital role in ensuring a product’s success, and we are fortunate to have a wealth of user research methods available as well as many user researchers eager to contribute to the overall product design process.
However, the quality of the data from user research can have a significant impact on the design and development activities that follow, and can mean the difference between a great user experience and a failed product. When the process goes well, benefits can include innovative product concepts, optimal selection between design alternatives, and refinement of significant design details. However, things do not always go so well. Well-meaning researchers may choose the wrong research method for a study or implement it poorly. Commercial constraints related to budgets, schedule pressures, or regulatory issues may force academically-perfect research plans to be compromised. Regardless of the cause, the implications of poor research are manifold: bad data can lead to wasted investments, recalled products, or even harm to users.
Common pitfalls seen across industry segments include:
Ill-defined research objectives such as asking the wrong or irrelevant questions, at the expense of critical lines of inquiry.
Unrealistic expectations about what the team can learn from a research study.
Focusing on short-term issues (e.g., a small formative usability study) rather than longer term (e.g., more foundational, up-front research).
Recruiting the wrong people to serve as participants in a research study.
Asking leading questions and/ or biasing study participants (e.g., by “selling” an idea).
Expecting participants to be able to verbalize things that they cannot, putting too much weight on what they say rather than what they do.
Allowing "group think" to spoil focus groups.
Focusing on ratings over rationale for ratings.
Trying to draw quantitative conclusions from a sample size that is too small.
Letting a focus on regulatory compliance relegate user research to a "checkbox" for agency approval.
Conducting research too late in the process to impact design in meaningful ways.
Allowing a sense of obligation to parent organizations result in researchers becoming product advocates rather than user advocates.
Alisa’s Perspective
As research findings are often direct inputs into design activities in the product development process, designers are key stakeholders of user research. As such, designers have a responsibility to recognize bad research, advocate for good research, and participate in the research process wherever possible. In turn, design researchers and human factors engineers are responsible for including designers in their efforts and ultimately delivering good research. In turn,
From early ethnography that yields opportunity for true innovation to late-stage usability that validates product safety, designers should be cognizant of the fact that the timing and number of touchpoints with users can make or break a product. Too few, and valuable resources will have been invested in perfecting something that doesn’t meet a need or can’t be used, and designers are unlikely to be given a chance to course-correct. Too many, and research becomes a crutch, paralyzing the process and short-changing designers’ ability to add value. Establishing a thoughtful research strategy at the onset of a program - or even prior - is key to successful design/ research collaboration.
Of particular importance to designers are the “why’s” behind participant responses; quantitative data such as ratings require assumptions to extrapolate this type of data for continued design exploration. At the same time, researchers should be mindful of the temptation to “design by research:” trained in creative problem-solving, designers bring a unique voice (and eye) to the table. As the well-worn story goes, if Henry Ford had relied on user feedback alone he would have produced “faster horses,” not the Model T.
Given this particular relationship between design and research, the two disciplines ought to work hand in hand to deliver the best results; when this is not possible, keeping each team’s objectives top of mind should ensure success. Ultimately, we’re all on the same team.
Karn, K., Harper, C., Rantanen, A., Edwards, R., & Bartha, M. C. (2020). When User Research Leads Design Astray. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 64(1), 1018–1021. https://doi.org/10.1177/1071181320641244
Note: these views are my own and do not necessarily reflect those of Nemera or the Insight Innovation Center.