May 2021 ARVIC

The May 2021 ARVIC virtual conference brought together practitioners to address two persistent pressure points in applied UX research: how to run interviews more efficiently when tools and team support are limited, and whether preference questions belong in usability studies at all. Across both sessions, the emphasis was on closing the gap between what researchers collect and what actually drives reliable insight.

Bringing Rigor and Efficiency to User Interviews

One talk focused on the practical mechanics of conducting user interviews under the constraints many researchers faced during the COVID-19 period: reduced staffing, limited tool access, and restricted participant pools. The methods presented prioritize speed in analysis without sacrificing consistency across participants.

• Replace verbatim note-taking with a time-stamped indexing system tied to your question outline. Navigate recordings by topic rather than listening end to end.
• Use a coded shorthand during live sessions to flag insights and quotable moments in real time, cutting post-session analysis time significantly.
• Write questions in conversational language and use a physical checklist to confirm topic coverage, keeping sessions organic while maintaining cross-participant consistency.
• After each session, print the indexed script and use it as a quick-count sheet to tally how many participants raised a given issue. This is faster than mining software logs.
• Go beyond the standard remote research four (analytics, usability testing, surveys, interviews) with hybrid methods such as self-interview videos, the make-up/break-up letter exercise, cognitive walkthroughs, comparative analysis tasks, and before/during/after session structures.
• Context matters for method selection: the make-up/break-up exercise works well with younger demographics in North American and European markets but may not translate in contexts where personal relationships are more private.

The Problem with Preference Questions

A second talk made a direct case against relying on stated preference as a proxy for usability. Drawing on psychology research and internal case studies, the presenter showed that what participants say they prefer and what actually performs better on usability measures are routinely misaligned.

• In both case studies presented, participants preferred a design option that scored lower on independent measures of ease of use, understanding, and satisfaction.
• Five cognitive biases actively distort design preferences: the framing effect, choice blindness, the substitution effect, the aesthetic usability effect, and the decoy effect. These are difficult to detect or control for in a standard study setup.
• The Pepsi Challenge serves as a real-world cautionary tale: sip-preference results did not hold up to actual consumption behavior, illustrating that preference data can mislead even outside digital UX.
• If preference questions cannot be removed, always collect independent usability measures (time on task, task success rate, SUS, CSAT) after each design option individually and let those measures outweigh comparative preference results.
• Ask participants to explain their preference choices to help separate aesthetic or irrational drivers from usability-relevant reasoning, while recognizing that participants often cannot fully articulate their own decisions.

Key Takeaway for Insights and Product Teams

Both sessions pointed toward the same underlying principle: common research shortcuts, whether informal note-taking or preference questions, introduce noise that compounds over time. The conference made a practical case for tighter protocols, not more elaborate tools, as the path to more defensible findings.