April 2021 ARVIC

Three talks at the April 2021 ARVIC covered ground that rarely sits together in one program: the long-term social consequences of UX decisions, the psychological experience of conducting research under pressure, and the mechanics of survey data quality. Taken together, they make a consistent argument that researcher rigor extends well beyond methodology into ethics, self-awareness, and the conditions under which data can be trusted at all.

Researcher Responsibility and the Social Impact of Design

The conference opened with a direct challenge to the idea that researchers and designers are neutral parties. Drawing on how features like algorithmic recommendation and infinite scroll were built for legitimate purposes but have been exploited to spread misinformation and drive addictive behavior, the argument was clear: downstream harm is partly a design problem, and researchers are positioned to catch it before products ship.

• Diverse, representative recruitment is a prerequisite for predicting real-world impact. Over-reliance on internal employees or in-person lab participants creates blind spots that remote, unmoderated methods can reduce.
• Goal-oriented, scenario-based tasks surface unexpected behaviors that literal task prompts miss. Diary studies extend that observation to real-world use.
• Common engagement metrics reward addiction over well-being. Researchers can push partners to reframe success around quality of interaction and measurable user benefit rather than time on page or scroll depth.
• NPS is a weak tool for evaluating individual design experiences. In-context true intent surveys have shown stronger predictive power for engagement, lifetime value, and churn.
• Sharing research findings with participants from genuinely diverse communities is a practical pressure-test for internal bias before it reaches shipped products.

Conducting Research Under Pressure: Emotional Honesty as a Method

A session on the COVID-19 vaccine rollout at Walgreens reframed researcher anxiety not as a liability but as a diagnostic signal. When a structured four-phase plan became unworkable as the rollout accelerated and stakeholder complexity exploded, the researcher's response was to confront what the anxiety revealed about unmet values rather than suppress it and push through.

• When the environment shifts faster than a fixed plan can accommodate, restructure the cadence. Spreading interviews across shorter, more frequent waves with a weekly alignment meeting kept findings timely enough to influence decisions.
• Embed in meetings where stakeholders already are, rather than convening new research touchpoints. It improves participation and keeps the researcher current with shifting priorities.
• Distinguish empathy from compassion. Taking on participants' emotions wholesale is unsustainable in high-volume, high-stakes research. Compassion without full emotional simulation protects researcher well-being without sacrificing care.
• In fast-moving research, two or three participants independently encountering the same problem is sufficient to flag a scalable issue for stakeholders.
• Vulnerability builds trust faster than professionalism masks it. Openly acknowledging uncertainty created the psychological safety needed to propose a major plan change four days before fieldwork was scheduled to begin.

Survey Data Quality: What Controls Matter More Than Formatting

A research-on-research experiment tested whether scale direction and orientation affect survey ratings. The headline finding is practical and actionable: among attentive, engaged respondents, neither scale direction nor orientation significantly affects results. The problem is inattentive respondents, who do show systematic inflation when scales are presented in descending order. Data quality controls eliminate the bias entirely.

• Always include attention checks. Respondents who fail them show systematically inflated ratings on descending scales regardless of orientation.
• Screen for straight-line responses. Zero-variance respondents show the same directional bias even if they technically passed an attention check.
• Removing too many respondents reduces statistical power. If more than 10 to 15 percent of a sample fails quality checks, raise the issue with the panel provider.
• Commitment statements at the start of a survey can improve data quality by prompting respondents to engage thoughtfully before they begin answering.