Used Only as Intended: How Researchers and Designers Can Improve the Social Impact of Emerging Technology

Hi everybody. Welcome. Uh if you've been here before, you kind of know the drill. Um this is the April 29th Accelerant Research Virtual Insights Conference. Um we do have a few ground rules. Um and if you've attended one of these before, I'm sure you know this. Everyone knows the first one. Uh we are at the mercy of of technology. Um after a year and however long um having to zoom with everybody, I'm sure we have all been at that mercy um probably multiple times. Um, also, um, just I mean this is good general life advice. Uh, always be respectful, um, both of our presenter Mandy, um, and with each other as you're able to chat and ask questions and things like that. And also, please use this opportunity to network, engage, interact with each other, um, and, uh, ask questions. It' be great. Vandy, um, is talking today. Um she is stop being wonky computer. She is the UX research lead at CreditWise um at Capital One. Um and she's going to pre be presenting today on used only as intended. How researchers and designers can help improve the social impact emerging designed technology. Mandy, I am going to stop sharing so you can take it away. Thank you so much. I am gonna go ahead and share my screen. Um, if you can't see my screen, please give me a shout. Um, but you should be able to um to see my screen at this point. So, my name is Mandy Drew and I'm the UX research manager working on equitable design at Capital One. And I'm really excited to have this opportunity to explore with you the unexpected impacts new designs and technology make on society. And then also to kind of think of some ways that researchers and designers might be able to help. When I first started thinking about this, I had just watched The Social Dilemma on Netflix. And while it has its issues, I was really intrigued by some of the questions that it raised. As we'll discuss, a lot has happened since that movie came out. So, I think it's even more important now that we incorporate some new ideas into our design and our research practices. So, for a little overview of our time together today, first I'm going to share with you my own personal story and then next we'll talk about why some new technologies were first developed and then some of the unintended impacts they've had on society. After that, we'll explore ways researchers and designers can help predict how our products might impact people before they're released, how we might observe their effects once they're out into the ether, and then how and what we might be able to do to mitigate any negative consequences that we discover. Then finally, we'll consider how all this might alter the way we think about the UX research practice as a whole, also about the design practice as a whole. So, one of my favorite things about being a researcher is hearing the personal stories that people share. So, that's where I'd like to begin today. I lost my family in 2020, not to the Corona virus thankfully, but to conspiracy theories and false claims promoted on social media. People who once shaped my own world view, who used to look out for me, and who would have gushed with pride to hear that I'm presenting at this conference to all of you fine people today. They're gone now. In their place are strangers who didn't call me during the holidays or on my birthday or even when our nation's capital, which is about five blocks away from my home, was breached. And while many purveyors of the conspiracy theories they believe have been shunned from Facebook, YouTube, and Twitter, those lies still persist now fragmented and splintered on lesser apps and fringe sites like Gab and Telegram. But I know there are a lot of people out there who can identify with my story because I've heard from them. And as we'll discuss, the people that we've lost are not gullible or willfully ignorant at all. They believe a lot of things that aren't true. Partly because the manipulation of emerging design patterns and new technology has made misinformation easier to find, consume, and believe than the truth. If you haven't watched The Social Dilemma, it's worth a day in court because the actual people who developed these technologies explain what they were trying to do in the first place. And it also shows how the misuse of those features have damaged our relationships, our society, and our democracy. For instance, it's easy to understand why algorithms and machine learning were originally developed and why the tech industry has worked so hard to harness their power. At their core, these technologies help us narrow down choices, make decisions, and find patterns without being encumbered by human reasoning, which is much slower, more emotional, and highly subjective. If like me, you're a researcher who's ever tried to analyze a mountain of qualitative data manually, then you know how helpful using a natural language processor can be. And that's an example of machine learning. These technologies are incredibly useful at helping us find what we want quickly without having to do much work. Can you even count how many times Google has answered a question for you before you could even ask it? And these technologies also enrich our lives by expanding our world, showing us new places to live, people to meet, and things to do that we'll probably like but never would have found on our own. But when we trust decisions like criminal sentencing and medical care up to predictive models, the absence of human reasoning can raise unsettling issues about fairness and bias. In her book, Hello World, being human in the age of algorithms, mathematician Dr. Hannah Fry describes how one court's reliance on a recidivism algorithm sentenced a 19-year-old to a lengthy sentence for engaging in a consensual physical relationship with another teen when he only would have gotten a slap on the wrist if he had been 36 years old. While most humans would understandably punish an older adult more harshly, the algorithm predicted the teen would be more likely to reaffend given he has more time left in his life to commit further crimes. And a 2019 study published in Science magazine found significant racial bias in an algorithm used in hospitals all across the United States. Researchers discovered the problem while examining the impact of programs that provide after care for people with chronic health issues. While running statistics they received from hospitals, they were surprised to find that people who self-identified as black were generally assigned lower risk scores than equally sick white people. The researchers found that instead of looking at biological characteristics to identify patients who would need follow-up care, the algorithm relied only on the cost of the medical care people had already received. The algorithm incorrectly predicted that patients who spent more on medical care were probably sicker than those who spent less and failed to take into account about 48,000 chronic conditions that impact black patients causing many of them to miss out on critical afterare. In my case, algorithms algorithms on YouTube correctly concluded that my family would find stories shared on alternative media interesting, but my family didn't know that those news sources aren't held to the same standards as traditional journalism outlets. On the sites where my family gets their information, charismatic personalities preach completely fabricated stories similar to those binge watched by a na man named Edgar Madison Welch in 2017. On a December night, Welch became more and more agitated by the steady stream of content that YouTube's algorithm automatically played based on his viewing history. In the morning, he drove over five hours to Comet Ping Pong, a pizzeria in Washington DC, to rescue children. The video said Hillary Clinton and her campaign manager were trafficking in hidden tunnels underneath. Welch stormed the restaurant with an assault style rifle, terrorizing patrons and employees, but was shocked to find no tunnels, no abused children, and no evidence that the Pizzagate conspiracy was real. for his crimes. The former firefighters served three years in federal prison and lost custody of his own children. The incident garnered nationwide attention and dozens of news outlets discredited the unfounded con uh pizzagate conspiracy. But as a 2018 study by Sloan and the MIT Media Lab concluded, false information spreads significantly farther, faster, deeper, and more broadly than the truth, especially when it's political and often by an order of magnitude. That study, which tracked about 126,000 cascades of news stories spreading on Twitter, found that false news is 70% more likely to be retweeted than information that faithfully reports actual events. I experienced this phenomenon firsthand when my family came to visit me in DC two years ago for what ended up being the final time. My family were horrified when one evening I innocently suggested we grab dinner at my favorite pizzeria, Comet Pingpong. No amount of evidence I could site could ease their fears. And that was when I realized that my families were victims of the same misinformation campaign that claimed Welch's freedom. isolated from information sources that value journalistic integrity. They are effectively cut off from reality. Along with algorithms and machine learning, data mining and ad retargeting can be really beneficial for us, too. Let's say you're in the market for a new mattress from a specific brand. You see it on a website for $1,000. You put it in your cart, but you decide to think about it before pulling the trigger. Later, you see an advertisement for that very same mattress in your Facebook feed for 10% less, plus free shipping. The information you entered in the initial website was stored as a browser cookie that followed you as you continued to surf the web. Data mining and ad retargeting just scored you over $100 off the initial price, but these technologies have their downsides. Social media profiles gather an enormous amount of information about us. Some things we knowingly share, but also things that we tend to think of as more private, like the pages and profiles that we visit, what posts we interact with, and even what we type. The most infamous example of a company improperly using that type of data was six years ago when Cambridge Analytica, a political analytics firm in the UK, harvested locations and page likes from over 87 million Facebook users, which they used to build individual psychological profiles they exploited to manipulate voters through targeted ads linked to the 2016 US presidential race and the EU Brexit referendum. Different ads displayed to different groups of people based on their scores on the ocean personality model which rated their levels of openness, conscientiousness, extraversion agreeableness and neuroticism. It's impossible to know for sure whether Cambridge Analytica's efforts impacted either election. And that isn't really my point here. The tactic of using personal data to pit psychology against private individuals foreshadowed the political polarization and the social turmoil that consumed much of 2020 and was a key factor in perpetuating many of the reasons my family no longer speaks to me. The final bit of technology I'd like to discuss is ubiquitous in all of our lives, but we barely even notice it. Infinite scroll is a seemingly innocuous design pattern that was developed to create a seamless experience so we can move up and down pages continuously without breaking our flow. But its elegant simplicity masks the powerful underlying behavioral and psychological constructs that make it so compelling. Infinite scroll is perhaps one of the most effective design applications of the hook model. a framework created by behavioral economist Mirror Isle to help companies create habitual users of their products. The idea is that some type of cue, be it an internal feeling like boredom or loneliness or an external reminder like a notification that someone commented on our Instagram post, triggers us to open the app and scroll down the page until we see something interesting which our brains view as an instant reward. rewards flood our brain with the pleasure chemical dopamine. That pleasurable feeling compels us to continue to invest more and more of our time scrolling so we can feel that dopamine hit again and again. And this can be a really great framework to follow if we want to help people develop healthy habits like building an exercise routine, sticking to a diet, and incorporating a meditation practice into their lives. This app called Streaks is a great example of a product that uses the Hook model to help people develop good habits and break bad ones. The app sends an external trigger, a notification reminding the person to complete an action they want to make a daily habit. If they do it, they're rewarded with praise and stats on how well they're doing, which encourages them to continue to invest time back into the tasks and the app to keep their streaks alive. But the problem with the hook model is that it precisely mirrors the cycle of addiction. People who are addicted to drugs respond to powerful internal and external cues like boredom and loneliness or a memory of a pastime they use the drug which triggers them to seek out and use the drug which instantly floods their brain with dopamine making them feel better causing them to use more and more and more. In fact, Asa Rascin, who invented Infinite Scroll, says he regrets developing the feature because it's as if they're taking behavioral cocaine and just sprinkling it all over your inner face. I believe all this evidence suggests that instigators are able to manipulate the combination of algorithms, data mining, and infinite scroll to spread a lot of the false narratives. Many people including my family believe today. For instance, when corona virus lockdowns were announced last year, many of us were kept isolated indoors on edge and buried in our devices, hungry for any information we could get to help us make sense of what was happening. False information spread faster than the virus itself. Doctors, scientists, health officials, and journalists worked tirelessly to debunk myths about the virus, its severity, and potential treatments. But conspiracy theories continued to prevail. Avas, a nonprofit advocacy group that tracks false information, published a report in August which identified the top websites that spread the most health misinformation and the top Facebook pages that drove the most views of the false claims those websites shared. Using that data, AAZ estimated that Facebook's algorithm drove 3.8 8 billion views of those websites health myths peaking at an estimated 460 million views on Facebook during last April. The same month the first wave of the virus peaked in the United States. While correlation doesn't equal causation, studies on the Ebola virus in West Africa and the measles outbreak at Disneyland have linked false beliefs with infection spread. And a working paper by the Becker Freriedman Institute for Economics at the University of Chicago indicates that areas that areas in the United States with greater exposure to television media that downplays the threat of COVID 19 experience a greater number of cases and deaths. And health myths aren't the only dangerous false narratives proliferated during 2020. Last summer, calls for racial justice intensified after several unarmed black people were killed while in police custody, and the nation erupted in protests after the brutal murder of George Floyd. While executives at Facebook, Twitter, and YouTube declared their solidarity with the Black Lives Matter movement, donating millions of dollars to racial justice organizations. Their platforms became meeting and breeding grounds for white supremacists, racists, and conspiracy theorists, spreading more of the deceptions my family now believes. Partisan provocators accused the media of liberal bias, which they said led to rioting and lawlessness in several cities. But an analysis of millions of social media posts by Politico and the Institute for Strategic Dialogue, a nonpartisan think tank in London that tracks extremism online, showed alternative media content was shared more than 10 times as often as content from more reputable sources. The report also showed that the most shared Black Lives Matter keywords on all three platforms returned results that falsely associated the movement with violence. During that time, I attended the March on Washington to show my support for the movement. When my family found out, they accused me of being a member of Antifa. The same report on Black Lives Matter also asserts that false rumors spread through social media erroneously cast doubt on the legitimacy of the 2020 US presidential election well before any votes had ever even been cast. People seeking to understand the risk of voter fraud in the months prior were exposed to fraudulent claims of widespread election corruption, which planted the seeds from which the Stop the Steel rally grew, which ended with the deadly insurrection of our nation's capital. This image is really emotional for me, as I imagine it is for many people. I'm proud to call DC my home. And I certainly never thought the tech industry, of which I'm a part, could help spread lies that would bring this city and our democracy democracy to its knees. If we could have predicted it, I'm sure we would have tried to prevent it. So that's why I'm here speaking with all of you. I believe that researchers and designers have an enormous responsibility not only to find out if the designs we help create are useful, usable, and enjoyable before launching them out into the world, but to help predict their possible impacts beforehand and observe how they're used out in the wild. Let's explore some ways that we might be able to make that happen. First, during every stage of the product life cycle, let's strive to recruit a diverse, representative sample of research participants by thoughtfully crafting inclusion criteria that matches the characteristics of everyone who will encounter our products in real life. For some of us, this might mean shifting from guerilla recruitment tactics like testing designs on internal company associates who likely have insider knowledge of how our products are supposed to work and who are invested in their success. We also may need to refrain from studying our friends, family, and social media contexts who might reflect our own worldviews and who could be more inclined to spare our feelings by telling us what they think we want to hear. And perhaps we should rethink over reliance on in-person research in general because it limits us to observing only people who live near our labs. I can tell you from experience that a graduate student in remote Northern Michigan University has very different needs, goals, and behavior than those who go to Columbia University in New York. Instead, let's embrace remote research tools. They can help us test pre-flight designs and products we've already launched, and we can build and we can pull from their panels to study people from a variety of different backgrounds, locations ethnicities educations incomes, abilities, etc. This is a great skill to develop during the pandemic because it's not really safe to bring subjects into the lab right now anyway. Another reason remote research is a good choice is because we can use it to observe people's organic experiences with our products. The Hawthorne effect indicates research participants sometimes alter their behavior when they're being watched. And in my experience, people do tend to act inauthentically when they're in a research lab using unfamiliar devices and fake test account data. Remote unmodderated research can mitigate the Hawthorne effect because even though participants know they're being recorded, they they're still by themselves in their own homes using devices they're familiar with. To help predict how people might use our designs and mitigate any negative impacts before they launched, we can craft tasks that are goaloriented to communicate the frame of mind a real person might have when they actually use the product. For instance, let's say we're researching how a new voice assistant feature could help people plan events on social media. We could write this very literal task to see if people can figure out what we want them to do with the product. However, we might not get a lot of good information because participants are likely to respond by saying something like, "Schedule a Zoom meeting for Tuesday and invite everyone on my Facebook friends list." From that interaction, we can tell people know how to use a voice assistant they're already familiar with to set up a meeting. But we don't know if people might have have unanticipated needs or could behave in unexpected ways. If we craft a task that better matches this real world scenario while giving them freedom from current technical restraints, our test subjects might say something like, "Help me schedule a lunch meeting on Hangouts for Tuesday with my old colleagues from the AI team and send balloons to Jamaica so they arrive during the meeting. Send breakfast to the people on the east coast and lunch to the people out on the west coast." we get much more information from that interaction. We might deduce that when planning events, people ask for multiple things at once. So, it's really important to include list taking and memory skills into our voice assistant. We can also tell that people want their voice assistant to understand complex tasks. So, we should start prioritizing work on our na natural language processor. Once our products have been launched, we can use similar techniques to observe how they're actually used in the wild. We could accomplish this by conducting continual remote unmodderated video diary studies, asking people to record their voices and screen actions every time they use our products. That way, we can see any problems that they encounter and we can understand how they're using it. But to take this idea one step further, we might get greater insights if we ask people to record themselves every time they have a need our product could solve. That way we can understand how people are currently solving that need, what other tools they use to do it, what that experience is like for them, and how we might improve that experience to make people's lives easier and better. And in order to gain a holistic understanding of the impact our products might have on everyone who uses them. We might sorry about that. We can triangulate our research by using qualitative and quantitative research methods. Quantitative data like AB test results, usability, performance metrics, and site analytics can help us predict and observe the scope and scale of our designs impacts like how many people use it, where they are, and what they're doing. This can be extremely compelling to our our to help our partners prioritize what we should work on next. And the qualitative data that we've gathered like videos and sound bites from interviews, usability tests and diary studies can help us understand why people are using our products, how they comprehend them, and the impact those experiences make on their lives. This provides rich contextual human context necessary to elicit empathy and compassion from our partners. And once it's time to share the findings of our research, we can include responsible, ethical recommendations that positively contribute to people's health and well-being. Three researchers at a university in Switzerland did a great job of this in an article they presented at KAI 2020 last April. In designing for digital detox, making social media less addictive with digital nudges, the researchers recommended and tested specific design patterns they called nud nudges meant to counteract Facebook's addictive nature by helping users become more mindful when interacting with the platform. The nudges they proposed accomplished this by hiding Facebook notifications, requiring people to choose to display video content instead of playing it automatically, asking people to pause after scrolling for a while and decreasing people's investment by making it easy to turn addictive elements off. The study found those nudges led most participants to reduce their use of Facebook and over half said they made the platform more enjoyable. The final way I think researchers and designers can help mitigate the impacts of unexpected consequences from our products is by encouraging our business partners to consider new and different ways to measure success that prioritize people's health, safety, and happiness. Many of the most common engagement metrics, page views, time on page, session duration, pages per session, and page scroll depth. They equate success with getting people sucked into our devices and addicted to our products, which really causes isolation. Digital ethicist and co-founder of the Center for Humane Technology, Tristan Harris, argues that optimizing products to improve engagement metrics has led to the downgrading of humanity by shortening our attention spans, and promoting angry discourse, narcissism, political polarization, and an over reliance on our smartphones. His time well spent movement called for tech companies to build tools to help people become aware of the outsized role digital media plays in their lives. As a result, Apple, Google, and Facebook all added features to help people measure and manage their time more mindfully. Harris's movement has since evolved and he now calls upon the tech community to develop features to help people find commonality with one another, promote healthy childhood development, and strengthen our democracy. In addition to engagement metrics, the success of many of our designs are judged by their net promoter score. And the idea behind NPS is that company growth hinges on the degree to which the experiences we design encourage or discourage people to recommend Capital One or or whatever company to their family and friends. Business partners like NPS for its perceived correlation to profits and because it helps quantify what appears to be customer loyalty and lifetime value. But there are many problems with using NPS to evaluate the efficacy of designs. Because even though someone might dislike an experience they've had with Capital One so much they wouldn't recommend it to others, they might still continue to use Capital One because switching banks and credit cards is a giant pain and closing card accounts and opening new ones could hurt their credit scores. problems with NPS um as an evaluator for designs become more obvious when we try to use it to determine the success of individual experiences. For one, usability can never be fully captured by subjective scores. Also, by sorting responses into bins, detractors, passives, and promoters, we lose the strength of respondents convictions, and we can't tell why people feel the way they do or how we might improve that experience moving forward. Researchers and designers can play an important role in helping enrich people's lives while improving our business metrics by encouraging partners to rethink what success actually means to our customers when they're using our designs. Instead of orienting business goals around the length of time people spend interacting with our products or whether or not customers would recommend Capital One or other companies to others, we should encourage our partners to consider ways to measure the quality of those interactions. For instance, former Google UX researcher research director George Zong told me that while he was at Uber, he tested net promoter scores against inapp true intent surveys, which pop up after a view a visitor has just finished using a particular design or feature. Zan told me that true intent surveys are far better at predicting driver engagement, lifetime value, and likelihood to churn, had a better response rate, and were better at predicting which design in an AB test would generate more long-term revenue than NPS. We could implement the same idea by incorporating interceptor surveys um immediately after customers complete high value actions. One final way researchers and designers could help improve the impact our designs have on society is to make it a priority to bring more diverse voices and perspectives into our research and design practices. Some little known facts about me are that I don't have a PhD or even a formal scientific education. I'm a proud member of the LGBTQ community and I'm a lifelong survivor of chronic PTSD. Life experiences such as these bring unique perspective, inherent human understanding, and boundless empathy into our work. As we build teams, let's consider searching for people who don't look like us, don't have the same education or abilities, don't think or act like us, and don't live the way we do. And let's share our work and our findings with those individuals so we can check our own internal biases before they make their way into our products. Today, we've learned how technology and design advancements can have unintended and in some cases devastating consequences. My experience losing my family has definitely reshaped how I approach my research practice. I am not at all suggesting that we stop using new technology or developing new advancements. On the contrary, let's use algorithms to subdue lives. Let's promote sources with scientific integrity and be inclusive of everyone who might encounter them. Let's be clear upfront how we plan to use information people trust with us and allow them easy ways to opt out if they choose. Let's take another look at the hook model and come up with ways we can use it to reward healthy behaviors and to bring people together instead of isolating them from from inside their devices. And let's try to understand the immense responsibility that comes with unleashing new technology out into the world for millions of people to use. And finally, this is not an exhaustive list. I would love to hear your ideas. How researchers and designers can supercharge our superpowers to create new design patterns and tech technological advances that enrich people's lives and make our world a better place. Thank you so much for listening. I'd love to take your questions, comments, and suggestions. Thanks, Mandy. If anybody has any questions, go ahead and just type them in the chat or in the Q&A. Um, we have had a couple come in. Mandy, um, early in the presentation, Alex asked how the issues that you mentioned can be applied to research, but I think the remainder of your presentation got right to that. And then in Q&A, Rebecca asks, "What do you think is the future of the NPS score? It can be so problematic. I mean, one, how likely are you likely to recommend anything to anyone, especially if you don't ask if people don't ask you for a recommendation?" I think we've all encountered that. Two, how do you recommend your their unique situations? Works for me, but I wouldn't recommend it for you because you're not in the same cohort, whatever that may be. I think we've probably all run into that, Mandy. Absolutely. Yeah. So, I I think of, you know, NPS is one tool in our toolbox and it's really important when we are, you know, considering different methodologies to really go back to what is the goal, what is the purpose of our study, what is the question that we're trying to answer, and then let's figure out what the best tool is to answer that question. Um so there may be you know situations where NPS is the perfect tool to use like let's say we want to know how many people would you know recommend um us to somebody else NPS is a great way to get at that information um but when we're really thinking about you know um evaluating designs and experiences particularly um they're super problematic for a lot of the reasons why I mentioned um and that's really kind of where I started Ed, you know, kind of discussing this with other researchers. Um, I connected with um the researcher from from Uber that I was speaking about, George Zong, and um and really started realizing that a lot of researchers are kind of doing their own um you know, sort of private, you know, um experimental research to determine, you know, what really, you know, is the NPS like what's the construct validity there? What is it actually measuring? Um, and is it is it the best way is it the best method to evaluate designs and experiences and you know at least according to George and and this definitely reflects my own anecdotal um evidence um you know he was seeing really not a lot of value in NPS as opposed to using these true intent surveys. So I kind of see the future um at least in a design like in terms of evaluating designs and experiences moving towards that in context um true intent um sort of popup survey um and uh and and hopefully you know I think that um that is kind of where you know Capital One might go um in terms of design evaluation. Excellent. I definitely agree. I know we here at Accelerant have gone through some of that personally in my career. I know I've done a lot of that where you have to just change it up. So, I don't see any other questions, but we can give it just like another minute and see if anyone pops up with anything. Sure. Rebecca does say thank you. You're welcome, Rebecca. I don't see anything else. So, Mandy, unless you have any closing thoughts, uh we'll wrap this session up and uh we'll take a little break, go to the restroom, get yourself some water, whatever you may need, and we will be back at 3 o'clock. Thanks again, Andy. Thank you so much. Oh, thank you. I loved it. Very good. Thank you so much. I really appreciate it. I can definitely commiserate. Yeah. All right. Well, I'll see you later. Thanks again, Andy. Thank you. Bye. Bye.