Seize the Moment: Researching Emerging Tech in the Age of AI

Uh, I'm really excited to be here today. As Bill noted, I was one of the one of I was one of the speakers in the very first ARVIC conference. So, kind of cool to come back years later. I was looking at my what my presentation was back in 2020, and it was about building rapport at a distance, and it seems to still be available. So, you can find it on my LinkedIn profile or probably on the Accelerate website. So, yeah, very cool. Very excited to be here today with my colleague Amy. So, we are uh, we've we've tweaked our title a little bit. Seize the moment, carpe diem, about research researching emerging tech in the age of AI. So, I'm Jenny. I'm a lead design researcher at Cisco, where I study cybersecurity and agentic AI. Prior to this, I worked for Ford Motor Company's autonomous vehicles division. At Ford, I researched a variety of potential uses of autonomous vehicles, including pizza delivery. That's actually going to come up later. I'll dig into that example later. Uh, I but we also researched grocery delivery and ride hail. I did most of my research in Miami, Florida, for that work in the form of contextual inquiry and observation. Here at Cisco, uh, I've been at Cisco since '21 2021. I focus on identity security. So, when you think about logging into your apps and your computer at work, you know, that's that's what I'm focused on is the security of that login experience. Uh, everything from how do chief information security officers think about access to how do IT admins monitor application security. So, this includes research into agentic AI, those mini tools that we can now use. I've seen a lot of actually agents in the chat, right? People are using note-taking agents. So, we can use these mini tools now to get specific tasks done autonomously, and I've done a little bit of research into that. I'm an anthropologist, and I got into this work after spending the first half of my career in the nonprofit sector. Amy. Thanks, Jenny. And I'm Amy. I'm a senior researcher also at Cisco, also specializing in cybersecurity and agentic AI. So, at Cisco, I really focus on zero-to-one product development and adoption efforts, specifically right now focusing on identity and access management. Currently thinking about how to help IT admins secure AI agent usage, like we just talked about, for their end users, right? Making sure that their environments are secure, but we're also enabling these new technologies. Before Cisco, I worked at IBM as a hybrid service designer and researcher, where I focused on advancing digital transformation efforts for external clients, helping them tackle large-scale technology projects like cloud migrations, data, and AI through rapid prototyping and research to understand what's the most important business challenge that we need to focus on and what challenge needs to be de-risked so that we should make an MVP based off of it. And before that, I was also at Jen also at Ford with Jenny, where I focused on design research, working on autonomous vehicles and services, the performance vehicle lineup, and luxury lines, helping think through future-proofing, expanding customer bases, and developing novel services and products. And also mobility services and programs in partnership with city government. So, a wide variety of experiences, but all kind of in the the world of emerging tech, things that don't exist yet. Um, and I do have a background also in research, but also design, specifically experience and service design. So, as we mentioned, Jenny and I have worked together on and off since 2018. We worked on pizza delivery together at Ford, with Jenny continuing to focus on AV research, as she noted. And at Cisco, we've worked on agentic AI and security, and I continue to work on more agentic uh, projects like securing agentic identity and platform agentic strategies. And so, let's kick off the talk now that we gave more of an intro of who we are and our background. So, want to talk about the elephant in the room, AI. The age of AI, as Bill mentioned, is like in all of the talks today. And in the last couple years, even months, things been changing super fast, and there's lots and lots of discourse that you probably cannot escape. AI's impacting a lot of functions. It's impacting the design and product development cycle and fundamentally transforming how we build things. From LinkedIn post to Reddit post to Medium articles, again, lots of discussion, lots of excitement around this. We're hearing also more mention of AI-moderated research. Maybe your stakeholders are asking you about synthetic users. Anthropic just published a study using its very own AI researcher. And for researchers, we want to acknowledge that this may feel like an extinction-level event or at the very minimum anxiety-inducing. Is AI the end of user research as this Medium article is titled? Short answer, our belief is no, it's not. Um, so Jenny and I feel really strongly that we as qualitative researchers in qualitative researchers in particularly particular are needed now more than ever. And our goal of our talk today is to inspire. Uh, this is what we'll cover. The value of qualitative research for work in especially in emerging technologies. Our new role as researchers. We'll talk later if it's new or not. And also some specific research techniques that we have used uh, for agentic projects and AI projects, emerging tech, and that we feel like can be used in adjacent fields as well, but particularly in this age to help seize the moment. And we're also going to save a good amount of time today for discussion. So, please gather your thoughts and questions. Feel free to add them to the chat in the meantime. And I will pass to Jenny to walk us through qualitative research and how it's always played a a pivotal role in innovation. Thanks Amy. Um, I just want to note, I am keeping an eye on the chat. So, if you have a really timely question, go ahead and I'm I'll, you know, we we can address it like as we're going through our flow, but probably we'll save most of the questions for the end. So, do drop them in the chat and we'll consider. So, yeah. So, again, thanks, Amy. Let's consider what is the value of qualitative research for emerging technologies. So, AI tools allow design and engineering to build incredibly fast and also much more cheaply. Code and user interfaces that used to take weeks might now take hours. But we can't just put everything that we're producing in front of our users. There are limits to what we should ask of the people giving us feedback. There's throwing spaghetti at the wall or in the case of our users, forcing them to like eat all that spaghetti and tell us how it tastes. All the concepts and ideas that we want to explore, it's it's it's a lot, right? So, users have a limit for how many decisions, variables, and versions of something they can react to. There are practical concerns when you're testing something that doesn't exist yet. And we can get stuck on particular ideas. As researchers, as design teams, as product teams, as, you know, three-legged stools, we can get really stuck. But we still need to know what to fully invest in, what to commit to. The project managers who have to actually make those high-stakes decisions about what to launch, they still have to sort through all of the opportunities and inputs. They have to figure out what makes sense. So, research in the in these moments does not simply de-risk product decisions. It gives directionality and it gives focus. I imagine that Sam Ladner is well known to just about everyone here. She has said, and this is a quote, "Qualitative research, therefore, is a cheap way to guide the company toward different product offerings and ultimately toward innovation." So, Amy and I agree. Qualitative research delivers a lot of bang for the buck. A well-planned week of contextual inquiry or in-depth interviews can yield material for multiple product cycles, even in this rapidly evolving age of product development. The AI that we're usually talking about when we talk about this new rapidity is large language models or LLMs, and LLMs are about prediction. Based on the data on which the LLM has been trained, what should come next? This is very important for getting through lots of data, data at scale. And it's important and helpful when we already have a deep understanding of the topic, but it's less useful in other cases and contexts. It's important to recognize that, you know, we humans are famously unpredictable. So, that's where abductive reasoning comes in, and this is something that, you know, AI is not great at, right? This is a term coined by Charles Sanders Peirce, and it's something that which we human researchers really excel. We can take a set of observations and infer a reasonable explanation, but we can also infer multiple reasonable explanations. We can identify patterns, yes, but we can also identify what is novel, making some sense out of human unpredictability. So, qualitative research is actually exactly what is needed to develop the insights that can inform decision-making, especially for emerging technologies. The insights that come from qualitative research inform strategy. They help us choose where to focus, and they enable us to figure out how to differentiate. Think of questions such as, what's our vision? Or Roger Martin's classic, where should we play and why? And also, what expansion opportunities are we failing to consider? These are very human questions to think about. So, Amy and I have developed have three examples of how we've done qualitative research to develop insights that support emerging technology specifically. Um, these are these are techniques you can use in all sorts of settings, but we found them particularly useful for emerging technology. They are contextual observation, digging into mental models, and then deepening our understanding of users' experiences via in-depth interviews. We're going to go through each one in turn. Okay, so Amy and I have already mentioned pizza delivery with Ford autonomous vehicles. By the way, this is not a thing. It's no longer a thing. You cannot go to Miami and get a pizza delivered by a Ford AV. So, um but there was a moment when you could. Uh back in 2018-2019, you could do that. So, um interestingly, uh there was a shortage of pizza delivery drivers. There might still be a shortage of pizza delivery drivers. I kind of doubt it in our current age of delivery apps, but there was a shortage of pizza delivery drivers. And so, in 2018, we were investigating the possibility of using autonomous vehicles to deliver pizzas specifically in Miami. Uh using simulated AVs. So, these were not actually real autonomous vehicles. It just was wrapped in, you know, like windows were covered up and stuff like that to obscure the driver and obscure the equipment in the vehicle. Um so, I was we were in simulated AVs. I would hide in the rear seat and the vehicle would start at the pizza place and drive to the delivery destination. And my job was to just watch the reaction of the person coming to get their pizza. But what would happen is most of the time the person would not come to get their pizza. Um and we'd sit there waiting in the AV at the curb for the person to come and get their pizza and no one would come. Maybe they were in their pajamas. Maybe they lived in a high-rise and they were not going to go downstairs. Maybe it was raining. I mean, it was Miami. Maybe they ordered delivery pizza exactly because they did not want to step outside their house, right? So, okay, I was there. I would grab the pizza. I would take it to the person. But this was the kind of thing that like, you know, in retrospect, we're like, okay, yeah, but we needed to do research to find out that people would not actually come out to the vehicle to get their pizza. Even when they had opted in to delivery by an AV. So, they in theory knew what was going to happen. They wouldn't actually come and get their pizza, right? So, um that was a lot of fun that research to do. But what we ultimately learned is well, AVs won't actually work for pizza delivery because no one's going to come out to get their pizza. Okay. >> [snorts] >> Another example. This is from a colleague of ours. And this is about eye care. A colleague shared with us some consumer goods research that they were involved in. Started from a list of pain points around the product as it related to eye care. The company and the prescribers for this product positioned the product as a medical product. But in-home contextual inquiry, like they literally went to people's homes and looked at how they were storing this product and they were appalled by what they saw. This product was not treated medically. It was not stored, say, in an in a, you know, temperature-controlled environment or away from liquids or anything like that, right? What they found was the product was treated more in a way that matched personal care products. So, the packaging, the presentation, even the branding of this product was medicalized. But by going into homes, they found out that the way consumers would were more likely to consume the product appropriately was if it were positioned as self-care. So, in order to really get consumers to invest their time and attention to proper storage and use and all that of the product, they had to abandon kind of this medical perspective and think about self-care and how do we position this as this It is actually a medical product, but we need to position it as self-care for users to treat it appropriately. Never would have found that out without in-home research. People would have answered surveys. They would have maybe even done diary studies, but without actually going to people's homes and seeing all these contexts, they never would have realized that. Uh third example though is from Amy. Amy? Thank you, Jenny. So, I'm going to go into an example from Agentech AI, uh going into this emerging tech again. So, during some recent interviews about Agentech AI, we mentioned that we did some foundational research in the space for cybersecurity professionals. So, we were able We noticed actually a huge gap in understanding among these security experts who actually had passed our screener, who were also in terms of what segmentation purposes, right, the perfect customers, like the perfect target, um and were already met were already technically um as labeled actual users of agents, but whose understanding of agents and actual usage of agents was very, very off. So, their words may have been the same, but their meanings and nuances were very different and their behaviors were also really different. To the point that there was an actual disconnect between our ideas of agents and our interviewers I interviewees ideas of agents. Something that maybe would have been dismissed and maybe we wouldn't have talked to those participants even um without qualitative researchers kind of being part of the fold. But because there was also qualitative interviews, we're able to quickly identify the cognitive gaps, pivot as necessarily pivot and necessarily, and also immediately probe in different ways. And it allowed us to gather really meaningful and valuable insights on what mental models are actually exist, especially with folks we think are the target users, and how we were going to have to accommodate this in future research, but also in future project products, right? So, this actually transformed our definition of what should be counted as what a user was and helped in terms of our future future strategy and future work. So, now that we've talked about those three examples, let's go into the what is our role now as qualitative researchers in this new age of AI. So, we've established that we know that AI and product development will need help with direction decision-making, especially because we're able to build incredibly fast and that um product development cycle has rapidly transformed. And this is where we this is where we feel like qual research is going to bring the most value, right? For direction setting and for focus and really driving that innovation, driving ways to enable differentiation. Back to the eye care example, this also allowed them to expand, right? Being able to reposition as personal care also meant that they could now expand into new and more products and expand what the customers were actually buying from them. So, AI doesn't get rid of our role. It just changes where we can make the most impact, like our colleagues, right? Things are going to change with them. Things are going to change with us. And what we feel like is happening is it's shifting where we bring the most value based on qual strengths or the moments that matter, which is a framework we use with customer research. We feel like really plays a a great role and is a great analogy to use here as well. So, the moments that matter most for research impact, especially as qual researchers, here is going to change where it's going to emphasize further up in the pipeline. So, we're not saying that uh we never need to be involved further down the pipeline, but instead this is a reminder where we should maximize human resources and where we should leverage AI capabilities, right? Because it's never a question of either or, but instead of both and, right? How do we use this also to maximize the overall impact that we can have and questions that uh we feel like, right, or we can lean into uh with human qual research are things like what problem spaces matter most? How is the user experiencing this problem? What is their mental model? What actually matters most in terms of the pain point, right? Like, is it really what they say it is or we observe them? Like, is there something else that is really motivating them? What opportunities for differentiation can we explore? What might expanding our role with users look like? These are all great opportunities for qual research to be able to explore and also bring novel insight in these areas. So, what exactly is new? Well, not really that much. It's more of a reframe, right? So, as a reframe of where we can be valuable to the business and where we can deliver outcomes both to the business, but also to the user. So, going from de-risking product decisions, which is a really prevalent and dominant reason why businesses say right now or or site right now why they're using user research, right? It's all about de-risking the product decisions, testing the solutions before they invest so much in time into building something, and then also like having benchmarking and usage and statistics. And what we're saying is those still those things, right? Testing solutions still might be helpful. Benchmarking usage is still going to be helpful. But really in this world, like leaning into directionality and decision-making. So, instead of just testing solutions, us testing hypotheses, uncovering what metrics matter most so we can guide experimentation, and also be able to effectively understand and sort through massive set like massive um pieces of data, right, and direct, right, if we have AI moderation, right, be able to like use it in the most um effective ways possible. So, now let's go into the research techniques that we promised in the beginning, right? We want this to be inspirational to seize the day. What does this look like? What are the techniques that we are using that we found particularly useful in this space? So, first I'll start with discovery as a key moment. So, really an opportunity to diverge and um inform early into this pipeline. So, earlier again the citing this eye care example was in-home research, uh but we can undertake more novel forms of research in the space. Things like ride-alongs that Jenny already mentioned, expert interviews, maybe not always having to be strictly a user, but who else else can we from in this space? Sitting with extreme users or like extreme influencers or experts, extreme communities, and of course there's also analogous research as well. I have an image here of a site visit we did in previous work to the MIT Media Lab to understand really deeply some upcoming technology research so that we can understand what are the forces that they are even projecting that might change our world. Um and be able to kind of inform that or in like embed that into our thinking before we did some user research. So really using discovery to immerse in the problem space, uncovering new mental models, unexpected connections and forces that are going to shape the future or may shape the future. And especially because accessing pre prior knowledge is going to be easier than ever, right? So how do we diverge in this space? And I wanted to emphasize that AI is rapidly changing behaviors, right? Like part of the the thesis of this is that the product development cycle is changing fundamentally. So because things are changing so much and mental models and approaches and behaviors are changing, this is really an opportunity for qual research to go in and understand what those changes are. What are the new approaches? Like what like how are folks uh handing off or collaborating with their uh with with their colleagues, right? How has that changed? Questions like prevalence of individual agents, changing worker interactions, questions like how are storing and sharing information different? Um how are expectations changing? Are roles and responsibilities changing? Things like that that we can really lean on qual research being able to um give us insight, but also help us capture that data in a more holistic way. Then moving into the next moment. Uh so this is about co-creation. So we're positioning this co-creation um to gather directional feedback and really help in terms of understanding where to focus, but also if we're moving in the right right direction with solutioning, right? Like testing those hypotheses and testing if we're understanding a problem correctly to really make sure that we are moving in the right direction and inform that directionality and strategy that we mentioned before. And here's really about closely partnering with product and design. So co-creation that is allowing for users to build uh with you and uh co-imagine with you. So without the fear of tunnel vision of already landing on a solution, but also a way that we can deepen understanding of needs while still using things like boundary concepts for deeper exploration of certain directions, um enabling also quick building, enabling also the product team to quickly build uh and and understand what they need to do and build confidence while also us as researchers being able to insert foundational knowledge. Um I call I like to call it like baking in the broccoli, right? Like how do we help them move fast, especially if they need to ship something soon, but we can still kind of close those knowledge gaps as needed. And this becomes also a more effective approach, especially in a world with a of creating for emerging tech where the solution may not exist yet or the solution is going to transform a behavior quite drastically. So now moving into final technique that we've been leaning into a lot in this world is immersion. So really empathy building, getting to write together, and leaning into influence and shaping thinking. So again, right, in this new world information is democratized, which I think is a fantastic thing. But the role of research isn't just gathering information or being the keeper of the knowledge, instead we are stewards really. So being able to immerse ourselves and our partners really into the context of users and the perspective of users so that they're able to have almost a cataclysmic effect um that happens with spending time in the field and that gets everybody to alignment and that helps them work faster, right? Sparking new ways of thinking that'll inform how they go into a knowledge repository that is uh facilitated through AI and search for information or how they decide what to vibe code and what to prototype in the first place as they're thinking through something, right? So really still leaning on one of the key impacts of research of like um shaping minds, creating mental models, uh digestible frameworks, just helping people think about things in in new ways. So to summarize, uh we need to seize the moment, right? Qualitative research is absolutely needed right now and we have skills in our arsenal that are already suited for this. And although there is a lot of change and we should learn and adopt new tools, right? We didn't go into this, but there are areas and great places to embed AI in the research process that are going to improve outcomes. We should also feel equipped for the moment and equipped to understand where we can use what when for the most impact. So hopefully this inspires us to think about your own moments that will matter in your org's design processes. What should you lean into? What does baking the broccoli in uh for your stakeholders look like? What does prioritization of of projects look like? Uh what impact uh should you focus on with with stakeholders? And that is the end and now we will transition to Q&A. Yeah, and Amy, there are few few things in the chat. So even we have some we have the discussion thoughts slides discussion thoughts starters on a further slide, but I want to just call out some things from the from the the chat. So um and I'm looking at the Zoom chat. I have no idea what might be being said in the YouTube and I know that Luke is here to help us out. So um Luke, I'm going to I'm going to jump into the Zoom chat and then maybe if if there's anything you want to flag from the live stream. Does that work? Absolutely. Uh anyone who is watching via the YouTube live live stream, excuse me. We are monitoring the chat and if you have any questions, feel free to post them in there and I will pass them along to Jenny and Amy. Jenny, please. Thank you so much. Okay, so um so I I want to first call out call out a comment comment by Julian Lud about immersion being one of the most impactful ways to get Julian's stakeholders out of their heads and into the minds and behaviors of our consumers, clients, and customers. And honestly, like if that's one thing, one thing you can do, if you can find a way to help your stakeholders, whether it's product, whether it's engineering, whether it's design, whether it's someone high up the food chain who's actually signing off on budget decisions, if you can figure out a way for them to really like like understand that, right? Like if you This can even look like an immersion room, right? Like Amy's done this before where you build a room, a literal physical room. Maybe they can't accompany you on research, but you build a room that kind of creates a space where they're experiencing what the user is experiencing. Um so thanks, Julian, for dropping that in. And uh and now I'm going to back up and I want to note that Jody asked what kind of qual analytics software are are we using? I'll answer that for myself. Amy has a lot more experience with this, but um I've mainly been using Dovetail, which is, you know, uh you know, you upload upload your recordings and meeting notes and things like that. And Dovetail does not Dovetail's AI does not score very well on the various frame like the various reference points, like how accurate is AI. So I really don't use Dovetail for much than much other than like initial initial kind of like, you know, identifying themes. I really do a lot of it the old-fashioned way where I'm coding I'm tagging and coding and, you know, sorting. I do a lot of work in Miro, actually. Amy, what are you using for analysis for qualitative analysis? Yeah, right now I would say I also am using Dovetail mostly just for the tagging. Um I have been trying to experiment of what does it look like for descriptive level, like very deductive style tagging to like how do I make the Dovetail work for me, but I even when I there's the AI-assisted, I it's all my own tags. I find I have to like turn it off. So I'm a big Dovetail person for tagging and then converting the Dovetail highlights into uh stickies that can be used in Miro for clustering for like old-school affinity mapping and clustering. Um Yeah, it's kind of what So still kind of low-tech low-tech high-tech, I think it's so what. >> Um I'm going to take one more question from the chat and then there are a couple in the Q&A box as well. So let me start with this chat question from Nicole. Can you talk a bit more about the qual data you got from the driverless pizza delivery? Did you ride along? Yes, I did ride along. So the the car was skinned, right? It had stickers on the window so that the driver of the car, cuz it was a simulated autonomous vehicle, so there was a driver, but you couldn't really see the driver unless you were looking and you knew what to look for. And I was hidden in a back seat, not a compartment, it was like a legit back seat. Um but you really could not see me there because of the stickers on the windows. And so we'd start at the pizza place, we'd load up special compartment in the back to keep the pizzas hot, yada yada yada, right? And we'd go to the delivery location, and then we'd wait. And we were using all the standard 2018 2019 standard means of communicating, right? It was usually text message updates, sometimes there were apps uh to communicate like, "Your pizza's here. Come on down and get it." And, you know, more again, as I shared already, more often than not, no one would ever come down. And Miami's a really challenging environment for delivery drivers. Um parking is really challenging. That's part of the reason they were looking into autonomous vehicles. They figured if if it was a vehicle that could kind of quickly wait and the person would come and get the stuff and then leave and the the moves on, you don't need to park, right? No one needs to park in order to make the delivery. But, that's exactly what people in Miami want is they want someone to literally take the thing from the car to their doorstep, whether they're in a high-rise or for whatever reason. Uh so, that was really really rich data that we got from observation. I also um another similar thing that we were exploring was just like restaurant delivery and and grocery delivery and but the same thing happened. People didn't want to actually come out. What the final solution was was actually to think of a last mile like um a sidewalk robot, right? But, that's not practical. It was it proved to be too expensive and um Ford actually abandoned this line of inquiry and did not pursue that and then the pandemic and all that sort of stuff. Um I'm going to go over to the Q&A. Alyssa Buchanan dropped a couple of questions in there. How have you identified companies that invest in research so deeply and allow for such budget-intensive projects? Okay, so that's really interesting. These are actually not Okay the in theory, the autonomous vehicle explanation is is is budget-intensive because you need a vehicle, right? But, Ford is a car company, so that's not really a problem, right? Like, you have vehicles available. The stuff that um that picture Amy, if you want to back up a few slides to that photo you used for the immersion. Um that photo's from some research I did in 2021 and it was it was basically we were doing walk-alongs and community engagement research in a neighborhood in Detroit. There was no budget. We had no it was our only budget was incentives for our research participants. It was very scrappy. I all of us on the project were living in the Detroit area at the time. We could just like drive ourselves down and meet up and you know, we would we would literally just scour the neighborhood for people to talk to. So, very very scrappy. A lot of immersion can be done in a very scrappy and um like kind of resource-constrained way. That said, a way to understand whether a company is investing in their research is in the interview process. If you have an opportunity, Alyssa, ask what kinds of research vendors they work with. Do they use recruiters like Accelerant, right? Like there's a lot of work that Amy is engaged in right now in particular, where finding the person to talk to. We described it that person as a purple squirrel. How do you find the purple squirrel, right? There are no purple squirrels, but you need to find the purple squirrel to talk to. You're looking for really really specific type of part participant. And so, that's where a company like Accelerant or another recruiter is going to be absolutely aces for you. So, you find out in the interview process what kind of resources do they have? What is their research operations pipeline like? Do they have recruiting firms? You know, like do they contract with recruiting firms? Do they have the various software tools that you will need? All this all this kind of stuff. So, that is the kind of stuff you could ask in a job interview that is not going to seem out of bounds to ask, but will give you insight into how much the company is actually investing in research. Um Amy, do you want to add anything to that about investing in research or doing that kind of stuff? I think too. I think especially in this moment right now, like things are very scrappy. So, even a lot of the the most recent examples we're using or even even Jenny's example that wasn't as most as recent of like everyone lived in the area, let's try to go there. And also for like immersion, something we've done in the past of like if you can be in a room together, what does it like like Jenny said, can we create a strategy room? How do you really immerse people? Is it like a singular story or give people a metaphor that's really going to stick with them? Um but ways to like kind of dive deeper like make sure you're like implanting in their in their brains. Uh cuz I do want to acknowledge, right? Like budgets aren't aren't uh what they used to be. So, things are a little bit scrappier nowadays. And that is something that like we we have encountered as well. I do also want to back up about the pizza delivery qual data. Um you may not have been asked about this, but uh also like in terms of the actual pieces of data, it was a lot of like notes and taking photos, right? Like just actually taking photos in the moment. Us also you had like writing field notes. Like Jenny would write field notes of like how she was feeling in that moment of like remembering those things. So, uh pretty old school in terms of like what the actual qual data is, but stuff that like is very necessary, very needed. Like that is that is the stuff that um uh allows us to kind of find those insights. Thanks, Amy. Um I want to answer another uh a question that Alyssa dropped into the Q&A. Favorite knowledge repository platforms, especially ones that encourage stakeholders to self-serve research findings. I have a slightly spicy opinion on this. Um first I'm going to say that at uh Cisco, we use SharePoint and we use Airtable for our knowledge repos. However, I think that the best knowledge repository platform is the place where your stakeholders are anyways. If your stakeholders are in Aha or Jira or whatever, like if they're product people or engineers, you need to put links in there. If your stakeholders are in some other tool, you need to put links in there. You need to it's it's really interesting. As important as our work is, right? As much money as invested in these products and as important as it is for us to get this stuff right, research is what is short-changed, right? They go on guts. They go on an attractive design, a really high-fidelity prototype, whatever. So, you have to just as a researcher, we have to try and meet all of our stakeholders wherever they are. Whatever is going to work for them, I think we have to push our stuff to them in that way in that space. Amy, what would you add to that? I couldn't say it better. Yeah, we got to meet them where they are. We're seeing things and we know um that folks are going to want to like access things through agents. If they have an agent, what does that look like? Um So, knowledge repository feels more like uh I don't know, like a lens instead of like an actual thing. I don't know, like what is uh where is the knowledge? It's wherever the stakeholder needs it at a given moment versus like actually a singular place. Um I think a lot about pinning things to channels, right? If you have Slack, pin stuff to whatever channel the team is in. Uh push we've been working on we haven't we haven't quite gotten this off the ground yet, but Amy and I and another colleague have been working on taking insights from older research that are still valuable today and kind of like using them as a kind of editorial piece, right? Like, where do we push those to say, "Hey, remember this research we did a couple years ago and this one insight? This is actually really useful in this moment. Here's why." And kind of like pushing that in the spaces where they are today to kind of um remind everybody that research that is 2 years old can still be very valuable. So, um this is again something we're only just experimenting with and uh so we don't have any answers yet. Maybe that will be a future presentation, but um yeah. And Amy's showing some other discussion questions that we just we just came up with in case, you know, you all didn't have any questions. So, we don't have to discuss these, but um we just kind of put these out as thought starters. So, Luke, is there anything from YouTube? Uh yeah. So, um you had talked about what AI excels at, pattern matching, predicting, but not reasoning. Uh so, AI moderators are here or they will soon be here to the masses. Uh if someone wanted to give AI moderation a go, are there any types of projects that they would be particularly suited for and do you have any guidelines on what to avoid? That those are great questions because yes, AI moderators are here, right? Um and the Anthropic study that Amy referenced is a great example. So, I think that with AI with AI moderation, you want to start with something tightly scoped where you know the space, right? You want to start with something that like maybe actually the right way to do it would be a survey, but you don't have enough money to do a survey with an N of 400, so instead you do AI moderation that gets you at an N of maybe 20 to 30, which is going to cost you very very little compared to an N of 400 for a survey, and yet it will give you that kind of like you're you've you've got your questions nailed down. You've already identified one or two directions and you really just need to validate them. Maybe it's even about a specific audience, right? So, that your moderation is focusing on a particular persona, a particular use case, something like that. Amy, what are your thoughts on that? Yeah, I I showed this slide again cuz I think thinking about like where you are in the process or what types of questions you are, but plus plus to what Jenny said, like things that are already already tightly scoped, right? If there isn't um a lot of confusion or you've already have a lot of clarity, maybe it's do we test these two different versions or it's very like specific more limited variables. And also you know that um the question that you're writing isn't going to like the it's not going to change, right? Like it can be a closed-ended question to an extent and that like you don't need to keep flexing it based off of the participants' responses. So, to me I feel like it's things that are more like further down, like more after directional feedback, evaluative type stuff is where that would be quite lovely where like you can maybe have a larger sample. It is something that I mean, could you have already done it in an unmoderated interview anyway? That those to me seem like fantastic opportunities for AI moderation because if if folks have experience with unmoderated interviews, sometimes like if you got you miss one thing in the instruction, it like you know, what comes back is like very, very different. So, I see AI moderation of like being able to like improve on the unmoderated interviews anyway. Thank you. Uh in the age of AI, how can we as market researchers future-proof our skill sets? Uh what are your thoughts? What should we be learning that uh will be helpful to us later on? This is an Amy question for sure. Take it, Amy. I am going to disclaimer first. We are I'm not a market researcher. Um so, this is my research background has been really in the design and product space, but I do still feel that in terms of future-proofing for for research, just knowing our deep skill set of being able to like look at the world and see patterns that other folks can't see. I know we say that AI is pattern matching, but we kind of do a different kind of pattern matching of understanding like what is dominant, what's emergent, what's residual, why those changes are happening, like what does it look like to understand human behaviors, and when things can't be predictable. So, I think still like leaning on that and just like deeply understanding humans and those like I feel like it's just kind of what we said that like the qualitative angle of it so that you are very able and capable to know how to leverage the data, right? If you have a deep understanding of what is going on just in general, you know what which ways to cut the data, you know like what questions to assume like market research you do sometimes deal with a lot more like larger quantities of data and larger samples. Like you will know like where to find all the directions or how to model something in an interesting way that other folks may not be able to know. Something that a colleague had shared with us we couldn't figure out how to how to include it, but also I'm sure we all have seen that like MIT media study of like your brain on AI. And one of the takeaways right is like if everyone's using AI, everyone's like outputs are going to be the same, right? And I think that's the thing with anything. And a good strength about us as qual researchers and even market researchers is that right? If you have a larger, deeper understanding of just like how things work and how humans work or how to understand humans, even being able to like have a different perspective of when you do use AI, like your inputs are always going to be different, thus your outputs are going to be different. So, hope that was Jenny, if I don't know if you want to add, but um yeah, hope that answers your question. Okay, thank you. Just a reminder to folks who are live streaming via YouTube, we are taking questions from that channel. If you have one, please feel free to post. Jenny and Amy, I don't see anything else open, so if you want to drop to your questions you'd like a chance to answer, that would be great. Great. So, all right. This This actually Amy and I collaborated on these questions. This first one, this really really is something that I feel very, very strongly about. Um low-fi prototypes are important. I don't like to show someone really developed wireframes because then it seems like the thing is done. It seems it it limits the imagination. So, this is something that I struggle with is to really advocate for low-fi, sometimes even paper prototypes, right? Like stuff on sticky notes. And so, I'm going to ask this question of Amy, right? Like like it's something that I struggle with. I'm going to ask Amy. Amy, how do you advocate for low-fi prototypes? Um I'm going to be really honest. I haven't been winning this fight, but I have been being able to uh if the prototype seems really high-def, I like work with my stakeholders of okay, can we just show them one? And instead of them clicking through it, like I want them to walk me through how they would. Like just use it as um a pretty picture and that is going to spark imagination. Um and then really kind of lean on how we position it. And I've actually found that even though I said I haven't winning this fight, I think, you know, the war the battle, not the war. Um I am seeing from stakeholders like they're seeing the difference, right? Of when they are over-explaining or when something is coming off as polished, what they get out of a participant versus something where especially when we're early in the pipeline, when the participant gets to kind of impart their own imagination in things. But I will say that because of the like AI prototyping, it is faster for them to give me a vibe like a vibe-coded prototype versus them like sketching something and then having to upload it and then any changes that needs to be made, but um yeah, so I think even like at least getting to the point of like if it looks polished at the very minimum, like we won't show that it's polished thinking with the participant. So, if they're like, "How does this work?" I'm like, "You tell me." I love that. I love that. The pretty picture, right? Like just stop it as a as a picture and not a not a thing. That's great. Amy, is there a question here you'd like to bring up to the group? Yeah, I think this third one. If designers are moving toward engineering and engineering is moving towards business outcome ownership, what do you think researchers are moving towards? So, this is something we talk about a lot in our on our team. So, Amy and I are on the same team internally at Cisco. We talk a lot about the importance of research as a social process, and research having a kind of unrecognized and certainly undervalued function in getting everybody to collaborate and align, right? And so, you could argue that's kind of a business outcome, but it's not really, right? It's one of those under like under-appreciated things about uh getting different functions to collaborate is is that actual how do you drive alignment? And um what I struggle with still is how do we get recognition for the value we bring here, but I think it's very, very important that research kind of steps up even further to this like leading on collaboration and a forcing function of collaboration. And then another thing that I think researchers need to move toward, and this is maybe a more wide-open space for us, is having opinions, right? When I remember when I was at Ford, another researcher was like, "Well, our job is just to do the research and not to make any recommendations." And that is not at all what our stakeholders at Cisco want. At Cisco, they want us to come and advise. Like, "Okay, the research is telling us this, therefore what should we do with it, right?" And so, that's us getting our hands dirty in a different kind of way and like literally what might a design look like next, you know? What should a next-step product offering be like? What are your thoughts on this, Amy? Yeah, I I like I think you're spot-on. I think also moving towards becoming more embedded also likes to be able to get to those recommendations in a way where we can still kind of have the data where like if we need to go back, right? The data's there. We're not going to let the the need for recommendations like color the data that we are um that we're collecting or that we've generated, but I do I am finding this also role of like researcher delving deeper into the domain as well. Not So, like we still kind of occupy a neutral space, but just being able to then understand like what is an assumption, what is like what's a recommendation, what is actually a truth from the customer perspective, and being able to like get like even better align our stakeholders because we can call things out. And I know there've been a lot of models in the last few years with research of like embedded versus democratized like spread out and going back and forth. I mean, I see in this world that like if we're moving toward something, it's seeing the value in really embedding and and partnering with with teams. Oh, I like this question that Deborah just posted. How do you overcome, say for example, engineers negating customer feedback in qual because the numbers are small? Oh, we experience this all the time. What number of interviews carries more weight? I think that I think that when you start to get into the double digits like 10 to 12, that starts to carry more weight. Definitely when we have when we have sample sizes of six cuz we just a colleague and I just did a very scrappy study, sample size of six, and we got a lot of pushback on that. And rightly so, right? Six was not enough for the question that we were investigating. So, when you get into double digits, but honestly, Deborah, I would say like even more than the raw numbers is when you start to actually see an interesting pattern emerging from those, right? So, but but then it can also just be like you've done enough numbers where like maybe there's only one person who said a thing, but that thing is really like really resonates with everybody. So, I don't think there's a hard and fast rule. Um, what are your thoughts on sample size for qual? I mean, I think it's like knowing your stakeholders. I think it depends on like what is the goal? Like did they need more number Yeah, like helping them kind of understand. I think it when they're it's hard because it's like real like sometimes you only can find six. Like we're working in a space where like people these people don't like there aren't about so many of these people who exist, right? So I think um yeah, like working with the stakeholders to figure out like what is it about the numbers that they want? Um, is it because they want to understand like market value which then we can communicate that like we still don't do that at the end of the day, right? Like this isn't what this is supposed to answer. Um, but yeah, I don't know. I don't have like a I don't have a a clean question for that, but I think you're spot-on Jenny. And thanks um, I think that uh, Jody noted insight saturation is a goal and yes, nice and concisely put. Thank you. I recognize that we're pretty much at time and Bill's got to come on at noon. So Luke, you want to wrap us up? Yeah, absolutely. Uh, I see no more open questions. Thank you so much for the presentation. It was absolutely fantastic. Uh, I see a number of people have posted the question, are these sessions being recorded? They absolutely are. They will be available on the Accelerant Research YouTube channel and uh, I believe selectively some of them may be available on the Accelerant Research website as well. Thank you so much. Thank you, everyone. Thank you. Have a great rest of your conference. everyone.