Without Research Thinking, Your Democratization Program Isn't Research. It's Research Theater.

All right. Uh, awesome. So, both presenters, you guys are both here. Uh, Jim, Denise, thanks for for joining. Um, I'm just going to go over some ground rules. So, I'm Breji. Uh, so I'm VP of research and growth here at uh, Excelon Research. Um, so I'll be kind of co-hosting, moderating this session today and letting our our awesome presenters kind of uh, share their knowledge. Um, but before we go into that, uh, we're going to jump into a couple different ground rules. So, uh, one, you know, thanks for everyone. I know we we have an ongoing conference and uh I'm sure people are going to filter in and out as we go throughout the day and I think we have a couple more sessions um to finish out today outside of this one but uh you know we are at the mercy of tech right um you know obviously we're we're always at the mercy of tech in the online world and and trying to make sure that uh things are going smoothly whether that's uh a video that you know is is in these presentations or just the ability to kind of share uh you know you never know what might happen. Um the other part is uh we love engaging conversation. We love you know comments, questions, all that stuff. We do ask just be respectful. Uh people are taking their time out of their day to you know present uh share the knowledge, share the experience they have within the industry. Um you know we keep this ungated for that very reason. We want to try to give back uh promote uh things that you guys care about. Um so please be respectful to everyone in the the chat as well as our presenters today. Um, and then finally, you know, use this as an opportunity to network, engage, uh, interact with each other, right? Take things offline afterwards. Um, you know, follow up with our presenters. I'm sure they wouldn't mind being able to field a couple questions about the topic that they're going to be presenting today. Uh, so feel free to interact with anyone. Um, you know, ask questions throughout. You know, this is meant to be engaging, meant to be, uh, a conversation. I'm sure that just makes it better for everyone uh, and the purpose of today. Uh but without further ado, um I will uh kind of hand it over to our presenters and let them start sharing their screen. But before I go into that, I'll do a quick intro. Uh Denise and Jim, but today they're going to be talking about uh research democratization, right? A hot topic uh for all of us in the research space. Um and I'm sure they're they're going to have some interesting things to say about what's worked, what hasn't. um but also the contention of that of uh are we actually training people to to do our jobs for us or you know what is what is the role a researcher has but uh guys I'll be uh here throughout the entire talk um I'll feel some questions if you're fine with me uh bringing those as they're asked um that's probably what I'll I I'll do best but uh I'll stop sharing uh let you guys uh take over and and do your thing Okay. And there we go. Hello. Uh, welcome to the presentation. Um, Denise and I are really excited to share this presentation with you. And we have a lot to say. And in fact, we have so much to say that we may run over a little bit or we may not get to everything, but uh, hopefully we'll get to as much as possible. Um, what I'd like you to do here is Okay. No, this was working this morning about a few minutes ago. So, talking about technical issues right now, we had a poll right here in which we were asking and for some reason it is not up and running. It might be a problem with uh the tool that we're using with Zoom. Um the question is though, and perhaps we could use your your chat interface here to say um what is your role currently? um if you're a researcher or if you're a project manager, a product manager or something like that. Um and we'll see why that isn't working. >> Possible. Yeah, go ahead. >> Possible to share the link in the chat. >> That That's interesting. >> If it's gonna let you do it. It's funny because we've went over this. >> You did it just a minute ago. >> Just a minute ago and everything was working. Here we go. >> There we go. >> If you guys could just take a a screen capture a phone uh of the QR code, um that will take you right to this to this survey quickly. Um and if you could help us out by answering these questions. In the meantime, what uh if y'all I'm going to go to the next slide for a second. Hopefully, you got the QR code. I'll I'll hold it for another couple of seconds while Denise introduces herself. So uh I I am a UX researcher, design researcher, uh and strategist. Uh our titles were backwards. Jim is actually the director. I am an IC right now. I've been a director before, but I've also my career trae trajectory has been that I've been building and scaling research programs without headcount which so I was democratizing research before I ever heard of it. uh and I know and I've had the experience across my career to know when it works and when it doesn't work. And one thing that we're also going to be sharing today's a little bit about is we couldn't be talking about research now without talking about AI and research. So we're going to cover that today too. >> Yeah. And from my perspective um I've been doing this for decades uh researching and also being um a UX researcher as well as a strategist. um um as as well as a designer as well. I've been doing this for quite a quite a long time, mostly as a consultant, but oftentimes in the in the the product uh as well. Um I I've known Denise for about 10 years. Um I was uh one of her mentors u and one of her graduate degrees. Um so uh I've known Denise. She's fantastic. I'm very proud of all that she's accomplished. Let's go back and see what this looks like. So, so far we have a lot of others. What is your role? If you guys in your chat, if you could tell us what the other is, if you've selected that, that would be great. So, we've got some researchers strategists marketing brand, executive leadership, and let's see what's down here. No designers, project managers, product manager, content specialist, engineering. So, we've got a lot of others. That's that's curious. I'd like to hear what that is. Um, and if uh Denise, you're ready to go on, we can come back to this. >> We can come back to this. >> Yeah. >> So, democratizing research isn't new. As I mentioned, I've I I I've been democratizing research across organizations since it's been well over a decade, but even before it was called de research democratization. One of my other mentors is Tomer Sharon and I'm not sure if uh if anyone is familiar with his work but I uh even though democratization was already happening if he kind of without coining the term he he leveraged it which is to make research our research. So but from his perspective it was more collab collaborating with your stakeholders. So bring giving them a seat at the research table. So you so researchers have a seat at the larger table. Right. And uh Erika Hall in 2013 made the case for making research accessible to everyone who's willing to be rigorous about it. And then Jen Cardello is still kind of she is kind of the democratization diva. She coined the term. She's the head of research at Capital One where they have a a very fully robust democratization program. And the root of the problem is that there's just never been enough researchers to go around now more than ever uh for for various reasons, right? So, uh we're going to go into that and more and what you can do about it like how how to make research democratization work. so that it values research, adds value to research instead of devaluing research. >> Um let's try this one next. Uh which statement best describes your experience with research democratization if you could select uh one of the following. Um, I'll leave this up um up there for another few seconds and we'll talk a little bit about our our methodology for the research that we conducted to produce this uh to produce this presentation next. Our methods and rationale. So, we did a lot of desk research and and we're in the process of interviewing uh people who are conducting research regardless of role. And what we found is kind of interesting. We wanted to understand what the people who were actually conducting research had to say because most of the most of the opinions articles uh studies around research democratization are by people who have more skin in the game. Uh whether it's researchers but from or or for the most part now it's really research tool vendors. They're they're they are conducting the most studies, the most attitudinal studies and that serves two purposes, right? It serves it it strengthens their brand and it they're continuously measuring product market fit, but they do come up with some really interesting >> Yeah. >> some really interesting insights if you keep in mind that caveat that this they do have skin in the game when they're doing their research. Um let's look back and see what this is looking like. Okay. So, never heard of it before today. So, this is this is news to a lot of you uh people. So, maybe Denise, if we just Oh, we're doing it and it's working. Okay. Let's see what else we have here. Maybe um just to clarify a little bit if you've never heard it today. What exactly Denise maybe a concise definition of democratization would be helpful? Well, it's interesting that um that these are the stats for this group, right? Because all research democratization really is is empowering pe people who are traditionally not researchers to conduct research. It's successful when it's embedded with research thinking and research training and tools. It's not as successful when it just when it relies on anybody can do it. But if you go to the next slide, Jim, this is really interesting that you're 50/50 doing it and working and never heard of it because uh next slide. The experience slide. Next one. >> Yep. >> When the people that we interviewed, it was a convenience sample. It was people that we've worked with, we've trained to conduct research uh and and and people who have and in one case of our a head of research that's implemented democratization into his research practice. Yet six of the 12 who we selected because we know they've been part of research democratization programs whether formally or informally half of them never heard of research democratization even though they're doing it. So it's not surprising that that you may not have heard of it, but but be but but research is democratized regardless regardless of of whether it's formal or not. Anytime people who are not trained researchers conduct research, that's democratized research. It's just a matter of whether it's effective or not. So what we heard from people is that that uh there was a there's a lack of organizational tracking of research activities. I think Jim can talk more about that because that that's that that's uh that's a thorn in Jim's side. >> Yes. Um with it's kind of chaotic right now in the field when you're talking about research. So many people are doing it and yet they're not talking to each other about the projects, the research projects that they're engaged in or even what their findings are or the tools that they're using. Um, typically when you're in an organization or a team, it's uh it's really these activities, these research activities are not being tracked at all. So they remain largely invisible. >> And research can be invisible when it's conducted by by professional researchers as part of a research team. But when it's in when it's conducted by non-ressearchers like say strategists for example when I was at when I was at Moody's the head of design uh mentioned that most of the discovery or all of the discovery was being done uh by strategists but they were a black box. They never explained how they got to their findings where they came from. Was there any rigor or or uh it was just came it came out in the in the form of strategy with no empirical evidence behind it but according to the strategist they had done research. >> Yes. I think it's even worse for consultancies because they may be doing research across multiple projects across multiple teams and there's not a single place where we all gather to talk about the research we're conducting and how we're doing it. I think the the other part of that is the the lack of an effective research repository. No one we talked to had a good repository for all the research that they're doing. Go ahead Denise and talk more about that. >> And even the one person who is part of a democratized research program and enablement program uh mentioned that when they did uh when they did stand up a SharePoint repository uh there was basically no usage of it. But then uh but then another person that was in a democratized program the they they noted that product managers were using it to validate their opinions rather than test them against the research in the repository and we all know that AI is extremely uh we'll say complimentary so unless you explicitly tell it to be critical it's going to tell you you are amazing your ideas are the best and you should move forward. Uh >> so it's hardly surprising that um the implementation of these findings in the research is is really sorry or lacking or in many cases it's uh decisions are made that cannot be traced back to the research that has been conducted to support them that's not articulated in any way. So there's uh and that's because largely there's no established process for turning findings into action. Um so that that continues to be a problem I think both on the product as well as the consultancy side. >> Okay. >> Um so uh what we were kind of surprised by from this uh 19 2023 May study is that 64% of companies report having a dem democratized research culture. Um and the breakdown with that is 69% of designers conduct their own research. 28% of marketers do it. uh 54% of project managers and um even 10% of engineers conduct their own research. What we've noticed is in the years that have uh since then these numbers have really escalated and they're much higher. So democratization is happening everywhere. Um the the uh people also report within the last year between the 2025 study and 2026 is that research is growing in importance and is valued more across all team members. Um in in 2025 just 8% of uh particip participants said that research was essential in all levels. Um and now that's gone much much higher. um uh it nearly tripled to 22%. And at the same time 40 41% report that research now informs both product and broader strategic business decisions. So um as you can see research is growing in strength and growing in importance um and it continues to grow going forward. Um also vital for decisionmaking. This was kind of this was an interesting chart here between 2025 this May study in 2026 is that customer support is increasing u the importance in implementing user research and their decisions sales increas has increased engineering has increased executive team that has shown increases data analysis and marketing what I thought was kind of interesting though was when you look at design and product below here it seems to have decreased slightly in importance uh implementing user research and Denise has an idea about what's behind that I'd like her to share. >> I I have a thought and we'll talk about this later. It's that it's surprising that design and product because they are would would value research less in their process now year-over-year because they're our closest partners, right? But this may actually be an indication that just as AI is pressuring research to increase velocity, right? Designers and PMs are under the same pressure to move faster and that there's a pervasive sense that research just slows that down. So if pro design and product and even engineering are under pressure to accelerate go to market, the first thing they're going to drop is research. And I've I have uh right now I'm I'm contracting at at Google as an IC and what I'm seeing there too is with AI that even from up from leadership there that they want to they're putting pressure on research to stop being a blocker to go to market. >> So so we don't see ourselves as a blocker. We see ourselves as as inherent to success, right? But the but uh if we're being asked to move faster, so is everybody else. And we'll touch on that later. >> U this is also kind of interesting. An incredible increase in AI use. Um nearly two and three researchers now are using it in their everyday work workflows. Um this is an 19% increase over last year. So it's almost 20% in in in a single year. And when you look at what they're actually using it for, this is this is interesting to me. Analyzing research data transcription that that makes sense to me. Planning and drafting research studies. So they're they're actually using it as more of a collaborative process, generating research questions, again using AI to collaborate more synthesis and reporting. I was a little bit surprised by the amount uh the high amount of generating content that kind of surprised me is for a researcher um that they're using it to that extent. Um taking inspiration um notetaking also uh generating AI prototypes, vibe coding, asking follow-up questions is also a little higher than I anticipated. What I was a little bit shocked by is the not using it as much for recruiting participants and I think uh Denise has a reason for that as well. >> Well, this again most of the literature and research that we found out there were by people with skin in the game and this this is a study conducted by Maze which is a vendor that's heavily invested in AI augmented research but not recruiting tools. So this so even though it's 500 plus UX researchers, PMs, designers and strategists and executives, it's possible their sample a sample survey size. Yeah. >> Yeah. It's possible that it's it's heavily weighted by their own customers who may not have who may not have access to tools that help with recruitment. But I know I know for my own use that that AI is really excellent at helping create screeners for example and and also for uh for analyzing screener responses and uh and uh deciding who to keep and who to let go. >> Yeah. Um here is interesting to the benefits of AI research according to this particular um particular survey. It boils down to speed. Um it very much increases um the speed of research turnaround and also efficiency. Um using fewer people um at a higher velocity is basically what it's about. What I found interesting about this though, the benefits of AI and research, look at the the the relatively low percentage thought that it was greater quality and accuracy of research. So, while they're saying it's speedy and it uh it's better for resource allocation, they're not really saying that it actually increases the accuracy or the quality of the research, which I found was very very troubling and interesting. And also here it's it's interesting with democratization in general in addition to AI there is an an incredible lack of training and support for u people engaged in research activities and this particular poll shows that um that there is um we're really um we're not achieving the desired state of implementing guard rails and support for non-reachers non-ressearchers who conduct UX research activity. ities. Um so, um the researchers themselves, yes, they're they're they're close to an ideal state, but um they're they they're not getting standardized templates. They're not accessing best practices of data governance privacy and compliance. Um and also there's a lack of sort a profound lack of training and certification before conducting studies. So, um, people are doing it. Um, unfortunately, they're not they they don't have the tools and they don't have the training in order to do it is what this particular survey is is saying. And again, um, uh, this means that u it's, uh, the quality of the research suffers uh, oftentimes with democratization. And, and Denise, I I'm going to let you do this one because you found this quote that I just think is so amazing. Well, what uh what I think is interesting is here is and it's true to to a large extent a beautiful deck can have terrible data in it can have you can they can come to the wrong conclusions but if it's a beautiful slick deck stakehold most stakeholders won't see through it. We'll talk about this later, but but you know who does see through bad research? Executive leadership. If you're presenting at especially at the seauite, they're going to push you on they're going to push you on every data point. So if you're if you're going to them with good data, right, with reliable data that that you've been triangulated that you know is going to move the needle, you can defend it. But if it's bad research, it's not gonna it's not gonna stink until it gets to the top. It's gonna look really smart and really good until until somebody who's seen hundreds, if not thousands of slick decks with uh with not with with varying degrees of of reliable data is going to see right through right through it. But a lot of people in between are not. I love the comment section, by the way. >> Oh, I I'm busy running this. So, Denise, go ahead and go ahead and say when you see them that that are great. >> Uh, no, no, it's we'll go to it later, but I but >> Okay, super. >> I don't want to lose momentum because we're both very chatty. That's so so there's out of the interviews that we did and just also anecdotally we found that there are four main and a bonus actually five main myths about research democratization one being the right tools talking to you AI uh anybody can do research and that research methods should be segregated by role is another is a flip side flip side of that which is stakeholders should be doing lower lower risk lower impact research and or more tactical research and professional researchers should be doing the the more more strategic more uh impactful research and research democratization programs and AI replace researchers. They can but they don't have to. We'll go show that, right? And successful democra democ sorry democratization programs originate from the top. And then a bonus one is only researchers being democratized. So next well the tools don't make the researcher. Most research platforms are built by engineers and they're marketed as demog democratization tools to everybody but researchers. and they've kind of moved they've kind of turned research on its head. Like for example, I came up through the ranks of recording my interviews, but having them the methodology in place that I'm walking away from my my study knowing what my findings are and only using the the transcripts and the and the and the video to remove doubt, right? and uh and to validate validate my findings rather than than wait and for the transcripts and the video to start your analysis. Right? So, so you have to really think about what the tools are really doing and whe and whether they're really facilitating or whether they're really facilitating good research because guess what? going back to transcripts and videos uh uh is ownorous and it could be dangerous. But thankfully AI does help with that and we'll get to that. But what you really want to do is if you have a democratization program, whoever you are, whether you're whether you're in uh program manager, whether you're product management, design, research, you you want to make sure that you're choosing at least at first the lowhanging fruit, the people who already kind of think like researchers, the people who have the aptitude and the curiosity, and I would even say empathy and humility. >> Yeah. to conduct effective research. The people who don't want to conduct research, they're g they're they're just not going to do it. >> But uh >> or they're going to do it poorly. >> Or they're going to do it poorly. >> Yes. >> Uh and or they're they're going to do it poorly or they're going to they're just they're no matter even if they don't do it poorly, it's not going to change their minds. So, and that's what the research is for. The research isn't there to validate. The research is to is to provoke thought and reflection and insights, right? And uh it's not there. It's not there just to tell you that you're awesome. Uh so re research meth segregating by role. Yeah, we can a lot of programs they they democratize so they can save the most impactful um strategic research for researchers. And I I say yay to that because that's the research I really want to do. That's the research Jim wants to do, right? >> Yeah. >> Uh but >> it's a fallacy to think that that that your stakeholders aren't already doing that. You're they're doing it and you don't even know about it. They're they're talking to customers. They're talking to prospects. They're they're out in the fields. A lot of the times they're uh they're running their own surveys. They're referring to surveys to to desk research that you don't that that you may not even have access to as a researcher, but it's definitely happening. And you can't you you can't stop them and you shouldn't stop them, but what you can do is educate them to understand what to do with that research. So, the answer to this is yes, it can. It absolutely can. But it doesn't have to be that way. The the beauty of AI actually is that in order to understand whether AI is producing accurate effective results is to have expertise in that domain. So you if so if you have the right framework whether you're you you're training human stakeholders or AI is another stakeholder really or or AI you need to have or you need to have the right framework the right and the right tools um and materials to implement successful democratization and to implement successful AI research and that's embedding that's embedding research thinking which is can can be partially taught. It has to be partially innate into that process and you can definitely teach AI to think like a researcher as much as AI is is is able to do that and it's and it's through frameworks. Uh successful democratization programs originate from the top. I can't even say this out loud. Um because it's it's it does not it really comes from the bottomup need with advocates in the middle that go to the top. So you need to meet the bottomup need and the bottom up need is usually there just aren't enough researchers to go around. So you have to rely on on uh non people to conducting research who aren't trained researchers. But you want to make sure that you're adding value with all of that research and that you have visibility into that into who's conducting the research and also who's who's training the research the the the people who are conducting that research because once you train stakeholders to conduct their own research and if they're doing a great job, you're not in those presentations anymore. You're not in the meetings anymore. show upper management think that they just hired these stellar PMs and designers which which if they're doing really great research they are stellar and forgetting that that uh that the knowledge that that the knowledge that they have came from somewhere and and depends on on again a framework tools uh tools and resources. So you want to make sure that there's visibility at the top. If the directive comes from the very top, it means they're getting they're not investing in research anymore. And bringing which brings me to the the big myth only research is being democratized or I guess it used to be a myth. It's not anymore. It's a reality. AI is coming for everybody. Oh, across domains. Uh and it came up a lot in the interviews that we did. AI is facilitating the merging of specialized roles. So product and engineering are can uh can skip requirements gathering documentation and design and go straight into prototyping and even coding right uh just to see what sticks. Uh a design manager at a startup told us that product and engineering merged into a single pillar with developers and designers competing with each other to create prototypes based on customer feedback. So they're competing with each other each other to talk to customers and come up with the bea the best solutions, but they're usually not real solutions. They're just uh based on real problems because if you don't know how to define the problem, your solution probably isn't going to fit. and uh and somewhere else uh where they have already eliminated all the researchers, they're requiring PMs, engineers and designers to all have the same skill set in terms of of prototyping and testing and even though it's they are embedding their they're using cloud code to embed their prototyping tool to ask uh you know problem statement questions who your audience is. Uh if you don't know how to articulate it, you're just going to you're just going to skip it, right? So, so in that case to survive layoffs, further layoffs because the researchers are already gone, the the head of design is encouraging encouraging the designers to maybe move into engineering or maybe move into into P to product management so that they can uh be a PM who can design or an engineer who can design rather than being a designer uh as own as its own profession. >> Yeah. Just as a kind of back you up a little bit, last week um a potential client asked me why I would use a human designer when I could do it. And I was kind of shocked to hear that. Um because I think we all know that even if you have AI do it, you're going to have to have a person there to look at it or to to to evaluate it. It's it was it was a little shocking for me to hear that. and >> okay we cool okay so um when we looked at uh research thinking when you're using AI uh research thinking is absolutely critical to the success of any engagement or research that you do um this was again from the maze report and again I was expecting to have these like when using AI research where is human judgment most essential I would look at all of these and think that they're all essential right across the board. And in my mind, it would be 100% across the top. But when you look at it, they they seem to value interpreting nuance and emotion is important. Ethical decisionmaking is important, framing the right questions, delivering strategic recommendations, and influencing stakeholders through storytelling. That you really do need that human judgment uh to to engage with AI. Um Denise and I call that research thinking. Um and we have a lot to say about what are the the corner or the foundations of research thinking and we'll get to that in a few minutes. Uh the guard rails are also incredibly p uh important uh according to this particular survey. But when you look at the implementation it's really patchy uh guard rails for AI use in in research. um only again I'm really surprised that this isn't 100% because to me it's like everyone should do this but 80% report implementing human review on AI generated outputs that means that the 20% out there is not even subject to human review um 11% report having no guard rails in place and um only 33% offer training in AI use so within organizations people are not even being trained and how to use AI to conduct their research. All of which for me is really quite quite uh alarming to see. And again here um this is a great quote from that that particular the the ways uh the ways survey in research AI excels at tasks like tagging synthesis and running interviews at scale. It doesn't replace human judgment. It augments our capacity. So it makes makes things quicker. It makes things less reliant on resources. But we really do need to uh to really see the human judgment or or as Denise and I call it that that research thinking needs to be present and and be articulated and in place uh while you're using it. And Denise, go ahead with this one. Yeah. So >> do this one. >> This is a recap. >> Yes. Yes. So u more people are doing research without necessary training than ever before. AI makes it faster, easier with fewer fewer resources and more people are using it than ever before. Uh research is being more and more valued than ever before. Um unfortunately, most AI and when they're using it is an impenetrable black box. When you enga when you engage uh a bot or when you engage AI, um you're not sure. You can't really, you know, open it up and see exactly how they're performing. a lot of the activities that they're doing. Um and in some case Denise uh you mentioned once uh uh it's when it's conducting like um IIA moderated interviews and it gives you it doesn't give you a video it just gives you a transcript >> some tool the the I worked with the tool that that we were experimenting with and uh and all it gave us were transcripts and a really verbose hard to understand report. So we had to question was there a study done at all because uh AI can come up with transcripts and a report without a study. So you really have to be care you really have to be careful about which which I AI tools that you use and also we have some and also we propose that that you create your own parameters to make sure that it's not a black box. >> Yes. Um, and with that in mind, human judgment factor and research thinking is really necessary to avoid what Denise and I call research theater, which is just a bunch of chaos and a lot of things that are a lot of findings that are not articulated or based on on solid research. um guard rails and governance should be in place and ensure quality to ensure the quality of the research and everyone should be aware of these and are are operating with the same kind of guard rails and also again um research thinking is is necessary now more than ever before and through the framework that Denise and I are have created and actually we're writing a book about it um if you follow these guidelines you'll be able to um you'll be able to implement your your democratization program and also uh rely on and and implement AI to a much uh a much better extent. So how to apply research thinking to AI is almost exactly how you apply research thinking to humans humans conducting research. So it doesn't matter whether it's AI or humans you you need to make sure that they have the proper context. In the case of AI, it's uh creating a custom GPT your project with all of the business goals of the project study materials including past research so there's as much context as possible engaging and collaborating with with your custom GPT to uh to make sure that they understand what those existing research research findings are, what the business goals are, discussing the project goals, making sure and making sure that the context that you gave that really is has been absorbed and is going to be the basis for all of their dec all of AI's decision-m and that that applies to people too, right? You will if the if you're working with people, you want to make sure yes, we we're going to share it. We're going to make sure that uh that the that you're training people and making sure that they observed what you've trained them with, right? And I can't emphasize this more as a researcher. Anybody doing research needs templates. We need consistency. It also creates efficiency. But in the case of AI and people, if you if people have templates to to work from, that's a form of training in and of itself, right? And and it it takes a lot of the guesswork out of it. and may and I I just can't uh I can't emphasize enough how much templates are important to the entire process, no matter who's doing it. Instructions and guidelines, right? Just like you would have a playbook for people, you need to have clear instructions on what the GP should and shouldn't do and how to follow clear research protocols like what what is your research framework? What how do you define a finding? for example, I'm I'm sure I see in the comments that some of the AI tools allow you to do that, but you need to have those parameters and guidelines because otherwise it produces chaos >> uh or produces uh results that >> unreliable results. >> Yes, results that seem smart but are based on hallucinations and prompts. Experiment until you get to the prompts that work. Keep a library and also verify you don't treat it as a black box. Uh I I do a lot of large scale um surveys now with with uh write in questions and I use AI to help create my code book. Then I go through all of that segmented chunked out data, make sure that that everything's been coded correctly and go from there. But I read everything. I I read everything uh uh so that I can trust that the AI is actually interpreting the the the patterns the way that I would. >> Yeah. Don't be that 20% that is not doing that. >> Yeah. >> Okay. How to get to um Okay. >> We're gonna we're probably not going to get through all of this, but we we'll share this with you so that you have it when we talk to people. I'm going to go through this fast. uh when we talked to people conducting research, we found that the the things to look for or people who are managing people who are conducting research, empathy, curiosity, humility, and those are the those are the innate characteristics of somebody who innately thinks like a researcher. The skills you can learn, you can learn objectivity and rigor and you can and you can learn meta-inking and you can learn perseverance really. But uh but these are the key things to look at look for when you're trying to recruit stakeholders to train and we created this as and then as uh it's taken from uh Tomer Sharon's uh research maturity ma matrix. This is the research democratization maturity. If you're in that that bottom left quadrant, there's there's no doesn't matter what they're what they're doing. They're they're not really doing anything. If but even if there's buy in but no resources are invested in it, that's ad hoc and performative. uh if if from the the lower right quadrant there, if you have the resources and tools, but you don't really have buyin, the research is is it's supported, but it's inconsistent. You're going to sometimes you're going to they're going to your stakeholders are going to be acting on really good researcher research and sometimes bad. But if you if you hit that sweet spot where there's buy and engagement, especially from the bottom all the way to the top and there's investment in research, that's embedded research thinking. I'll give you a quick example from one of the projects I'm on at Google. um working with kind of uh younger PMs and and designers who who do take in the research but don't always act on the research but research the research maturity at Google is such that the executives above us ris will ask did that come from the study like let's make sure that that our decisions and our roadmap are based on the research So, so they might not be democratized but they have but research thinking is and really this is about culture change and accountability and demonstrate demonstrating value with value with evidence. Uh you you can't really change the overall tides, right? But you can you can make some change if like I said before if the idea is to just eliminate research researchers altogether that you're you're you're sis it's a sisophian task but you can still do it but sisophian but but it starts with culture change. You want to make sure that you're using research and demonstrating value in research that changes decision-m from intuition based to evidence-based. And that is the real key to everything you do. Whether you're democratizing research or or conducting research or presenting research, you need you really need to that that's the that's the key culture change. And you and now what you want to do in terms of how to how to start is before you start again you want to see what the motivation is. If it's coming from the very top you're going to do your best but you're also going to put your you're also going to look for another job. Uh and uh cuz you you might be wrong and they might love what you do and you end up being the head of a large research program or they or you might be be uh be looking for a job uh involuntarily very quickly. But you want to evaluate need against investment. You want to make sure there's bottom-up need and top down demand. And I'm going to go through this quick because we're already running behind. >> Yeah. >> Apply the diagnostic we gave you to understand where where you are and what you're dealing with, right? Understanding why you're democratizing research or what why are you being asked to uh to move faster or why are you being asked to uh to have uh people who aren't researchers conduct research? What the one the why? Because the why is going to inform the the how, but so are the who. You need to have partners. Any any research program, you need to have pro partners. You need to have advocates. If it wasn't for that that senior director on the project that I'm on that kept that reminded the the uh PM and designer to follow the research because we did a lot of it. It was reliable. uh they wouldn't the they wouldn't be where they are right now which is which is with uh with a really strong MVP. Uh and here's the framework. I'm going to actually let you show the framework. >> Okay. Yes. Yeah. The research framework there there are four elements to that define research thinking. The what what research really exists and what your options are. Uh Denise and I were both surprised that we did a huge amount of research about what actually tools are out there and we didn't find a common repository for all the research activities, the types of research that you can do. So um Denise and I actually assembled a list of like over three dozen different research activities that we weren't able to find anywhere else that you can really identify okay this these are the tools that or processes that we can use to actually do the research. The how also was extremely difficult for us to to look at to find the the thinking standards of how it's done. I think I hunted through like 50 different websites uh to talk about research methodology and what that was all about. So really um having that compiled in one place and saying okay this is how we're going to conduct research and this is how you do research effectively and what to avoid. um that's also going to be very very important for you when you develop your framework. The who is also important understanding who's actually conducting research. I was surprised that and the companies that I've been with that this is not listed anywhere that there isn't like a company directory of like which skills each person has and what they could do and the research projects that they've been engaged in the past. Um and also the hierarchy of who's reporting to who. it's going to be important to have a network or at least a a database of who's conducting research when they're doing it um and what their capabilities are. And then the the other one is the when the actual when are triggers what what triggers research. What do we have to look out for going forward? What are the different things in the business climate that could trigger research? If you combine all four of these elements, these four cornerstones, that goes into research thinking. And that's something that um Denise and I think should be a part of every organization when they're thinking about research. >> And I see a couple of questions already. Uh I do have a couple of examples of organizations that do this. Well, Capital One has been doing it for quite some time, but they also hired within the past couple years. they hired Jen Cardello to to strengthen it. Researchers at my understanding uh is uh is that researchers are considered more like teachers at Capital One. They still have a very robust research program and a and a strong research team, but but researchers are researcher coaches. And I know that a that um ADP is has a really strong program. It's new, but but there's there is and it addresses bottomup need. And there's there's support from the middle and there's support from the top. Unfortunately, that's also the example where PM uh used AI in their repository to just to validate what they wanted to do. Anyway, so they're still they're working on that. It's not perfect, but it's really strong. >> Yeah, I think this points out that not all research is created equal, right? Um it is very much a what's the level of rigor, what's the level of maturity within the organization, not just on a research standpoint, but also uh throughout the different business units and orgs um and how they kind of uh interact with each other. Um and that same aspect of democrization of research, there's democrization of design, right? these new tools like Figma create, things like that. Uh so what we're feeling as researchers, whether it be market research, UX, CX, etc. Uh our counterparts are also feeling um in terms of the the squeezing of do more with less, right? Uh Denise, you talked about um you know the the aspect of uh people moving seats, right? Uh at Lowe's Corporate, that's a big thing right now where you know they started as uh content strategists and we all know that that's something that you know AI is taking over. So those content strategists moved into a product design role. Instead of a an actual design, it's more product designer, right? Um and then the uh another example um for for Pam uh Walmart uh Walmart design is doing a really good job. Uh they've created a curriculum uh about uh the democrization of research right from the bottom up of you know the the three-step method, right? Do you do Neilson Norman heristics into pure into usability testing and you can jump in and out uh but gives you guys a framework of who what when where why right uh of how to actually do that research as well so establishes I guess it's more establishing that framework is really what you guys want to leave the the you know the people that are watching today with right >> yes and I want to answer sesh and that is yes absolutely uh AI can make anyone feel like a researcher. And if you don't have the if you don't have the skill set to even work with AI, it's going to validate you as a genius. >> Yeah. >> And it's going and it's going to produce it's going to produce gobbledegook that sounds really smart, but it's going to lead you in in uh it's going to be based on your own biases and whatever whatever the large language model thinks you're going to love most. So yes, this these quick and dirties are disguised as efficiency, but it doesn't have to be that way. It doesn't. That's what I I I I'm a seasoned researcher and I'm I am I've been I've been working on AI development for some years now and dog fooding AI tools and it's uh the one thing that I haven't seen enough of is is tools that allow you to create true guard rails, true a true framework of what is a finding. Oh, well, like uh you have to I I I'm a little bit behind now, but I remember testing some some vendors and they they're just like, "Oh, look, it'll write you an executive summary." Based on what, though? >> Yeah. >> Oh, so bas >> Yeah. And just to kind of follow up a little bit about that, I've been using it. I've I enjoy using AI in my research. I I have a wonderful time with all my different entities. Um, I found that for me it's all about context. You have to start out and supply you set up your projects and include as much context as absolutely possible and then engage your AI in a dialogue so that you both are understanding what the weaknesses are, what what you're interested in, what the what the business goals are are all about, so that you're engage in a dialogue before you do the first studies and help use AI to analyze the the the data because that's really going to be the changer. And again, that's the human judgment that has to be involved and the the the research thinking that has to be involved right off at the top. >> And it's the same principle as training people versus AI. Do you want AI to be a peer or do you want AI to be an intern or do you want AI to be a an associate level? Right? But when you when you train people based on their on where they are, they might still be at an intern level. You you don't want them to feel like an intern because they're probably an executive, but you want to give you want to train them to to train them up with but you have you still have to understand this is their this is their skill level. Uh some some people are born at the associate level. So it's it's it's just trying to understand what what are the different I'd say um personas for democratizing research that you can also apply to AI. >> Uh appreciate your time both of you. Um appreciate all the the thought uh the work that you put into this and um obviously your experience. I know your guys emails are on the uh the screen there. I'm sure you have no issue with uh those that have watched today reaching out to you asking some follow-up questions. I know in the chat there was uh a question about the framework. Um feel free to pass that over our way and we can distribute that. We have a list of everyone that did attend so we can follow up and and send out that framework to everyone that attended today. Um but really uh really appreciate the time, effort, energy and and you guys uh speaking about the things that you are extremely passionate about. So uh appreciate your your time today. >> Great. Thank you. and do feel free to reach out to us because it's a two-way street. Uh unfortunately because of the the the nature of uh a virtual conference uh we didn't interact as as much but please feel free to reach out because we we want to help you but we also want to learn from you. >> Awesome. Um we will uh pause there. Uh we'll we'll give you guys a short break uh to to you know do whatever you have to do. uh we'll pick back up around one uh for the next talk uh which is uh another amazing talk um from not a research expert but a a research um adjacent kind of discipline um so Luke Williams will be leading that from Accelerant uh but Julia Schwan will be the the presenter for that. So uh I'll let you guys take a short break and we'll be back at uh the top of the hour. >> Thanks so much. >> Thanks so much.