Internal and External Listening in the Age of AI

Well, welcome everyone. Um, so my name is Molly Hagen. I am a uh research lead and strategy manager here at Accelerant. Uh, so this is kind of the continuation of our um conference today to kind of set some ground rules. Um, you know, we're at the mercy of tech like right now. It wants me to not be in full screen. Um so definitely if you have any issues feel free to like leave and rejoin uh whether you are a speaker or a visitor. Um you know and always be respectful. Um you know our speakers today are here voluntarily to share their knowledge. So we want to create a welcoming environment for them. Um and also as well you know use this as a great opportunity to just network, engage and interact. We all know that this is a small community. Um, and we run into each other all over the place. So, um, without further ado, um, I will introduce our next two speakers that we have. We have Laura and Ben, both from indeed.com. Um, little bit different roles. Laura is the head of Indeed insiders, which is that customer community, whereas Ben is a senior UX researcher. Do y'all need to share your screen or >> Yes, we do. >> Okay. So, I will stop sharing and let y'all take over. Um, I am in the background and will be available to like field any questions as they come in. Um, but I will turn it over to you. >> Great. >> Share my screen here. Awesome. Can everybody see my screen? Yes. No. >> Yes. >> Okay, cool. Thank you. Can't see you all. You're on another tab. So, thanks everybody for for us today for this little talk on internal and external listening in the age of AI. as we were introduced. Um, I'm Ben. I work for Indeed. I head up sort of more of the internal listening side, so some of that HR side. Laura, on the other hand, works on more of the external side. So you'll get to hear both of our experiences and how how our use of AI has has changed sort of how we work, but then also some of the best practices that we're gonna that we're going to talk about, we're going to show you and and some practical lessons that we've learned just just having pitted to everything AI in the last, you know, year or so. I like starting out with this quote. Um I'm not afraid of AI. I'm afraid of people who think they know AI. And so what this means is that individual like folks that that think that AI is an end- all beall can replace smart judgment can replace smart people in seats that could automate everything and everything and anything under the sun is is those are the folks that that I'm more afraid of versus AI itself. And we'll kind of get into some of the best practices and governances and and sort of why I think that So, what are we trying to solve when we're thinking about using AI, especially in either an internal or an external listening sphere? Is it is it surveys? Not not really. Like, we're pretty good at designing surveys. We we know external customers. The issue of we do with it after we have a we have focus groups, we have days of of even you know big product use or anything like that. But how do we sort of string all those insights together? What we do with that feedback afterwards is is arguably obviously much more important than than collecting the feedback itself. So if it's not collecting data then then what is it? What are we trying to solve for? It's capacity. Honestly we have thousands of comments year over year. We have limited analyst time. I've you know I I person I don't uh work 24 hours a day, seven days a week. I like to do other things as well. So limited time. We have pressure to move fast. leaders, product teams, developers, they want insights and gosh dharn it, they want those insights yesterday. So we are we are trying to move very very fast and honestly sometimes we come up with insights that sometimes sit in a deck and they have zero action. So we we might get a little nugget somewhere and and we think it's important we put it in a deck but you know what do we do with that signal? Uh was there enough N for that signal is something that we always get too. um we know that that that obviously there is a signal and we just don't always have the bandwidth to use it and so this is where AI has changed at least in our work some of the ways in in which we listening obviously it doesn't replace it it itself it scales it which is really really nice and removes a lot of the bottleneck um previously when we did open-ended feedback in in surveys you know we can conduct like a simple sentiment analysis or do you know an LDA which is a technique where you get word frequencies and you have to sort of art it into actual themes but trying to do that across 3,000 comments that that's that's a lot of lift. Text analysis was always something that's very specialized, very technical, and and sometimes can can take a long time. But with AI and LLMs, we can analyze 3,000 comments in in days instead of weeks. Theme detection at scale is is what we're trying to get out here. We have obviously survey listening channels, but then o open listening channels. So think about use of Slack, think about use of email, think about use of Zoom. We have different and and alternative forms of feedback that now we could take advantage of whereas before we just we just didn't have the time unless we hired you know dedicated headcount. It's faster iteration. We can figure out what's working well in an org, what isn't working well, what's working well for a product, what's not working well for a product. Those those those things come now at us fast and they come at us at scale. And then obviously we can turn that around and conduct faster reporting. So as I said earlier and I think everybody knows everybody's under a ton of pressure. We want insights, we want them now. Where's the market going? Well, now we can generate some of those those reportings at scale. Scaling allows us to to like I said move quickly. We can rerun themes easily. Hey, this was an important theme we detected for our org or feedback um product feedback. Is this still a thing? You know, we can test out anecdotally. Uh we can test out anecdotes basically. So, we hear in this part of the or this part of the product they're having they're having issues with X. Well, is that true? We can actually go ahead and look at that. We can look at subgroups without delay. prior subsetting data for specialized parts of the organization or specialized products that took time that took a whole bunch of of skill sets to generate those those insights. Now we can do it pretty quickly. Um we could also test different prompts to create different insights. So part of of using AI and and LLM in particular, it's a little bit of an art. It's a little bit of a science, right? So we have we have prompts that we like to use but how can we refine those prompts? We test different prompts. Do we get different insights out of it? Um we can use LLM to to iterate on stories instead of creating a a generic cut of data and report out on that. So what I was saying for that is that prior we would have just a generalized cut of data that we would share out with with org leaders. But now we can create specialized stories to to to uncover those different pain points in those specific organizations we might have glossed over. So that's going to help us with action planning um from like a product and employ product perspective. that's going to uncover different pain points for different segments of customers that previously you might have might have glossed over. Yeah, you have a high users or small user. Again, that there's a little bit of signal there. You might have glossed over it because again, you're you're small analyst team. You're only a couple of people running those comments takes time. But now with LM's like, oh, hey, this might be actually important that we take a look at this. And obviously this is definitely a byproduct of of having time. Speed is great everybody. You know AI can do this. We can replace we can go fast. Well that's that's amazing. Wig whizbang technology as they say. I I like we use it. But for all their magic can and do. They invent numbers. They sound confident while being very very wrong. They collapse some of that nuance. They drift across runs. They converge in weird ways. They surface PII that that isn't supposed to be surfaced. Um, so let's break this down. What do we mean? LLMs can invent numbers. So if you're asking LLM to analyze data and say, oh, hey, are the differences in scores between this, you know, time one, time two statistically significant or are they different? LLMs can come back and say, oh, these are statistically different scores. when in fact they may or not may or may not be significant scores. You have to double check some of their work. AIS I think we've all sort of seen this AI sound very confident while being very very wrong. Um you'll you'll prompt AI and it'll say that's a fantastic idea. Let's do it. And is that is that a good idea? Maybe maybe not. Um, it's just sort of personally I think one of my I like to use LLMs to do recipe generation because I I do love to cook and sometimes LLM's um will will say like, "Oh, hey, if if you're working on this this poultry glaze, yeah, you could throw it in the oven at 500 degrees." Like, well, no, you can't. It's going to burn the glaze and turn it very very bitter and kind of gross. like this is not this is this is this is incorrect and LLMs will come back and say hey you know what you're absolutely right so LLMs are very confident or being very very wrong the best way to think about them is a very for those of of of us have a very very confident three-year-old a very very smart three-year-old but they're still kind of three right um they can collapse nuance so that sort of dubtales into the convergence in odd ways when when we used LLM before and that's speaking from practical experience. We sometimes see that the way in which an agent is built in which an agent is run can start to converge into a couple of topics. And so you'll feed it maybe you know 1500 comments or something and it and it'll generate a list of insights. Okay, great. you'll feed it another separate set of comments and it generates the same amount of comments and the same sort of themes and you know for a fact that that can't be correct. So the agent in the background is actually converging in in weird ways where they're honing in on a couple of comments. They're using that to base all of the actioning off of um and and that's what's being generated. So that's a it's a big risk for an org. they can drift across runs. Um you can prompt an LLM one way it'll run it'll run and and generate insights for X Y and Z for instance. Uh speaking high level generalities obviously and you'll prompt it again and rerun it again and now you're starting to see some slippage. Some themes that that it had indicated were very important before now are starting to become a little less important and other stuff is starting to surface. And so that's something that as as folks that use it, as practitioners continue to use it, um something that definitely work and obviously sometimes PII potential like leaders names, manage names, um customer names that could sometimes be surfaced. Oh, you know, John Doe was using product X here and that's a well, you know, is that is that something that you really want to surface to to leadership and and for especially for us that work in HR? Obviously not. You do not want to surface those that PI. So what do you do? So one of the things that we found in term of in terms of like overall risk mitigation is using psychometric guardrails, right? So, some of the best practices that we do is if we're doing statistical testing, especially on survey data, we just go ahead and conduct that outside of the LLM. Uh, just because LLM's can analyze data doesn't mean that they should analyze data. They can hallucinate numbers, they can come back with with um wrong results, but having skilled analysts who know what that is less likely to happen. To that end, statistical models, they obviously don't invent significance. If you're if you're running a t test, well, yeah, I mean, that's that's the t test. The the R is going to generate a number for you versus an LLM, which might hallucinate that that t test, which will generate your significant scores. Um, when we do additional analysis, favorability scores are from validated analysis. So we go back through, we calculate all of our favorability scores. We make sure that whatever functions we we wrote to kind of do this at scale are correct. Um, you know, you just hand crank those out to double check everything before you pass that to the LLM. And then themes that are tested across runs and prompts. So we one of the best things to do is to take your prompt and go ahead and pressure test the crap out of it. uh uh throw it different different themes. Throw it little curve balls. Make sure it's still returning the same thing. Adjust your temperature. We'll talk about that a little bit that it's consistently delivered and sort of a good uh a rule to think about. And so it'll detect the patterns in your data. It might give you some some actions to to think about when you're using your data, but the final conclusions should rest with the analysts who are who are in it every single day. Uh the other thing to do risk mitigation wise is have some governance guard rails on it. Right? I think that um there is a bias for some leadership and some individuals to to think that because we kicked something over to a machine learning algorithm or we kicked something over to an LLM, it goes into this black box where logic happens and it spits out a result and aha that that is the result. It cannot be wrong. Well, yes, it is it is wrong. So you should feel free to question the model and have policies about its use, right? Report agg insights in aggregate. Again, sometimes LLMs can hone on very specific comments that seem very loud and build themes off that. So what does that mean? Well, it means the tail's wagging the dog, right? Go back and double check some of those comments. Make sure if you're reporting out have safe minimum end sizes. So, you know, if you're going to go run an analysis, make sure that you have enough data to where you could be confident in generating some sort of conclusions off of it. Don't let that small N bias completely take over. One of the things that we do is we mask PI before sending it off to an LLM. So, we make sure that anything, especially from an HR perspective, some of that stuff's redacted and then we send it off uh before theme generation. Otherwise, you could get, you know, it could surface policy violations or stuff you don't necessarily want it to surface. Better yet, if you're able to do it from an orwide perspective or even a personal perspective, just go ahead and get your own API key because that will based off their terms of service that will protect some of that confidentiality in there. And again, if you have analysts, use them. Like they're they're smart people. have them review any sort of the the output from LLM and and review it before leadership sees it. Yeah, LLM will generate insights, but sometimes it doesn't make sense or it needs a little bit more um of proddding for for why the LM generated that or the the language is off or the numbers are off. just just get that just get that extra set of eyeballs on it before you send it off to leadership instead of just saying aha the machine generated it can it can be wrong some of the best practices you know how do we monitor the model uh we know that hallucination isn't hypothetical like I think we've we've seen it like I said it hallucinated that I can throw a a glaze with with sugar in a 500 degree oven and and burn a burn a heard like that's you know that that was a hallucination very confident hallucination but one nevertheless what are some ways to mitigate this well go ahead and check that interrun consistency so yeah prompt the model run it go ahead and run it again maybe run it a third time make sure some of those themes that are spinning out are pretty consistent themes way to do it is to also vary the temperature and look for drift and model output so the temperature controls how much the model is allowed to to kind I think you do like a temperature like a zero is more deterministic but even a temperature zero deterministic should be reproducing the same results every single time doesn't necessarily reproduce the same results every time you could surface new themes new sentiment analys new new sentiments new new ratings where like before this comment was a positive rating now this comment's a negative rating obviously that could happen with a human interviewer too or a human analyst where you you look at a set of data like I'm pretty sure it's negative. You come back to it like well in the scheme of things maybe less so. But um for LN especially that to to be constantly flipping all the time. Some flippers are okay but if it's consistently kind of going hair wire then then obviously you don't trust the output. If you're conducting thematic analysis one thing that we found is to go ahead and generate supporting quotes. So if it said, "Oh, this is an important theme for product XX or an ORX." Okay. Yeah. Well, show me your work. You know, don't don't let the blackbox and sort of take over your thinking about it too. Make it show you its work. Make it, you know, the product again. Hey, how confident are you that this is an actual theme? How confident are you? These are the actual sentiments. It'll it'll give you something and you'll kind of figure out that maybe it was confident at one point, maybe it isn't, you know, and and should we trust that insight? Maybe, maybe not. And some of that confidence is sort of a diagnostic tool, right? So, like I said, sometimes they sound very confident, but you should do your job too to ensure that you're confident in whatever that output is. Um, at the end of the day, the output should always pass that snip test. From a practical perspective, what is that what does skilled use of AI look like in the future? Is it just prompt engineering or is it a or is it requiring sort of a different line of thinking? And from what we've seen and sort of how we're using AI, it's it's less about some of that like hardbased skills code. It's it's more of a shift thinking from like input output to systems. So how do these systems talk to each other? Skilled practitioners should design those conrains prompts. Don't Yeah, the LM needs needs a prompt to reason over and it'll reason over your data, but you have to prompt it within reason. Again, it is kind of like a three-year-old, a very very smart three-year-old, but a three-year-old nevertheless. So, constrain it a little bit. Don't let it wander off into La La Land. Have it do the things you want it to do. And don't be afraid of asking it to do the things you want it to do. Separate some of the stats from the summaries. Make sure that your stats are grounded in real data that if a leader goes back to a dashboard and handcal some of those differences like differences in the AI, what differences that the AI said are very important is actually backed up from the live data coming in and document some of those model drift based on prompts. start building a little prompt bank like, "Hey, this prompt worked pretty well, but we caught we started to catch drift. Now we're going to switch over and figure out and test these prompts and instead don't just, you know, waddle up and and do prompt of the day, you know, consistently use your prompts and consistently interrogate them. Definitely challenge that output." Um, again, LLM like you don't necessarily work for it, it works for you. So, so definitely challenge some of that output and then orchestrate how AI exists into existing workflows. So, for us especially, we've been leveraging sort of automation and then AI on top of it to to deliver decks and insights at scale. So, how do we get those systems to talk to each other? How do we go from using a stats package and and functions NR to N8 to you know OpenAI's LLM back to Google Sheets back to the deck to create all the way through delivery. How do we how do we integrate systems to deliver everything at scale but then make sure that our integrity is is good to make sure that we did the things that we are supposed to be doing. little different than standard copying and pasting. Does AI replace listening professionals? Absolutely not. I know a lot of a lot of talk out there is on, hey, we could just kick it over to AI. It's going to replace all this all these workers. It's going to replace all these analysts. It I I don't think it will, nor do I think it should. Um, AI helps reduce some of the mechanical coding, but your analysts still have to know what's going on. AI sometimes lacks some of that contextual analysis. So, for instance, anecdotally, we've built agents that that work with some of our our systems data and we have specific guidelines in in place and we try to sometimes make the agents follow those specific guidelines. But what we found out is that you can feed it all the context you want and at the end of the day it'll still hallucinate incorrect answers and say like, "Oh yeah, that sounds like a great idea even though it violates a couple of company policy." Well, that's not very good, right? The live individual definitely still has that context, still has the judgment, still has some of the legal awareness insights on the org and how to best action plan. So, so thinking about uh like you know the people that work for your company, especially from an HR perspective, you know the people that work for your company. Okay, great. You should definitely leverage some of those insight like you like first name B. It it thinks it might but and so you bet where where the human comes in. AI is really good at augmentation. So in terms of listening, what's the real opportunity for listening team? Um we've seen from some data we've ran and employees are pretty optimistic about using AI at work, right? We can if you're a listening on a listening team either internal or external, this goes for for both. Um you can increase that capacity, increase responsiveness, shorten those action times. you can protect some of that rigor but then also at least for the HR perspective protect that employee trust right um at the end of the day we work for our people and so we always want to be sure that whatever it is we're generating has the rigor behind it has the statistics behind it has the quality of insights behind it um is defensible legally is defensible from just somebody casually passing by and taking a look. So we definitely want to protect our employees trust and so we have to build safeguards in in terms of using AI with LMS kind of you know just letting go. So what does it all mean? Kind of quickly summing up some of the points we've been talking about. AI is definitely is definitely going to be part of employee listening. Um it's not a question of whether or not we use it, but how are we going to use it? So again, it's that thoughtfulness of use. Um there are some data suggesting that you know employees are kind of weary of them the myriad of AI tools. So how are you going to use AI? which AI tools are you going to use and then how you're going to use it is extremely important. Um I think we've all seen the HBR article that said that you know a lot of initial ROI on using AI and LLM at work wasn't wasn't all that great because everybody thought well let's kick it to the kick it to the model it'll pro results and we'll print money. The teams that actually succeeded were the ones who used it targeted for very very specific things. So again using AI not as an analyst for using it to augment some of their work removing that bottleneck uh helping them do things at scale is the most efficient and best use of AI. So how are you going to use it right? uh if we use it like I said casually or like I just said a replacement for those skills we we do run a huge amount of risk right um you know if if you're using AI to do talent mapping or if you're using AI to to you know god forbid reduce headcount then that is going to run a huge amount of risk and so um the computer doesn't necessarily know the legality of the situation and it thinks like oh this is a really great idea when it really isn't. And so it's never going to be a replacement for those skills of a human analyst. Like I said though, but if we do use it skillfully, we can expand some of the capacity and improve the overall employee experience. Yeah. Now we can target different parts of the organization that that may have gone unnoticed. We could surface real pain points. We can um act on those pain points faster. We can be a little bit more confident in our insight. But is that is that a result of just the LLM use? No, it's a result of of us skillfully using the tools that we have to improve the overall employee experience. Um and and kind of to that end, if employees were thoughtful enough to leave some feedback, no matter how small, we should be taking advantage of it, right? You could have previously conducting that sort of, you know, LDA where you're just looking at like number frequency of words. It's it's it's not going to surface themes that could be vitally important for vital areas of your organization, small areas that that you need to surface some of those pain points. We conduct we can conduct more of those targeted analysis um before where now whereas before we couldn't necessarily do any of that. Thank you so much. >> I'm gonna give you tomorrow. >> Yeah. Thank you so much, Ben. Thank you all really for being here and taking time out of your busy lives to to spend time with us and learn. Before we move on to external listening, I want to give you all a moment to ask questions of Ben. And the reason being is that so much of what Ben had mentioned principally applies to external listening. But he and I have very different customer sets. So for him it's the employee, for us it's our customers. Um people use our products, invest in our products, buy our products. And although much of the principles that we share are the same and processes, um, I want to give you all a chance to ask Ben questions because I work on the qualit qualitative side of the house. I do a lot of focus groups. I run a community of customers whom I engage with daily, weekly. I run um cabs uh quarterly with very senior clients. So, it's a lot of face-toface interaction. Ben is very technical as you all probably witnessed with R coding um building agents himself. So before we move on to my section, please please send some questions. We want this to be engaging and interactive. So if you can even come off mute, let's have a chat before we move on to the next section. And then I want to make sure that Jason's question gets addressed, Ben, too. >> Yes. So just calling the attention Jason asked um are you speaking broadly of all LLMs or are there specific platforms? >> Yeah. So in terms of just LLN usage overall Jason like uh basically that's that's broad LLM usage. So if you're thinking like claude or or open AI chatbt that's that's sort of indicative of the stuff that we've seen. >> Any other questions for from anyone else or comments? >> Um I had a question um actually. So Ben, you were talking about kind of rerunning things to to get that validation or kind of quality control. Um, do you typically do that within the same tool or in what cases might it be useful to to use different tools and kind of crossch checkck each other? >> Yeah, sort of depends on your use case. So if I'm looking at validation within a specific tool, then I'll rerun it again within like let's say I'm using Open AI's tool, then I'll rerun it again with Open AIS to make sure like my prompt is stable and it's generating the the results that I wanted to generate. If like I said it go starts to go really really haywire then you can integrate other tools and run prompts and other tools and sort of develop that like I said like hey this prompt works really well this um that's what that's what we've done and been pretty successful at. >> Great. Jason's following up with another question. Um by the way I don't think they can unmute. So, um, are there tools that can detect if AI was used to create a response or or analysis? >> That's a really good question. Um, if I'm not entirely sure, we haven't really looked into that. Um, we you do know like having worked at BLMs and not just anecdotal like from my own experience. So anecdotally obviously um if an output is from LLM it it's not quite right. So you even though you're generating insights at scale it does behoove you to go back through and maybe reword some of those insights it's generating to make sure it actually makes grammatic sense and it makes contextual sense. Otherwise it does it does seem like an AI generated it. Yeah. So if it doesn't if it seems like a little off, it's probably an AI generation. >> Thanks for sharing. Laura, do you want to go ahead and go? And then as more questions come in, um, I'll kind of filter if y'all don't see them come in. >> Yeah, definitely. And I' I'd love to have some interaction, too. Um, you know, I think external and internal listening, it's just different customers, same same principles. Um, if you don't mind moving forward, Ben, thank you. Um, we are we are at such a pivotal moment in our in our lives, in our department. Things are changing so much. It's it's hard for me to even capture this on a slide. And I'm I'm assuming that many of you on this call feel very much the same. I think we are at a time of exploration, a time of the beginning phases. I think to be honest I think our lives will look really different in terms of being listening professionals in a year, 2 years, 5 years. I think 5 years we're going to have be having a different conversation today. I wish we could all get back together and compare notes. So we are at a stormy but exciting period I would say in the external listening world and and particularly in my department. Um I do find one of the biggest positive outcomes has been yes speed I don't want to belabor this point we've talked about it over and over um in this conference and with Ben but the outcome has of speed has created actually more consolidation internally uh many tech firms where Ben and I work and consult to externally these are very complicated organizations that grew so fast then decreased headcount it's constant chaos to create revenue. We are in complicated places. Okay. And often with that does come siloed departments in many ways. So what we have found is a beauty of this is insights have created actually more partnerships. Um there's always traditionally been um a tension between sales and product and what they're developing and designing with what they're selling. And we have found actually internally at Indeed that because we can get insights faster, consolidate them, that product is listening more and cares more. We have found really a breakdown of walls between traditional BI departments and market research departments in marketing and completely different teams are partnering, want to consolidate, want to build a story and one story. So that's really a huge positive element that we've seen here and in other organizations. We have also found that it's not just an analyst being an analyst and creating the most beautiful survey, the most sound survey, statistically significant survey, right? We are moving beyond legacy surveys. They do happen. They are radically important. These metrics are key things and KPIs that are measured uh internally. But we instead of just saying presentation to senior leadership saying here's what we found in the MPS survey or here are the seesat results of the month, we are actually knitting together bigger and greater stories. And what do I mean by that? So internally at indeed and in other organizations we have found and have consulted to Ben and I have said yes we have one story but what about the unstructured data? There's so many other different listening channels that are being brought together into a big presentation to drive strategy and to create insights. At Indeed, what we've done is we have scraped our social platforms, Reddit, Tik Tok, LinkedIn, to look at unstructured market signals and data. What are what are our customers saying um unprompted about our products, our AI solutions? What do they think about our brand? And we look at glass door, trust pilot, all of these different touch points, which are just so complicated. We used to analyze each one of these listening channels independently and it took so much time. Now we're able to really figure out what are the themes, what are the saturated themes we're finding across our mega surveys, across these unstructured listening channels too to create more powerful stories. So that's been really exciting. We've also found that um internally as well as in other organizations that um again tech tools have been very siloed. So only sales and CS um in particular care about gong, right? Um and what the recordings of of um and what I mean by gong is um tools that record conversation with live clients between sales and CS. Um, I know all of you probably use different types of of tools internally for this, but typically that data only went to sales and CS. It was pretty complicated and again it was siloed. So what we're finding now is that we are investing in tools that sit on top of multiple tools and analyze them. So we are looking at interactions of customers and sales and CS reps via Slack, via Gong and um they are spitting out insights and stories um into like very very simple simple ways that we can then share very quickly with product teams with sales and CS and with others internally. So again, it's kind of a march away from this is my tool, these are my insights and they stay here and we're we're really breaking barriers in in that way too. Um we are using AI also again it took so long for our analysts and BI and voice of customer teams to look at sentiment analysis. um looking at our customer accounts with vigor and detail to see who's going to turn, what's going to happen, what are accounts we need to target. This kind of proactivity of strategic analysis just can happen so much quicker. So, we're able to identify specific behaviors like risk to churn, um account health, um go to market strategies, how how did it go, what could have been improved. Um, we're asking questions like what are new customers? What are the personas we need to pursue? What are passive customers in the market? Who do we target? When do we need to target them? How do we target them in specific ways with our go to market strategies? Um, we are able to answer these questions so much more rapidly. It's incredible. Instead of an annual cycle of figuring out one customer segment, we're able to do that continually, rapidly, quarterly for us at this point. I mean, it probably could be even faster in the future, right? So, we are just feeling excited, overwhelmed. We feel that um we can be more proactive, more strategic in in really more powerful and and faster ways. Do you mind going to the next slide? Yes. With this excitement and new ideas, uh comes comes some speculation too. Um, you know, I I talk to our clients weekly and often daily and about their trepidations about AI and everything that they tell me is we need to keep the human in the loop and that is exactly the stance that we take internally in my role specifically as well as um to other companies. We are really the harbors of the voice of the customer. I feel very passionate that um it's actually becoming even more important for external listening professionals to make sure that the integrity, the accuracy, the um nonbiased voice of the customer is is really at the heart and center of everything that we design. So the human just cannot be removed. Um, AI technology, as Ben had said over and over, it just cannot see the nuance of of human. We are just complicated creatures. The nuance of emotion, the nuance of tone, the nuance of everything that we do and live and breathe. So, it's just not there yet. It's very hard to replace this. Um, we have many departments and companies experimenting with uh tools that moderate through an AI bot. So, outset AI, discount, um, user testing, sorry, dcout, um, etc. There's so many different tools on the market now. We're experimenting internally with some of these to supplement our live sessions. So, what we're seeing is a lot of hybrid experiences. Yes, we may experiment with outset um if we need more of a general population viewpoint, but we're also supplementing with live sessions, focus groups, um cab quarterly with our VPs and plus who need more handholding and feeling more special. So everything with AI output is supplemented with the real person. That is what we have found to be the most accurate. We have found we feel more confident when we feel s when we find saturated themes in live discussions as well as AI moderated discussions and and that's what I think the future will continue holding. We are in the age of experimentation. I we internally and to other companies that Ben and I have spoken to people feel totally overwhelmed. I don't know if you feel this way but there are just so many options and with this is a positive as I had mentioned it's breaking down the barriers with just siloed tooling internally which is amazing but it's also there is so much more rigor around what vendors do we use who do we contract with what do we recommend Ben and I to other companies it's it's just so much um I also feel that my way of working and with all the professionals whom I speak with we have to use multiple tools these days. It's not just onestop shop. So for example um I was asked to very quickly turn around um an analysis where I had to look at 15 different sources of data um a lot of like very complicated research pockets from UX from market research and to put all of that together into like a 10 slide PowerPoint. So what did I do? It was a lot of pressure. I looked at Gemini. I looked at Claude. I looked at Dovetail. I did I used at least five tools for me to get to the output. And again, the output was very general. It was still the technology is still not there. I had to have a very firm perspective first before I trusted the output of what AI was was giving me. How many if you all can write in the web chat around how many tools do you use? I use on average five. What what do what do you all use? if you want to put in the chat. And are there three? Okay, awesome. Four. Love it. Okay, two. Great. One. Four. Yeah. Okay. So, we have a lot of threes and fours. My point is it's never one. Are there any tools that you all really recommend and use for your analysis that you can share with everybody? We have an enterprise Gemini account. We also use Glean internally. We use Dovetail. I use Dovetail a lot with my videos. So, I upload focus group videos to dotail and tag them. Chat GBT. Awesome. No. Yep. Notebook. LM. I've used that too, Molly. Thank you. Yes, I agree. It's better than Dovetail. Agreed with you on that. I agree. I find them all better and they're same. CL. Yep. Notebook. LM. Great. Yep. Perplexity. When I create um when I ask uh when I ask uh Claude and Gemini and ChatgPT, I've done an experiment where I asked them to create a moderator guide for me for a focus group. And every single one of them was slightly different and had different pros and cons that I kind of cobbled together to create my own script. So, I agree. Thank you all so much for sharing. I really appreciate it. Another discussion that is really hot in the market right now that we've explored internally and with other organizations is the use of synthetic audiences. So these are essentially fake audiences that are created um using LLM. They act as a proxy to mimic real people. So it's pretty extraordinary. It's it is amazing technology. I think it's still growing and and and getting more sophisticated. Um we I think there there's a lot of risk to this, right? If you're only using synthetic populations to get outcomes, I I think there's got to be some rigor around this. Um I I don't fully trust the outcomes yet. Um we at indeed have experimented with again supplementing synthetic audiences output with real survey data with real qualitative information. So again everything that we're doing is very much a hybrid approach. It's never just solely AI um and we're trusting the output there. Um we are again my my last point is that we can't rely at you know in voice of customer we often have to give information up to leadership and our CEO to make million-doll bets and we do not feel comfortable doing that just using AI output everything has to be a hybrid experience we are still with the approach of using multiple listening listening channels um to create a story. What's amazing is that those multiple listening channels are just getting faster and easier to consolidate. So again, we're using information from the market research department, from the brand team, from BI team, from sales. Everything is is about consolidation and bringing all of these listening channels into one story, one strategic perspective to drive revenue, to drive strategic initiatives here, to drive product development. And everything right now for us is about ease of digestion as we're breaking barriers barriers with all of these teams. So, our BI team is very much focused on creating a consolidated dashboard. We've done it. It just needs to get better where all of these different data sources plug in and they create a perspective and an opinion that a leader or product development manager, engineer, salesperson can look at and see, okay, I get it. This is what our customers are thinking about X product, about X dynamic in the marketplace. Um, so really the future for us is creating an amazing consolidated dashboard and we can't really do it without the help of AI on the back end. Next slide, Ben. Thank you. So I pulled this quote and I think it goes really to what Ben had said. We are at as listening professionals. This is everything about hybrid transformation. We are not fully toward trusting everything with the AI. We are the shepherds and we guide our wonderful friend to help us get to the outcome that we need. But um really for all of our roles, it's the destination. We are going to have to use AI. Um we use it today. It's it's a little bit of chaos. I think it will only get better. We'll figure out our ways of working and and what's accurate versus not through testing, hybrid approaches. um hybrid research methodologies. So, let's stay excited. We have to stay flexible and creative and we do have to stay analytical. Let's let's not trust the bot. We are we are really the leaders um into creating um the best of the best outcomes and insights. Any questions for for Ben and I? I have a question while people might be typing theirs. So, we kind of talked about synthetic audiences. I'm wondering what your thoughts on what are your thoughts on using AI as a way to like test a discussion guide or a survey instrument? >> Um, is that something you see like valid? >> I do. I do absolutely as much as we can test the better, right? Um at at each stage. So definitely we've we've done that internally as well. Um absolutely do it. But again, test it, see what works, see what doesn't. But again, our approach here and in other companies is we got to get live data too. live survey results, live focus groups, live people to supplement everything. >> Definitely. And Ben kind of touched on this as well, right? We are and you we're we're in this really transformative time where we're seeing lots of hiring in the past few years and now a lot of layoffs. I guess what would you say to people entering their careers? um you know what how do we make AI not this big scary thing that's taking our jobs and disrupting the market versus like an actual useful tool. Yeah. I I actually talk a lot about this with um I run a community of customers and I think that actually the rise of live customer engagement. So communities even like really cabs as we call it um like really senior level engagement so VPs of companies and and CEOs of companies and higher is actually really rapidly expanding the popularity I and I think it's in reaction to AI because the more live discussions we can have um the more powerful. So I I feel like customer communities internally, externally are just growing in in in popularity and we talk about this topic a lot about Gen Z. What what are they being asked in schools? What's the expectation when they come into the workforce? I think there's a bit of tension because in in an academically rigorous environment where integrity and not cheating and not taking shortcuts is is a priority is is difficult because they're coming out of those environments where it's discouraged often to be used. I think in some curriculum it's now um like use it any any yes there are curriculums where they're encouraging usage of both but but at the end of the day the output has to be your own has to be your genuine voice. So we have found that genenzers um have to do a lot of work on the side outside of the academic environment to really upskill to understand the different platforms what their pros and cons are um to really land that entry level role. So please do the work on the side, go to workshops, talk to your peers. I don't know if it's gonna if you're going to be able to practice it like writing a paper. >> Um >> I don't think so. But but you can incorporate it in in other ways in your daily life or or just test yourself on the side because it is a requirement. That is a requirement. They will be asking you um out of college do you do you even know even just knowing about the industry in general feeling comfortable talking about big concepts >> and how it fits into your role. Um there's actually an interesting uh example I have. So, we had a customer just hire a person to um who needs to learn who needs to know how to create bots >> um internally to create to create more efficiency in systems. >> And so, this person knew enough coming out of college. They weren't complete experts, but they knew enough to talk about it intellectually and to actually create those bots. So, make sure you know how to how to create things. Um, you don't need the expectation is that you you don't know you don't need to know it perfectly because the technology is so new. Talk about it intellectually. Do reading and experiment on the side. Um, because those types of roles will be coming up in the entry level world about creating bots, creating efficiencies, things like that. Ben, would you say the same? Yeah definitely. I think having a good knowledge of of how to use it is pretty important in terms of of upskilling. I think yeah, being okay with embracing some of that ambiguity on like is this cheating? Is it not cheating? Like you're you're going to be expected to use it, but then also I think embracing it in weird ways like yeah, this is a stubborn little assistant, but it's still like my little assistant I think is going to be very very important. We have a question that came in. I just we're running out of time so I want to make sure we get it answered. So Emily was asking you, Laura, uh you mentioned synthetic audiences. Can you share any trusted tools or vendors that provide that type of service? >> Yeah. >> So we I do you know Qualrix? So Qualrix is definitely um a hu massive survey vendor and they are definitely experimenting and working on this furiously. What I like about Qualrix is that they have diversity of customers so that they will have diversity of synthetic audiences which is great. There are other vendors who very much focus on particular sectors of the world. Um look at Empathy Lab, look at Lacmoose AI as well. Those are other vendors that we've looked into. Um, I mean, there other vendors that I've looked into that just focus on media, that just focus on logistics and trucking. Um, so pick your poison. But I think that Qualrix has so many different range of clients that they cater to and speak with, so they might be a really good first bet to talk with. >> Great. Well, thank you so much both of you for for enlightening us all today. Um, definitely feel free everyone else to follow up with them on LinkedIn or other channels. Uh, we are going to kind of prep for our next speaker. Uh, but we are still around. Thank you guys again. >> Thank you. Take care. Bye. Bye. >> Thank you.