AI-Driven Member Sentiment and Experience Insights: How Experian Consumer Services Elevated Understanding of Member Feedback

Cool. Um, so we are at the bottom of the hour and it's about that time for our next presenter who is Pamela Calman and she is uh with Experian Consumer Services. She's going to be talking to us about AIdriven member sentiment and experience insights. How they've elevated uh understanding of member feedback. Um obviously AI hot button issue and we are all sort of figuring it out as we go. So to the extent that we can get any kind of case studies or or sharing of examples of how this stuff is beginning to be used in real life, it's fantastic. So we're all very interested. So I'm going to shut up. I'm going to let Pamela do her thing. Um I will stop sharing. With any luck, you'll be able to. Uh but if there are any issues, I'll I'll step in and let you know if anything's going on there, Pamela. >> Okay, I'm going to share my screen and start my presentation. Can everybody see my presentation? Sure can. >> Okay, let me go back. Let me go to the beginning. I'm sorry. So, I'm Pamela and I um am working in Experian Consumer Services and I'm a senior research analyst with RBOC group and I have been with Experian nine years and we've gone from having two surveys to now multiple surveys that go across all our different touch points. Um, we have our app, our website, we have our products and our call center and we've been building up our VOCC over these years. And what I wanted to be discussing is where we've come from and where we are evolving and where we hope to go. Um we know from looking at all our data, we have tons and tons of data that um when we're looking at the impact on retention that understanding sentiment drives growth and builds strong brand loyalty. Feedback drives improvement. Member insights refine products to most customer needs. And proactive design shift from reactive fixes to proactive experience enhancement. And what we discovered is a few years ago we were looking at our member feedback and we saw categories like customer service, app, website, product, everything had a bucket. But buckets aren't clarity. So if we have 200 members that mention app, we don't we didn't know like are they confused? Are they frustrated with it? Are they happy with it? Is it the onboarding that's maybe a problem or navigation or some kind of feature gap? We had structure but we didn't have always have meaning and we didn't see a connection across the surveys. We were knowing that our members were telling us a story but it was just scattered. And today uh what we've seen with um involving AI is that this allows us to connect those dots across surveys, across channels, across the journey and truly understand how members feel and this is how we've evolved. So we started out with a manual text categorization. Uh we have been working with a platform called Qualrix and Qualrix in the very beginning when we were using it I was actually manually coding everything. I was going through all our comments from our call center survey from our member survey and manually categorizing them into these topics and subtopics and that was very very very time consuming. Then Qualrix added something called text IQ and it isn't really AI but it was a lot better because it allowed us to create topics and categories um kind of based on the traditional rule-based analysis focusing on the verbatim text and that made it a lot better but it still wasn't where we fully wanted to go. So we then were exploring working with some AIdriven sentiment analysis um modern methods that would help us analyze the sentiment across multiple surveys for deeper understanding. And we then are using another tool which is called they do which is going to help us with the journey. So it's helping us know how our members feel about different points within their customer journey and we gives us an understanding of what it means for action actionable decision-m. So the first phase with the text IQ this is when we move to a structured categorization and the capabilities for this we had custom categories and subcategories survey based comment tagging standardized taxonomy for consistency and faster reporting than manual review. But when we worked with this over time, it is pretty siloed. We have now multiple surveys and we were trying to use this structured topic subtopic type of way to analyze our member comments, but it was pretty siloed and we couldn't really get a full understanding of what's going on. If we look at all of our surveys together. So the strengths as I mentioned before is you had standardized taxonomy which it improved consistency, faster reporting, increases efficiency and it was better than the manual coding that I was doing previously. But the limitations are that you have a dependence on predefined categories, limited contextual understanding, and we really realized that we were struggling with nuanced or emerging themes. So we explored some options into our phase two. What type of AI powered topic code frame analysis could we bring in to our ecosystem to help us really understand what's going on with our members from all touch points and all aspects of how they're communicating with us and using our products. So we went using Qualrix has a partner which is called XM Discover. And for any of you that might remember there was a product called Clarebridge years ago and they have turned into this AI powered tool that you're doing a topic structure but it's AI trained on a well-defined framework which is a lot more accurate. Um what this is better is that we have a multi- survey analysis. So it analyzes comments across surveys for comprehensive insights. And the advantage of this is that it's a consistent taxonomy. So it will apply uniform categorization at scale effectively. So throughout this AI journey that we're using with our two tools, these are the key insights for app navigation. We're able to see the usability and the digital workflow friction. This is much more apparent that we are going across all the different surveys. product clarity. We have a better understanding um and expectations from how our people, our members are using our products. Service speed, it's helping us with response time and efficiency, support quality. We have some real gaps here, but going across all these surveys, we can find um where the issues are. we can come up with better resolution and effectiveness journey stages. So from when our members come into our ecosystem from onboarding to account management, we don't really have a linear type of engagement. So our members come in, they're using our products, they may stop using them, they want to come back, they have issues. We can now know along these different stages where there's friction and how we can improve this. And the best feature of using these two tools is that we can see the sentiment by channel. So cross channel emotional insights which is so valuable for us. So when we're talking about our strategic b business outcomes um when we were investigating and we are still kind of honing this tool um we're working really um heavily on this to improve how we're connecting all the different surveys together. And what we are able to see is issue differentiation. So we can separate the systemic problems from isolated cases. What we were realizing when we were looking at survey by survey but not having a cross look at all our surveys together is that we would be focusing on some things that we thought were problems but they might have been isolated cases and we weren't really able to dig down into what are some systemic problems. And that was the issue with the way that we were looking at our categories and our subcategories is those were great and we could see we're having an issue with this product, but we still had to go through and read those comments to kind of understand more fully what was going on. And by using that kind of methodology, a lot of times we were not really able to see something that was really systemic because we would be looking at one survey for example. Now, putting all these different surveys together and even putting chat experience, social media experience together, we're able to get a much bigger picture of what's going on with our members, how they're using our products, how they're getting help that they need. The accelerated reporting aspect is by automating all of these together with the emotions. um we can create very quickly executive ready summarized um a a view of what's going on and we can do a snapshot. Is this a digital thing that we want to focus on? Is this a customer service? Is this product related? And is it a usability issue? Enhanced member insight. So the AI is helping us reveal emotional drivers and patterns. And because we are tracking quantitative metrics, we can look at the emotion with our comments and that helps us really hone in on what is really affecting a part of a journey and how we can improve that journey feature. And it's real time trend detection. So this is constantly feeding into both of our AI tools and we are constantly seeing what's emerging or what's going on and it's helping us drive a faster deeper insight for strategic decision- making. So basically um we'll have a better understanding of what's driving emotions. you know, where in the customer journey are we having the biggest trouble or issues or gaps? And as I said before, we can distinguish between something that is a few issues or something that is an ongoing systemic problem so that we can fix this and we can address it for our members. Um and there's a lot of reduced manual re review time. So by having this coding structure set up, it is constantly enhanced and improving as we get more and more member feedback across all the surveys. And as I said before, we can put together something very quickly and have it ready for executive review if we need to talk about something. and our executives want to know what's going on in a different area of how our members are feeling, we can put together that quite quickly. And we have databacked roadmap decisions. As I mentioned, we have quantitative information that's pulling in and we can tie that back together with our open-ended verbatims so that we are able to really really quickly find out what is going on and how we can fine-tune a lot of areas that we're working on. And one of the key takeaways that I'd like to go over is our evolution. So, as I mentioned before, we started out kind of really in three phases. We started out originally where we were manually coding. So, I was looking at two surveys and going through thousands and thousands of comments and tagging and manually doing that. And then eventually we were utilizing a text IQ tool as part of Qualrix and it was a lot better because you know you had to still invest time setting up topics and subtopics but it still fell short because we weren't able to really tie all the surveys together with one common look at the whole picture. So what's going across? what's some common threads that are going from all these different surveys together with a uniform look. So the advanced um AI intelligence as I mentioned earlier is allowing us to take a cross survey journey insight and we can enhance visibility and strategic value and for a strategic optimization we can we're transitioning from descriptive reports to experience optimization through action. Now, what I wanted to kind of focus on too is that this is something that we've had to really invest quite a bit of time. I know the data really, really well. And so, just implementing AI, it makes it a lot faster and improves, but I also really have to keep going back matching up what I know to be true with the raw data. And when we're laying when we're overlaying AI, it's helping us speed up the process of looking at what's going on, we get real time knowledge of emotions and how our members are feeling about different parts of the journey where we're falling short. And this is really important because we're really using a lot of information from our app more and more. um members and this is true of a lot of different companies are using apps. So probably 61% plus of our members are accessing our you know website or our app through they're using our Experian app. So more people are using their phones and the app over just logging into the website. And we know this is so important and why the introduction for AI for our digital look at how our members are experiencing using our app is really really important and prior to introducing our new tool which is discover and our other tool that we're using is called they do and they do is a journey mapping type of experience and that also is taking just free form open-ended comments and helping us map what our quantitative with our qualitative. And we can see along these journeys when we're looking across all these surveys some common themes that are appearing that are really where we were not really aware. We weren't able to do such a deep dive. We have a lot more clarity here and it's really shown us how much better and quicker and more effective we can be on all aspects of how we're looking at our data. And it's an evolution. I'm continuously working on it. The coding structure that we've set up is really strong, but it's something that we're constantly revising as we see emerging trends that weren't present when we first started this journey. And um this is something that I'm so excited about it. It's really made a huge improvement in the way that we're able to share out across our UX teams um and UAT. It It's just made a huge difference and I'm excited to see where we're going to end up going and how we're going to be able to take this further. So, I finished a little earlier than I thought. Does anyone um have any questions that you want to kind of go over or talk about any aspects? >> It looks like we do have a couple of questions coming through and as you mentioned we have a few minutes to go over those. Uh let's see. It looks like Kristen has uh the emotion aspect of your approach is very interesting. Are you uncovering the emotion in the open unstructured data or a quantitative metric? Not sure if I missed this. Thank you. Yeah, so with the XM discover tool, we have um a section where it is rating on emotion. So we have dashboards. So this is allows you to create these real-time dashboards and it zooms in on the level of emotion. So on a particular topic uh it will hone in and you'll see the level of emotion whether it's negative or positive. And what's exciting about this is we can tie it back to our quantitative metrics. So we know how our members feel about the ease of use. For example, you know, when you go into an app and you are trying to access your information, um, we're experienced, so we have information on your credit. We have information on what's going on with your whole history. And if someone comes in and they're not able to use it or they're not able to find products, the level of intensity of how unhappy they are helps us tie back to why they're giving those scores. and we are able to better know across that journey. For example, um we're tying together our call center information, we're tying together our app information and our member information. and that commonality of that emotion across all those different surveys. And also, we have a a chatbot experience that allows us to really hone in and help our UX teams improve that particular area where people are really unhappy. And what it has allowed us to do is are they unhappy because they don't understand? Are we making it not clear how to use the product? Is it a technical problem? Are they really truly having problems navigating through the app? And all of this feedback is allowing us to make improvements as we go. And this is real- time data. We are uploading daily. All this information and we can see emerging trends and we can dive into it and make appro improvements. Did that answer your question? I hope. [laughter] Thank you. Um there was one comment or request earlier in the presentation and that was if you could uh list out or maybe shoot me an email uh with uh the names of the tools uh that you mentioned early on in the session. Um or if you even can provide a a leave behind PDF of this deck. >> Yeah. Yeah. I I absolutely can. I I can you know kind of give you kind of an outline of Qualrix itself and you know they have different types of platforms. Now the current platform that we had been using with Qualrix and we are still using it is we have full access to um doing dashboards. So we we can create these really amazing dashboards. Um we do a lot of API. So we have our app which is tied back to Qualrix, our website. We have um intercepts on our website which allow us to capture all this information. Um, and that's where we ended up kind of moving into the discover tool is that that took all that information that we are collecting from all those different touch points and this allowed us to have a more robust look at what our members are saying and that we have a better understanding of what's really going on. And I can totally send that to you, Bill. >> That'd be perfect. Um, and we'll get that uh distributed or available for for for download. Um, you know, personally I I I thought this was a really good talk. It um, you know, I like your phrasiage of, you know, talking about this as an evolution or a journey because I mean that's that's what it is. This AI thing is it's emerging quickly and lots of different applications are are coming along at light speed and you know, we got to just start sort of jumping in and starting to use and figuring out. Um, and it's talks like this that kind of help to at least illustrate, you know, how these things are are being applied. So much appreciated. And we're not done with the Q&A, so get ready. Actually, Sesh had a question and it was uh >> just in regards to the uh you know, some of the conversation around emotions. Um, how good is are the tools that you're using in uh picking up emotions like sarcasm or or nuance such feelings? >> That's a great uh question. So when we were using text IQ that also had sentiment, but I don't think I could get above 75% accuracy. It was it's really hard. So prior to working at um Experian, I worked at Toyota and a lot of that comment was driven around product and product was a lot easier to capture because they're really talking about my car isn't working well, things like that. When you get when I got to Experian, a lot of it is going across multiple sentences and you get false positive like you're talking like I love this but I hated that. So you get like kind of this mixed sentiment and the text IQ was really tr troublesome for me. I'd have to go in and try to adjust it and read through things and it was just very time consuming. I would say right now and we're still evolving and still, you know, doing our proof of concept to prove out that we're getting this accuracy. I would say it's more like 85 to 90% accuracy. I'll put it in between, let's say 88%. And it's really I think as the tool goes across more and more, as it's getting more and more comments, it's getting more intuitive and we're seeing a better accuracy of that. I would not say it's 100% at all, but I think it's definitely much more accurate than what we were looking at before. >> Um, what about the next steps on this this evolution? Um, is there anything you're you're excited about or kind of looking forward to unveiling or unrolling in in next chapters? I I think that you know what we're really hoping to see is real time lift in you know members wanting to come back and you know retention improving brand uh I think the adoption of you know people more and more people wanting to use our products and wanting to stay using our products and I think what we're hopeful is that by really getting a much more intuitive and better understanding of what's going on with how our members are using our products that we'll see we'll see some huge changes by being able to leverage this. >> Uh and it looks like Todd has a question over on YouTube. Um and this may be you know for yourself as well as the audience in general but uh how do you prevent emotionally intelligent genai bots from polluting research data sets? >> We have they have uh in the qual itself we can tell by the on the respondent level where it was a suspected bot you know in the way that we're collecting the information. So when we're doing surveys there I mean it's only as effective as it can be. I I think absolute prevention of bots even when we're doing surveys working with other providers where we're getting feedback. We we try to do the best we can to eliminate bots and they you know tell us that they're trying to eliminate bots. I don't think there's a 100% way to eliminate all bots, but we're doing our best to turn on features to try to reduce that. >> Yeah. And it's the cat and mouse game. You know, as as our analyses and and speed to data are getting easier and easier through AI, so too is, you know, the ability to be malicious and and, you know, cheat your way through through surveys. So, you know, we get the fun of not just the analytical side of it, but also the policing of of, you know, data quality, which is, you know, this is a challenge and that's why we have these conversations. >> Well, it one thing I wanted to add is that when we're setting up, so we do surveys, but we also, you know, get information from social media and different sources. So, a lot of the surveys that we're reviewing, Qualrix will kind of give a snapshot of how well the survey is structured and things to look out for. and it will catch like if it sees a high instance of people that are speeding through that kind of will flag someone that may something that may be a bot. So to answer your question, we fine-tune it. We try to uh like put things in place to try to filter those things out. We do our best. Um and then also we know that we're pulling in um information that's applied to members. So, if we see something that's blank and doesn't have any any um attributes to that person that's answering that survey, that's a good sign that that's a bot. Yeah. You know, moral moral of the story is yes, advancements in technology are helping tremendously, but you know, never never take it at face value. you still need a human being, an intelligent research researcher who can kind of decipher, you know, the data quality, make sure that this stuff is is uh is accurate, is correct, is is, you know, passing the sniff test as it were. >> Yeah. And I I I think what I you know my most important thing that you know I wanted to say is that um always be mindful that you know it is something that is used to enhance what you had already. So um be mindful of your data. Be be really well informed about what your data is saying on a very raw level aspect and use this tool as something to help and enhance you. Um so I do spot checking periodically. I go through, I make sure that these numbers that I'm pulling through, this data that I'm pulling through still aligns with what we know to be true from, you know, our actual raw data. This tool is just helping us be better about how we look at our data, help us make better informed decisions and better real time information. >> And there's your mic drop. Um, that was that was great, Pam. Thank you very much. uh I got to hand over to our next presenter. But yeah, fantastic. I appreciate your time and appreciate the good questions. Keep them coming. >> Thank you so much. Thank you so much.