Customer Experience Management Systems: Do's, Don'ts, and the Power of Open-Ended Feedback

and we're at the top of the hour and since you john you're batting cleanup not going to go through the whole spiel of you know rules and regulations because everybody's behaving themselves yeah i don't anticipate us having any issues tech's been our friends so we're going to knock on wood that you know that nothing goes wrong in this last one um but otherwise you know got a great compelling topic we're going to keep the cx party going so i'm gonna step aside and let you uh let you get it rolling john very good and just chime in if there's any questions that pop up happy to answer those bill thank you absolutely sounds good well thank you bill and uh good afternoon everyone you know this is unacceptable uh when i when i think about the situation at this point i am beyond frustrated i i don't want somebody to contact me i don't want to talk about this i've already decided that absolutely not i'm not going to renew the contract next year wow customer feedback doesn't get much more brutal than that and that was the situation i faced a few years back i was on the phone with the sales vp and the account director that owned that customer relationship and i knew on that call the stakes couldn't be higher this this customer's contract was worth close to 200 million dollars critical for the business i knew if that contract actually was not renewed that account director would lose their position that sales vp would not be able to make uh their quotas for the next year just wouldn't be able to make it up with the other customers in that region and not only that there was the potential spillover effect to the broader customer base this customer was well regarded in their industry seen as a thought leader an innovator and was looked to for best practices and so it was highly highly likely that if this customer went away competition would use it against us so we were on the phone that day strategizing around how can we re-acquire this customer and keep them from leaving so i'm john whitaker really great to be on this call today and and thanks to bill and the accelerant research team for inviting me to talk about a topic i'm always excited about and passionate about and that is customer experience management system uh mark's presentation uh was a great setup for what i'm going to cover today and focus in on and i've had the very good fortune in my career to work on customer experience management systems across a lot of different both businesses to b and b to c systems in a variety of industries and over the years i've benchmarked and talked with practitioners across a variety of businesses and one of the interesting things i've seen are reoccurring things reoccurring themes around how to best extract the best value out of what these systems can produce from an informational perspective i mean you think about it customer experience has never been more important as we come out of the pandemic customers are going to be finding a new way and finding their way to a new normal and trying to figure out what does that new normal looks like so there's going to be a lot of change happening over the next couple of years and so these systems are critical for understanding the changing environment and how our customer is feeling and showing up and how best to meet the needs of this evolving customer so really exciting to share the themes that i've seen over the last number of years on the do's and don'ts around extracting best value out of these systems of course we're familiar with the common components mark went into wonderful detail around the common uh key performance indicators that we're familiar with and there's a lot of different ways to deliver that customer survey and the reason i have both of these elements on one page is how you deliver the survey can greatly greatly influence the results you get for your key performance indicators mark talked about one of the most common ones net promoter score and so how that survey is delivered to your customer can greatly impact that mps result that you see on the backing and bill i'd be curious from accelerant research's perspective how have you seen the interaction between these two elements how the survey is delivered versus what then metrics come out of that survey yeah and it's i don't know it's it's always been a fascinating topic to me i i guess over the years have been beaten down so much that i i've gotten to a point where you know i've just surrendered to the fact that surveys customer experience management is going there's going to be inherent biases right you know depending on data collection method what what have you um and it's always you know it so long as i the research manager can you know keep it at least a common level of bias or equal bias so that i can compare results across waves or across um you know regions or or stakeholders within the company you know admitting that yeah the bias is what it is but again if i'm making at least fair comparisons then i'm i'm okay with it i agree each method comes with its own benefits and drawbacks something i experienced in my career was the following and the data are amassed but i inherited a legacy uh program and the survey was delivered via an invitation on the purchase receipt uh the customer got at point of sale and over time we started to experiment experiment with delivery of the survey via a mobile app and the beauty of the mobile app is it leveraged the the capabilities of the smartphone and the key capability was there's a gps chip in those smartphones and the gps chip knows where people are and so it would know if if the shopper was in the location of interest for us so you could push out after that store visit push out a survey invitation and we saw in this case customer satisfaction we saw a dramatic difference in scores i mean it was a very large difference and it immediately begged the question well gosh why why are we seeing such a dramatic separation in scores the the url on the purchase receipt tended to give really high strong customer satisfaction scores whereas the scores coming out of the mobile survey were they weren't terrible but they weren't great either they weren't anything to celebrate and so the fortunate thing is though having the purchase receipt it could be trapped back to a specific transaction that could be connected to the customer database and looking at the customer database you could see long-term behaviors of these individual respondents and what we found was those responding to the invitation on the purchase receipt tended to be highly highly loyal highly engaged shoppers for this brand conversely the mobile survey tended to show uh feedback from a shopper who was much more representative of the full diversity of shoppers in our database both good and bad in terms of loyalty good and bad customers and so what we found was the mobile survey actually yielded a much more realistic result much more realistic of the full universe of shoppers that we served versus the receipt invitation uh and and the the older method of the shopper having to type in a url address into their browser and i don't know bill uh have you seen similar kinds of things in in your work where you see a dramatic difference in terms of results same survey just different delivery vehicles you know mute there we go uh yes it can be huge um you know and it again delivery vehicle and channel that that someone is coming in based on um and what we find also you know that opens up the whole line of questioning into you know survey design based on you know where they're where they're accessing what device they're using if it's you know a screen like i'm speaking to you on or if it's you know a smaller like mobile you know that interface can change my participant experience i know we're talking about customer experience but you know there's there's additional factors that can come into play there that's a great call out i love that language you use bill participant experience and that's so important we'll talk a little bit more about that here in a second and you also hit on very important survey design one of the things i've seen over the years is if your survey is too long you reach this inflection point where the the respondent becomes so fatigued start to check out in the scores all else being equal the scores start to drop dramatically and what i've seen over the years is moving from right to left that inflection point keeps moving closer and closer to the left side of this graph indicating that you know the customer's attention span and willingness to take a long survey is becoming less and less and less over time i think it's just the effect of social media how we consume information today how it's consumed in very small bite-sized pieces so a big long survey can also work against you and and fully agree bill with your observation that that survey also has to be optimized for how it's delivered what kind of device is it being taken on and that's something you can very easily monitor with your survey software and it's important to know how are your customers completing your survey is it on a laptop is it on a mobile phone what is the device and that can be helpful guidance to you for designing your your survey because of course from an experience perspective we want to at prevent what this cartoon shows we want to deliver an experience that reinforces the brand promise our we want our survey to show our customer that we care and we care about their time and that we care about their feedback and that we're going to take action on their feedback what we don't want to do is have the survey be a point of frustration i like how mark referred to uh car dealerships and how obnoxious they are in in this regard and and really create an experience that leads to customer regret and so making your experience as powerful as possible from a surveying perspective is important i recently i took a survey from american express as a as a cardholder of theirs and i really loved it because up front i was told it would take less than five minutes it only took about two it was very respectful of my time and the questions were very well designed and and clearly designed with me and mine as the customer and making it easy for me to provide feedback and i really enjoyed that experience and it reinforced positive perceptions i had about that brand one of the things to watch out for and this is a big mistake a lot of companies make is tying the results of the customer experience survey to compensation it's good-natured and and well-intended to do that you want your organization to be customer focused you want them to deliver great experiences but what this cartoon illustrates in a humorous manner but it happens every day in the real world is when you start to link nps or customer satisfaction or any other score to compensation it starts to drive and incent bad behaviors let me give you an example a real example i experienced a pre-pandemic so this would have been 2019 i think it was in august later in the month my two teenage sons and i went to buffalo wild wings we always enjoy going there and generally had a good experience at the end of our meal the server came out and with great fanfare i was presented with the chat but told if i completed a customer satisfaction survey right then and there i would get ten dollars off my next visit my sons and i visit with enough frequency where that was compelling so i got out my phone the survey was pretty easy to take knowing how these surveys work and knowing that likely in the manager's office they were watching the survey come in in real time i gave them good scores across the board because i wanted my ten dollars sure enough i i finished my uh survey hardly even put the phone away and the manager comes right running out she's all excited she's beaming she runs up to me and is profusely thanking me for the great scores i had given and not only was i going to get ten dollars off my next visit but she was so happy she gave me 10 off my visit that day it knocked off over 25 of the check crazy so of course you know the next time i visited buffalo wild wings you know what scores they got on that server survey so bribery absolutely works but of course that distorts the system it distorts why you have a customer experience management system in the first place if you're paying your customer to take the survey and you make it very clear that good survey scores result in monetary reward you've totally ruined the purpose of the system there's no way that system is yielding useful data if that behavior is widespread in the system there's no way it's yielding useful information to improve the customer experience and i don't know this to be true but an implication could be let's say you have a customer who comes in and they're in a bad mood and nothing you can do really helps them be satisfied or happy with their meal my guess is maybe maybe they don't get invited to do the survey so linking incentives to survey scores ultimately destroys the usefulness of the system mark talked about this nicely in his previous presentation and customer experience isn't just about the customer the employee is so important in the experience now you might be thinking well my customer my customer is engaging with a website but keep in mind behind the scenes behind that website are company employees who hopefully are continually improving that website experience acting on customer feedback to enhance the information and capabilities of that website and continually improving it so everywhere there's a customer interacting with your brand there's an employee somewhere in that equation so understanding the customer and their perspective of the experience that they've had is really important but so is the employee and the employee is equally important in this relationship and an example from my past many many years ago uh well over 10 years ago i was in the fast food industry i didn't work for mcdonald's but like all fast food restaurants the drive-through is critical obviously it's important during the pandemic but it was equally important 10 20 years ago and so in the customer experience management system i ran now the first cut of the data was looking at the customer experience for dine-in versus drive-through and since 70 percent of the volume at that time went through the drive-through spent a lot of focus and energy on that drive-through experience and our data was coming in continuously it was a great program and all of a sudden we started to see a strange trend we started to see customer satisfaction slowly start to trend down and then we went to the we went to the next level diagnostics to see what areas were were we coming up short in and two things popped immediately one was order accuracy and the second was rudeness perceived rudeness of the store employees and these same complaints were showing up in the open-ended comments that we would get as well and so this was puzzling why did this all of a sudden start those two attributes start to trend upward and in lock steps so clearly there was some association between order accuracy and perceived rudeness and it was strange so i connected with the operations organization and right off the bat we've got feedback very meaningful feedback that a speed of service initiative had started the same time those scores started for those problem areas popped up and the speed of service issue or speed of service initiative was around the drive-through and really the only way to increase your sale if you're a fast food restaurant is to increase the throughput of the drive-through because important day parts like lunch it's a very fixed period of time so the only way to increase sales is to just serve more customers in that fixed period of time so the fact that there was an emphasis on speed serving more customers in the same amount of time made perfect sense that order accuracy had have become a problem area you go faster you're going to make mistakes makes a lot of sense but rudeness why were rudeness complaints going up that was just puzzling and so i realized a mistake i had made was focusing too much on the customer side of that experience and not thinking about the employee side of the experience so we went behind the counter we went on the other side of the window and spent time with the employees it was actually an in-depth ethnographic study and we wanted to understand what was the impact of serving more customers in a fixed amount of time what impact did that have on employees and taking that employee-centered point of view brought it all to life and it unlocked the key to the mystery we were dealing with and what we found was that in serving more customers in a fixed period of time it was creating a lot of stress and strain on the store staff and of course moving more quickly order accuracy mistakes inevitably increased and of course what happened then is the customer would come into the store and say hey i ordered french fries they aren't in my bag can i get my french fries please and what we saw was the crew member was so stressed about filling this order that when they had also at the same time get product for the customer who had an accuracy issue it created a lot of stress and what the customer saw was the employee reacting to the order accuracy issue the employee going oh because they're stressed about getting that order out the window but at the same time now they've got to do something additional that stress that body language that stress brings on was perceived as rudeness even though the employees were not rude and did a great job when not stressed with the customer so it was a great learning for the operations organization and they were able to change the order processing procedures to make order processing more efficient easier to get the order correct the first time and it eased the stress on the staff so when there was the need to engage with the customer the employee was less stressed out and able to provide that friendly feedback and and interaction with the customer so really digging in understanding that employee perspective and experience is critically important in understanding that ecosystem you're measuring and another opportunity to validate the importance of employee feedback i had a large data set many many years of data and i made the simple assumption that well gee you know i'll look at customer satisfaction for example the higher customer satisfaction is the better sales will be in the future makes perfect sense but what i found was zero correlation not able to create a meaningful predictive model of future sales from customer feedback and this was puzzling so i partnered with those in hr that do human capital analytics and and track things like employee tenure uh performance review ratings uh things of that nature and when we incorporated that data the employee side of the experience into the customer data we were able to find a very robust and strong correlation predictive of future sales so it showed to me quantitatively and reinforced my experience in fast food that that voice of the employee is so critically important and so integrating that employee data is important but there's lots of other data sources that can be incredibly uh powerful for you uh obviously social media generates so much data daily there's so many ratings and review sites that that provide information that are important for your system so integrating everything into one space is it can really increases the power of your system and even in a b2b context extracting data out of something like salesforce.com can further add to the power of the data and there's a lot of great dashboarding tools out there uh some uh do a lot of really amazing things but can be kind of expensive as well but even something as simple as tableau can be a great way to get your program uh more accessible the data to your uh key stakeholders and and to do so so uh in a simple way in a very cost effective way and i'm curious bill from accelerant's perspective what kinds of best practices do you see in this area both data integration as well as dashboard yeah it's been amazing in you know i said the past decade the emergence of you know all of these different platforms and the ease with which we can now sort of weave them together um i can look you know i may five years ago when you know if you wanted to do a lot of this data integration dashboarding you have to create something custom you get you know developers involved and just ridiculous amounts of time effort and ultimately cost to the point where it would often become cost prohibitive and never get off the ground um you know i feel like we're moving into a space that's much more equipped to be able to to handle a lot more of these in more of an off-the-shelf manner right well thank you i agree the technology's advanced so much just even last few years and it's amazing let's keep moving along uh this is somebody uh if you don't know who it is you should they uh fred reicheld invented net promoter score uh it was it was uh first published uh back in 2003 and uh in the harvard business review and it was well received at the time and you fast forward today it's amazing how many fortune 500 companies use net promoter and and believe in it as as a key metric and i'm always surprised that across a lot of different industries you'll even hear companies their ceos talk about their mps scores on earnings calls uh for example so it's interesting to see how what fred created has blossomed a lot across a lot of organizations uh to be honest um you know as a one number system uh i'm i'm i'm not a believer in solely focused on net promoter i like a more an index approach but i certainly have incorporated net promoter in in many surveys i've managed bill i'm curious what what's your perspective or uh experience been with with net promoter and and your clients um yeah i mean it is wrinkles and all um such a ubiquitous number and i think because it is just easy to comprehend easy for executives and non-researchers to grab it and run with it um you know we have an accelerant we have our own loyalty model that we built a lot of companies do we're very proud of ours from a you know a strength standpoint and predict predictability of in our case share wallet um but every loyalty model you know that i come across includes likely to recommend you know advocacy as you know a key component so there's no question that mps is on to something it's just you know it is a portion of the equation um but you can't argue with the fact that it is just so widely used and accepted and you know we just sort of have to get on board with it because it's here to stay that's for sure you're right it's so widely used and uh so many of us use it in our surveys so i thought it was interesting to look at what does fred think about his creation fast forwarding today because as the creator if anybody's got a credible opinion it would be fred and so what do i appreciate is he he mirrors what we've talked about today thus far he's talked about that absolutely not do not uh link it to bonuses financial incentives the like you know don't focus in on that that single metric but rather it's the open ends and the richness around open ends that matters and in the example i opened with that customer giving really fairly bad open-ended feedback you know their closed-ended scores including net promoter were not as bad as what their open-ended comments revealed in other words mps they were a detractor but in terms of the scale usage they were pretty close to the point of indifference and so too often what i've seen in customer experience management systems is the scores don't always reveal the truth about the customer's experience and aren't nearly as actionable as the open-ended comments that you get and experienced practitioners of mps mirror this across a number of different industries i think a useful and interesting example on this page is usaa typically insurance companies stink on net promoter and and get lower scores than many other categories yet usaa kills it on that promoter uh because they deliver an outstanding experience but i think in part they deliver an outstanding experience because they recognize it's the open-ended comments where the magic is and that's what's actionable and that's what can ultimately lead to a virtuous continuous cycle of improving the customer experience and i'm kind of curious bill what what have you found in terms of the balance of you know you get that quantitative metric back but then you've got all this really rich and wonderful open-ended feedback as well i always give the analogy of quantitative versus qualitative research so you can beat internal executives over the head with charts and graphs you know about what's happening and what you're seeing but you know sitting in a back room with with an exec and the light bulb that goes off in their head when they hear these words come out of the customer's mouth um you know yeah i've been telling you this forever but that guy just said it and it is just so incredibly powerful and that's you know that's the nuance that you get from it from these open minds i agree and it's like going back to my example the call to action was the open-ended comment i'm not going to renew your contract you know that that inspired action if we looked at the mps score likelihood to recommend and saw we got a five it would have been like it would have been nice if that was higher but there wouldn't have been much of a call to action now of course getting through those open-ended comments that's always the challenge isn't it so one of the important things i found is with real-time dashboarding being able to process those comments as they come in as they are provided can be really powerful especially if you're in something like restaurants or hospitality or retail where you've got a lot of different units out there you know getting those open-ended comments in real time to the the unit level leaders greatly increases the actionability of those open ends and and makes it much more easy to internalize and process those open ends because then you're not getting hit with a new tsunami of open-ended feedback if you look at just a quarterly or even monthly up and it's great then many times if it's coming in real time service issues can be addressed and service wins can be celebrated to reinforce a great experience you know something i've done many times is it's easy with python code or using something really easy like alteryx to set up exception reporting that flags open-ended comments that should be reviewed by a human being and and setting up multiple different rules that that provide that that visibility so you can see is there something that needs following up with with respect to that customer or not and that's fairly easy to do and put into place and then adjust over time and of course there's a lot of talk you go to a lot of conferences there's a discussion of machine learning and specifically natural language processing and i think there's a lot of power in that potential in it i think it'll slowly but surely get better over time the downside is you need really really big data sets and the other thing is it's going to be tough for still many many years for machine learning to be good as good as human beings so going back to my example you know that open-ended comment that was so harsh ultimately the folks that were best able to interpret that open-ended comment were the vp of sales and the account director who knew that customer and knew that respondent very very well so they had that perspective of context and history of relationship and what was going on with that customer and so no machine learning systems going to be able to impart that kind of wisdom and judgment to an open-ended comment i'm kind of curious bill are many of your clients using natural language processing for their openings yes and with varying degrees of success you know i i think i'd agree and just echo the sentiment of there's a lot of potential but we're not there yet and i think when you accept that and sort of weave in you know the human element with the machine element that's where the magic happens at present right absolutely you know something i found in the past of course this is dependent on your customer segment and how they like to take the survey but if you're with a consumer that tends to be responding via a mobile device capturing their open-ended feedback via audio can be incredibly powerful you tend to get much more detailed feedback it's a much better experience obviously for the respondent because it's easy for them to talk and they're used to talking into their phones and and the software is getting very very good to do the voice the text translation so think about your customer segment and how they're completing your survey and that might be a good alternative for you to test and try out now in b2b it can be less important and if you have certain uh customer segment where that doesn't work out then obviously that's not a good option but something worth trying if you've got that kind of customer and they're using their mobile device a lot so let's go back to my story so a very brutal situation 200 million dollar contract that's pretty close to going away obviously bad things will happen if that occurs but the open-ended comment went on that that customer provided detailed feedback why they were unhappy and specific points that were incredibly useful for us to create a re-acquisition strategy the points given uh mirrored some feedback other customers were providing action had already been taken and investments being made to address those customer experience pain points and so a plan was formulated uh starting off with a quarterly business review that was coming up and in that quarterly business review uh talking about with that customer uh the investments being made to improve the customer experience and hard data showing that system-wide the experience was slowly but surely improving and using that as a starting point to rebuild trust and build up a track record of improving the customers experience improving it a way they could readily recognize and their staff recognized and over time that occurred and a year later when it came time to renew that contract trusted trust had been rebuilt the experience was dramatically better for that customer compared to what it had been and that contract was renewed and so full credit to that ultimately that open-ended comment being actioned against by the selling team that owned that customer relationship and they were ultimately able to harvest that relationship but the key point was the magic was in the open-ended comment not in the closed-ended feedback that we got from that customer and with many customers that was the case it was the open-ended feedback that was so powerful and important and so what i've learned across a lot of customer experience management systems companies industries b to b b to c is you know we've talked a lot about the don'ts and those don't do's are reinforced by the inventor of the net promoter score and so from a best practices perspective i think the most important thing is not linking financial incentives to the score because that will only just result in bad behaviors and and things that will ultimately bias the data and make your system less than useful for improving the customer's experience but absolutely so much power in the open ends and so as much as you can automated reporting of those closed-ended comments to free up team time to focus in on the open ends can be very powerful and leads to some really great insights as we talked about integrating other data sources particularly the voice of the employee and so for example if you're working on a b2b business uh getting the voice of your sales force and those front line reps interacting with the customer can be incredibly powerful and helpful for your system and make sure your customer sees that you're acting on their feedback and that they can see in the experience that it's slowly but surely improving because then they're going to know that when a survey is executed and they see that survey invitation something will be done about it so i really enjoyed our time today hopefully you've learned something and it was and hopefully complementary to what mark shared previously but open to any uh questions anybody might have or comments and any questions for john please pose them now um i have a couple um so if i john am a member of you know a small corporate insights team three person um you know i have this customer experience tracking instrument on one hand but i'm also juggling product development research brand tracking and you know copy and ad testing um with that how do i keep from getting just completely bombarded and buried by the open-ended responses yeah that's a great question bill i i think something that's simple and easy to do is is one automate as much as you can the closed-ended stuff so and the way uh you can do that fairly easily uh what i've done in the past is you get your data set coming in from uh either your survey provider or the survey platform you're using uh then process that data through alteryx and then have that closed-ended quantitative data then flow right into a tableau dashboard so you just run that uh alteryx routine and then in real time that then that dashboard gets updated but in terms of your question bill about the open ends that same alteryx routine you can put in there the exception reporting and what i typically done is in alteryx then a pdf report gets generated and provides the open-ended comments along with the respondent id so then you can look at fairly quickly uh the open-ended comments and and process those fairly efficiently and if you do that with some kind of regular frequency during the week certainly it depends on how much volume you you get back from your system that can make the processing of open ends much much easier so a good example on what kind of rule you might set up in alteryx something i've done in the past is there were certain key metrics that were important in terms of indicating happiness or unhappiness goodness or badness with the customer and then using those metrics to then flag open-ended comments that might need to be reviewed i think you may have a little bit to this um in your presentation but you know when those flags exist what's your opinion on granting access to [Music] i guess internal stakeholders even down to like you know if we're talking retail store manager level right oh that's very important oh that's critically important because like when i was in restaurants so well over 10 years ago but when i was in restaurants each store uh manager had access to their scores only their scores and that was important and having that real-time access was important because if there was you know for example some customer service misses you know that could quickly be addressed with coaching of that crew member and many times it's just a training gap because for example in restaurants you have so much turnover you always have people on staff who don't know everything they need to know so that real time access is important or on a b2b system or having real-time access for your business owners is important and even getting that to the sales leadership that own those customer accounts is important because ultimately they can't the sales people can't hit their sales quotas and goals if they don't know how their existing customers are performing and feeling and so that's critically important and so i i would highly encourage the democratization of data within an organization and flowing it out to those individuals who can action against it i agree um i always find that to be a a difficult ask um you know as insights professionals right you're the the stewards of the data right we you have to go through me in order to get to this information um you know opening up and sharing can be a scary undertaking at times it can but what i find is bill when when you cascade the information out to that lowest level of of customer interaction it allows for the actioning against those tactical needs and concerns and then that frees the team up to think about what are those two or three macro strategic things that that need to be addressed so for example going back to my order accuracy and rudeness example you know that allowed that because the tactical data and the actionability of that was pushed out to the restaurant store owners and managers we could then put focused resources against that employee employee ethnography i talked about and spend time really decoupling that employee experience so that it could then influence that program so it should be freeing for the team and allowing the team to do much more added value work high roi work for the organization what about and i mean i i have lots of questions what what's your take something on i guess comparisons within a a hierarchy right so you've got different stores or or restaurants um i've i've always been interested in and taken by the sort of the conundrum that exists that in many cases your highest traffic units tend to also be much lower performers when it comes to you know the sat and loyalty metrics right um you know they're much busier so if you're comparing you know high traffic to low traffic stores it looks as though the low traffic are really killing it from a sat loyalty standpoint how do you reconcile that type of thing oh you're right you're exactly right bill it's a u-shaped curve so what i mean is as store volume or customer volume goes up uh you'll you'll see a positive relationship for a while but then it goes back down because let's say it's a grocery store i've not worked in grocery but if if the line is 10 deep for checkout because it's such a high volume store of course as a shopper you're not going to be happy with that and i've even seen the same effect in b2b uh like like a b2b selling to like for example a 100 billion dollar organization they're pretty hard to please and it's easy to have hiccups because they've got so many uh corporate sites that you're trying to serve so you see the exact same kind of thing in b2b as well as retail restaurants hotels and so on and so you you've got to that's part of that data integration piece and it's important to integrate in that volumetric data like revenues for example and margin and those other kinds of business metrics knowing that then oh wow this is a 80 million dollar a year store they're killing it they're busting at the same so of course their scores are lower and of course there's things you can do statistically as well to recalibrate based on that once you've established there's that kind of relationship awesome um let me just do a final check any ques any last questions for john before we wrap up no you're the end of the day so absorbing quite a bit well thank you for the opportunity bill and thank you for those of you who uh were online since 11 a.m that that's quite the run that it was really fun to be with all of you today and really enjoyed the opportunity thank you bill thank you to all of accelerant research for the opportunity it was a lot of fun this is great john thank you um great closer um and and really appreciate it and appreciate everyone attending uh be on the lookout for info for next month's event and everybody stay safe out there all right thank you everyone thanks bill bye now