Calculating the ROI of UX: A Four-Step Framework

looks like we're getting some people come in both on the uh Zoom as well as the uh uh the Youtube so you know we can we can get started um if you want to you want to jump in so uh a little bit of a rundown for you guys um obviously we have our January 16th uh virtual conference here a mini conference we're doing a lot more of these standup conferences uh you know we're going to have a a pretty packed February March uh as we we go into the year um with a couple of these and you know uh some ground rules to start um you know obviously we're at the mercy of tech so you know some things might be a little bit delayed uh you know be respectful um you know this is kind of for the networking the engagement the interaction uh you know something that Trevor and I talked about as we were standing this up is you know putting good back out into the industry um and that's a big portion of you know why we wanted to produce this today is uh you know there's materials like this don't exist uh so you know being able to to uh put you know things that are relevant that might help people you know take a little bit of a one or two tidbits from this this conference and and be able to to go back and actually utilize it in their their day-to-day so uh on that note we have a couple webinars coming up uh the next one will be on February 11th uh with Bill mcdal president and CEO uh you know timely with with Super Bowl happening um he'll be you know talking about the creative testing for ad Effectiveness for the Super Bowl Edition uh he's a big Sports guy so I know this is something that you know hits near and dear to his heart uh the next is another accelerant conference with one of our own um our director of Client Solutions uh Julie she's gonna be doing a second part series back in December she did a uh panel Integrity um the next part is going to be you know beyond battling the Bots winning the war on data quality so uh part of the the two-part series that she's doing and that's on February 20th uh we have a couple more that are scheduled for February and then some that are in March uh but another one uh full day conference that we're going to have in March uh our our conferences uh that we traditionally have is on March 27th um it's going to be you know a full schedule a full day we are starting to to fill that out and I think you know Trevor uh and I were just discussing that he might be uh shown up to to one of those as well so um you know he might be another speaker for a whole other topic uh for for the March one but um and then where can you find these resources so we do record all these uh we do you know store them so if you go to our website and go to the resources we'll have uh an area where you can go to not just articles uh so you know I don't want to steal his Thunder for today but uh Trevor actually produced an article to go along with this we're going to be having it there but also all the conferences from today from the future uh and past you can go to our website and you'll be able to to see um but without further Ado uh you know I want to give a a welcomed intro an actual intro to Trevor uh a little a little bit about him right Trevor he's a principal ux researcher and author of the popular weekly substack blog ux research in the wild uh it focuses on doing highquality ux work in the real world uh Trevor comes with 18 years of experience in ux has led ux for you know companies that are startups midsize companies and multi-billion dollar Tech conglomerates uh in his spare time uh he helped build the next generation of what ux curricula is going to be for uh higher education institutions uh and lives in state College PA which is home to the Penn State uh University uh and then has a you know during his free time likes to spend time with his spouse and and two kids but um Trevor you know thanks for for joining us today and I'll I'll kind of turn it over and you I know we wanted to make this a little bit more informal and have some lecture but really a conversation between you and I so uh balls in your court and and I'll let you kind of share your screen and walk us through it all right thanks Brett yeah sounds great yeah today we decided to do something a little more than just a tacet sit and watch some slides so Brett and I will be kind of talking a little bit it'll be more conversational style um feel free to ask questions in the chat Brett if you can keep me honest on time because we're going to go free flow here so we will I do have some slides we'll go through those but I'm more than happy to answer questions uh talk about stuff and we even have some participation things here so if you're joining us uh via Zoom get ready because we're going to be asking you poll questions you will be quizzed so I'm going to share my screen tell me if this is looking good perfect we're good all right nice so there it is make sure to reach out to uh accelerate research.com and accelerate research at any time uh and also on LinkedIn I want to talk to you about calculating the ROI of ux and this is the the new business Paradigm there's a lot of chatter around the uh the ux sphere on the internet about about how how can we ensure that we're making impact and amplifier impact and a big proponent of Bottoms Up um I've never actually been on a SE Suite yet I've been able to sit in the room with the SE suite and I'm going to explain exactly how to do that a little bit more about me yeah I'm going on 19 years here uh I have most of my experiences is in in-house ux teams so um I my last you know about last seven years I've been more of the operations level I'm trying to put together uh uh rules and tools for other ux research and other uxers within organizations and this presentation here represents about seven years of work um so yeah I'm distilling some of my life's work here in a very quick four-step process because like Brett was talking about my whole my whole Spiel is that I want to make things easy and actionable um so everything here is very very practical very real world and I've seen it work in small orgs said it's not just me other people have used this framework in their own organizations so I'm pretty uh I'm pretty confident that at least a portion of what we go over today you'll be able to start using right away and I write a I write a weekly substack so go ahead and if you want you hold your phone up or whatever grab that QR code um you'll see this uh I've been doing it for over almost two years now this is that complimentary uh article it's the one that's pinned here calculating the ROI VX you'll see it says a fstep framework we're going to go over four steps today so for those of you that want to really dive in deep I highly recommend going to the article itself um and you can see I do silly illustrations because these are Big dense topics you know each each one this is not for the faint of heart it's about a 3,000-word article but um trust me it it will give you all the tools that you need and furthermore we have a handout that goes along with today's presentation so when Brett and I were talking about this we're like let's hit it from all angles so uh we'll be able to put I'm assuming Rett you'll throw the link up in the chat somewhere um but you'll be able to download this and it'll give you all the stuff so don't worry about taking notes all the important bits are here uh in the handout and let's just start getting into it Brett anything you need before I get going here I I think we're good to go yeah all right let's let's do it so again here's the four steps the fifth step would be elaborate on even further in the in the suback article but today we're going to go over these four things and I believe they are one-offs they're complimentary but just if if you do nothing but watch this presentation you'll really be along your way um to being able to quantify your work and your team's work so yeah let's start off let's um step one here we're going to talk about ux metrics and benchmarking so first I want to just talk about the framework to establish these metrics we want to make sure we have terms and definitions aligned here because everyone calls these things different and they might mean different things but here's just the basic definitions I this is how I like to think of it again I try to make it really easy understand and you if you Google this you'll find a much longer definition but metrics are just Quantified data right and then ux metrics are Quantified data that deal with you users and ux and usability so um there are just tons of studies that prove that when you integrate ux into your business uh you know there's that famous I think it's McKenzie study it says like 204% increase of XYZ when they was a design forward org how do they get those numbers how do they quantify it's rigorous it's real it's systematic but it's also simple math and we're going to go through those steps and then benchmarking um this is I really want to talk a lot about how benchmarking is critical for this so it's not as mentioned much in this presentation but the real key to this is to set the this up as an ongoing ongoing practice and the way you're going to have to do that is by finding benchmarks this is to establish a bench a baseline measurement so that you can compare over time and what we want to do here is we want to take just match the way your org thinks and take what we know about users and user experience and meld those together it's it's quite a simple idea um yet this is one of the biggest questions I get uh day-to-day from some ux practitioners Brett from your standpoint um your clients are are thinking about this a lot are they coming to you how does this work in your world yeah I know we talked about it yesterday but uh think benchmarking especially you know 2023 2024 and then into the new year right uh has been an area that we've specifically focused in um you know obviously teams are shrinking we're all aware of what research whether that be market research ux design Etc um we're being asked to do more with less uh and and kind of being held accountable right uh and with that you know the best way to do that and show you know that return on investment that that ability to say hey we've produced this and this is where it's been going is that benchmarking process right and um I know you and I have talked in in detail uh more more technically about like how do we do that and and the different methodologies to do that and all that stuff but definitely something we're seeing a lot more of um in a lot of our major Fortune 500 companies uh but also interesting yeah right like it's it's how do we how do we figure out or quantify like where our product is and you know as we make those iterations whether it be in an agile environment or more longitudinal you know quarterly or whatnot uh you know we want to see Improvement right we want to see the the track go up we want to see either you know seat scores or NPS or you know these other components improving as we produce a feature and be able to correlate that and uh you know we've we've talked as well about you know how how to do that and how to do that well um and you know that's a whole other topic that I I think you and I are going to hit on back in or over in March uh but definitely something that is is huge for the industry right yeah we I'll touch on some methods here but it'll be kind of a over a glancing look but yeah and and I I just really I I boil it down to just this you want to match your efforts with the language of your organization right so everything you said and then let's convert it and if you look at my little Graphics that I have in the corner they they're meaningful right so this is a virtuous cycle it keeps us as researchers and ux practitioners honest um and it really uh it really feeds this kind of like Loop of value than our ORS and and quite frankly it's it's what all the other disciplines we're working with it's the standard they're held to so I think it's time that we really kind of figure this out I I think that's where I've seen you have the biggest success in your career has been the translation component right even thinking about you know this webinar series of you know something more visually oriented like this this deck that we're walking through today with folks or you know a one pager that we're going to produce or that we're going to share out after the end of this or even your blog post right like it hits on the different skill set the different maturity within these organizations and benchmarking means something different for you know a major Fortune 500 company uh that has an established you know ux core team versus one that might not um and you know the different uh platforms of the different types of materials you've produced allow people to dig in in a different way right so it's it's I think that translation piece as well is is huge into kind of communicating that Roi yeah and this this presentation is um it's kind of extrapolate L enough that I believe if whether you're already doing a lot of this and you just want a little bit of guidance maybe in some specific areas or if you're starting from zero you'll be able to just upskill and onboard like immediately with this kind of framework so I think it's what the Thomas Jefferson quote was like I wish I had more time I would have wrote a shorter letter um this I was able to condense all this so thanks for that that um kind of like feedback there so yeah I it's just like if we want to be taken seriously like the rest of our colleagues we're GNA have to do some benchmarking so I just want to leave the main takeaway so all right let's do our first Zoom poll this is going to give me a little bit of understanding of of who's on who's listening I'm going to cater what we talk to here but um so does your organization currently use benchmarked uh ux metrics yes we use them no but we're planning to no we don't this will help me Cater kind of which slides we Linger on which slides we don't yeah as as we wait to uh these results to come in you know would love to know more Trevor about uh you know you you've talked about maturing an organization from you know 0 to 60 right uh and and some of that framework that you've used and that benchmarking that you've established in order to kind of push that out there right yeah I think a rule of thumb for this is to start small and there I I do have a recommendation I think it it's either in it's in one of the three communication pieces either in this presentation it's in the uh the handout or it's in the the blog post but a big recommendation is to if if you're starting from scratch do this on one smaller project if you just follow these next four steps that we're going to detail and then try to like Let It Blossom out I think of it as like concentric Rings you know so if you have success in one area it could just Blossom out and this another thing is think about how you would add this onto your current process so many time the push back I hear is that this we don't have enough time to do this kind of thing well I I kind of in practice in the real world especially if you're starting small and you're not trying to boil the ocean right away it adds little to no time this is very straightforward very simple I just think people get Meed down in the details of the steps um that's why I love this presentation because it is you know toward the end of the presentation you'll see where I start to lose people but because we're going to go along the way if you're engaged throughout uh you know these series of Zoom polls and our conversation that I think you'll really be able to grasp on to it and we'll we'll we'll really be able to understand even if it's the first time you've ever thought of this by the time we get to the details you'll be able to understand deeply what we're talking about here yeah from from the initial poll right uh uh with the people that we have in in in the conference today on on Zoom looks about 57% don't actually use benchmarking ux metrics but about 23% uh plan to implement them uh in the coming year so looks like a lot more are not using them yet uh about you know let's say 80% are not using it and you have that 20% that are nice yeah you got some growth mindset there right they're not using it yet that's another beauty of this um the handout itself is a great reminder and gives you the playbook for how to do this so hopefully you know we've boiled this down in a way that's entertaining and memorable but the handout I think will be able to jog people's minds all right let's get on here so yeah we were talking about methods just let me go over some methods so okay Trevor like here's the next question that I always get okay Trevor we want to we want a benchmark but how like what what do you mean by this like what metrics what what way are we going to get this data and that's on you you have to go get this Benchmark data so to gather the data there's a couple different methods I'm going to go over just the basics here you know there's your surveys you've got your analytics this is usage analytics if your code is instrumented Google analytics for websites um I I I I very clearly said quantitative usability testing but this is all behavioral testing you know think of it at the uh at the human behavior level less at the kind of like uh uh uh the um qual level here um support or help reports so this is your tickets stuff like that these are kind of the four buckets that I want to focus on today I'm sure every um organization is unique and you don't need all four of these I'm just putting these out as places in which these popular research methods could really help and I've got a nice and I've color coded them because everyone likes color coding so these are just some examples no I didn't put M uh M uh MPS in here so that was by on purpose talk to me later about that if you want to talk about but these are just some ideas so if any one of these and this is in the handout too right if any one of these jogs your memory if you're one of these 57% of people that are starting from zero just take a look any any one of these something you have today I really love the the use of secondary research look around your or what do we what do we have today what are we currently um capturing and guess what support support's almost a treasure Trove of great uh user Centric data so that's always a really good place to start in fact I'm going to use that as our example today um because I love to use real world examples yeah I think that's interesting in terms of the you know Jura air table there's a lot of Technology out there now that you can create those tickets that are very uh cross functional to Engineers right I think jir specifically um in all the Fortune 50 companies I've worked in that has been the go-to kind of platform right and and kind of the ability to get a ticket or submitted a ticket and how you see that go through the workflow process uh we had a opo group at one of the companies I worked for and that was literally their job is just tracking who was responsible and what was kind of responsible and then we started you know meshing up with them to figure out okay how do we see all the things that we're recommend in all the things that we are doing where is it going so then we can start start converting and tracking like hey I have this research team that's doing about 50 tickets every quarter this team that's doing about 10 and then we start correlating a sizing to it um in order to justify like hey why is there a disparity of 50 to 10 and what's the size of the actual ticket right uh but completely agree with everything you have on here yeah and that's that's the that that's the first step here so we're talking about these first steps the the only thing to your story there is then now let's quantify it you know into convert it I mean into some kind of like business forward like unit so um I like to put this in just I love to do these thought exercises so instead of like you know if you Google this you'll find a bunch of academic research and a bunch of Frameworks but I've boiled it down to just this if you just ask yourself a couple questions so I'm going to use this framework throughout just ask yourself you know which specific measurements will demonstrate the Improvement in the user experience over time that is a really good question to ask your yourself to make sure you're Gathering good metrics all right now I want everyone to pay attention because we're we're going to give a quiz here um a Zoom Quiz so we're going to do my favorite example is fake University um I made it up it is fake go fake go unicorns um we have some sports fans here so they're going to be playing in the big game coming up I think so here's the scenario listen up to the scenario we're working on the online registration process for existing undergrad students so imagine this web form um we know from theist to the details because I'm going to try to trick you in the answers right so know from the past research that students often struggle to register their accounts again this framework simply ask yourself which specific measurements will demonstrate the Improvement of this user experience over Time quiz time let's pop up the poll so the question is what are some possible ux benchmarks for the registration task and select all that apply here and you know as as people are voting on that there was a a comment in the chat about uh like research Ops right and and you know the role that research Ops plays how they can start showing their Roi and you know how it's commonly a a team that you know Services design or research or content are overlooked at times right are not supported not given the investment um what is your opinion what is your feedback of like a research op specific role uh and and that support how can they show that return on investment like what are what are some tips tricks what are some things that you know you've seen be successful yeah so in one of my past careers I was I led ux research for a multi-billion dollar um uh Tech conglomerate and I was the head researcher for the Ops Team so what we did was we built like I said rules and tools for all the rest of the uxas and one of the biggest things we did was to help ux leadership was to be able to give them methods for quantifying this and it started just it's funny it's exactly if you're a research team of one or if you got a huge ux team presence with within your org depending on where your starting point is if your starting point is at zero it's the same recommendation you know you find that sympathetic project you find some lwh hang fruit something that may not you on the surface may not be as difficult to quantify and you go and do it once and the idea is like an opt-in approach you do it it's that virtuous cycle like that little image once people see the outcomes and then uh it'll Blossom out and you'll get more and more people more and more sympathetic people within your or it's like a bell curve right so there's going to be a bunch of early adopters there's going to be that 80% and then there's going to be people that never want uh never want to integrate this kind of thing yet if it's merit-based and you actually are able to put some of this stuff together over time it just it's it's like a meme within an organ organization I've seen it grow in in a matter of months from one little effort by one little researcher into a huge effort going across all things and it doesn't take a lot you know it doesn't take a lot and try to pick things that you can rinse and repeat so when you look at you ask yourself that question which of these uh task specific things will be able to uh will be able to track over time think about that across especially if you're a large or you can think about across all your lines of business across your different product lines across whatever you know which one touches most things and you can set that up from the beginning yeah I think you hit on the scalability component is is the biggest part of that right is what can I do as one person and then hey the justification of if you add another one of me like look at all all the things we can do and then you start growing a team under you right and and be able to shape that um this will this will uh make you happy but terms of that second poll there uh looking at um you know task success rate came in as 80% uh and then task completion rate came back as 85% right so uh a lot of uh feedback on success rate and uh tax completion I mean again we talked about quantitative usability and that's really two of the major aspects of what we're looking at um what will make you even happier is MPS came down towards the low that's about 5% so I know you have go yeah all right so yeah let's go unicorns fake University let's look so again I love this framework just ask yourself this question let's reflect on the poll results in fact I mean I was trying to trick you on purpose you know like the stuff the survey stuff MPS is something you can track over time I mean it's fine I just don't it's not my favorite it's great though I it's better than nothing for for sure so you could do success rate completion rate ease of use um oh wait I have MPS over here as uh doesn't yeah sorry MPS is one that's like hotly contested so I encourage everyone to go look at measuring you uh their synopsis of this on their website but uh essentially what we're looking at here is brand awareness doesn't really give us uh the participant recognition and the company's uh brand it doesn't really match the task specific part and with the MPS uh does really knowing how people would recommend does it really help is it really persuasive so the reason why I put on the bad list uh MPS in this uh case was simply because um if it resonates at your org it may be a good place to start right but I I say that as a lash Resort I I I think that's buying so so for me Roi is part of like the Buy in right and NPS could be that buy in to get you that seat at the table and then how do you help mature that or to get to step two uh which you know the kpi element that you're you're going to jump into now yeah it's it's kind of like my recommendations here are like like it could work it's just the last I just I really want to kind of steer people away from uh MPS as one one um but uh if if that's all you got go for it I think it will be beneficial in in some way um especially if there's like wide adoption throughout the org um the the reason why and I talk about it in my blog post the reason why MPS in particular is not the greatest is because it doesn't quite it does a good job mapping to most of the business needs so think of it as a big vend diagram there's business and then there's a user need and we want to hit right in the middle we want to hit that overlap and it does good on the business need meaning it maybe language your business AR speaks but it really doesn't do a great job on the user need right it doesn't resonate as much with you the the usability or the efforts that a ux team are doing although there's studies that show the correlations it's it's just not the greatest all right step two picking good kpis so now what we have to do now that we found our benchmarks we need to have kpis so again I love this idea to just have this framework ask yourself what does my organization care about and what are the metc that everyone not just the design team pays attention to so this is kind of the inverse of the MPS problem so this would be like you you probably don't want to like quantify cognitive load right that's probably ux jargon and I believe it or not I see so many ux teams losing leaving their companies behind with ux jargon so the main takeaway for this is picking kpis it's different for everyone again I want to make it actionable so most people will not say this because there are you know there's tertiary examples in which this isn't correct but in most cases let's be real it should tie back to money um it's just the bottom line This is mostly what your organization cares about if it's I mean say you're a nonprofit or you know there's other there's you you can extrapolate these learnings from that but just know that in most cases if you're like most of us you're going to want to tie this stuff back to money that's speaking the the uh the language of your org all right quiz let's see if people can get good kpis that was is a two slide that's just a two slider so which of the following make good K good what I'm calling good kpis perceived usability customer acquisition rate profit click-through rate operational cost savings retention rate I can just go too if we don't want to wait and you know I think it's more of a you earlier in my career it was very much tied to doing good right and and putting good out making experiences or good experiences people love uh was was a big slogan um and now obviously as the our industry shrinks and uh you know we're underst staffed and I I see that either coming in through the YouTube comments or even uh on the comments on on the zoom link here uh you know it's quantifying that with the dollar signs like you said right at the end of the day you know we need to justify our roles and we need to find a way to make the business more profitable and a lot is is optimization of designs it's a lot of making experiences easier to use and and getting people to where they want to get sooner um you know spending a lot of time in retail uh that was a big component how do we get conversion rate up how do we get people from Discovery to transaction uh sooner rather than later right yeah efficiency is a great one it kind of you know it's it's a great place to start if you have any kind of time tracking data that's a good one if your code's in instrumented already like you know that's a good place to start that's a good point like talking about an easy starting point that's always a good one yeah still waiting on those results um looks like they finally came in uh retention rate is the highest um at 72% and then we have customer acquisition rate is about 69 70% there um so those are those were the top two for the results here yeah I mean the TL tldr is just doesn't equal money you know profit that's literally money acquisition you know convert to profit or project to profit equals money operational savings cost is a type of money you know it's a negative money but you can save negative you know you can improve on negative money retention rate uh lifetime customer money over time so it really is just ask yourself does this stuff tie back to money um and perceived usability you know it is important to understand this but how does that convert to monetary value the perception of usability right this is what we'll get through surveys you know those kind of like some of those big uh sus type surveys um click-through rate although it measures engagement it rarely translates directly to revenue or cost these are more what I would call marketing metrics you know this kind of Falls in that Gray Zone of um of The NPS all right so now we've got our metrics we understand we've just asked ourselves that simple question what specific thing can we track over time that gives us our metrics now we look at all the possible things that that fit that criteria within our organization and we say which one of those are easy to convert to money and then to Brett's point if you're thinking about scale you know opt for things that go across different product lines across different workflows if it's a single product something you may be able to enhance later on and then the last do the conversion this is the part that I lose everyone this is the part that uh people just have trouble doing um a vast majority of the consultation work I do with other ux research departments is on this component so we got to convert those things so let's say we're uh this is where we're talking about average time it takes to provide this is just an example right operational savings so I'm going to just straight up do an equation instead of trying to talk the theory let's just we're going to backtrack this we're going to reverse engineer it so let's talk about operational uh savings equals the amount of money saved right couldn't be any simpler but we have to do that conversion because this is not literally the same thing time we talked about time earlier you must convert to money so think of it very simplistic so liters to gallons you know 7.6 I love this example because it's like not a round number because you're not going to find these easy conversion points right so this you could easily see how when you're converting whatever it's like metric to whatever I forget what it's called the the nonmetric system what we use in the United States but like you know those kind of conversions this is where it kind of gets complicated and they Compound on one another okay here we go pay attention we're back at fake you let's go unicorns for the big game the scenario you talked with your colleagues in the registration office and they said you've been spending a lot of time they've been spending a lot of time on the phone helping undergrads so there we go there we know this is a problem where oh we have have support tickets we have support time that's being tracked great this is a great way to tap in a great starting point so we decide that your ux metrics is time spent on calls why is that the metric because it's something that's specific that we can track over time so we take the average salary now this is you have to go within you have to go on a little bit of a a fact finding Mission so we think in in research land you know like this kind of stuff is not in inherently a ux researcher job but we're perfectly suited to do it we don't really think we think about secondary research as like past interviews you know user interviews past usability studies past you know go back and look at the integrated metrics like in the past th those kind of things but there's a whole wealth of information that you have to track down within your org so if your first step is to identify the things figure out what your good kpis now you need to actually track down the real data so we find I we find out that this is the average salary of a help uh person we take it down to hours you always have to kind of think about what are we going to actually do so this is leaders to gallons gallons to leaders and the final equation will look something like this so total time spent before the redesign by the way see this semester you also need to have converted units right days quarters semesters whatever makes sense for your work minus the total time spent okay so uh uh this is we're trying to figure out uh cost here is equal to the time reduction of the cost pretty simple take the time reduction multiply that by the hourly cost and we get our money saved I'm going to go through that one more time so the time spent before minus the time after gives us the difference take that difference this is trct over time right what I mean by the difference then you convert that time to money and you get saved money this is pretty straightforward pretty simple an example that anyone could run with today so let's actually plug it in it could be something like this x amount of hours minus the amount of hours after the redesign wow look at how look at how fewer hours we've we've done we did such a good job on that web form good job ux team uh fake you is very they they hire only the best at fake you um take those hours multiply them and now you've got a big old number so I show people this and they're like whoa okay is this real world how does this work yeah it is real world we I come up with numbers like this all the time for basic things now in this registration form example I picked one thing now it might be a micro workflow interaction and it might not be that big of a thing so you want to you want to make sure that when you do the conversion don't just immediately say okay I did it high five report to my stakeholders make sure it's meaningful if you have to you can um I like to call it just daisy chain a handful of these different equations together to make one macro finding one macro kind of insight and the beauty of it when it goes to money you can always add those together that's where it gets really complicated so this was a one variable conversion imagine multiple Vari variable conversions now I have detailed examples of this step by step in the uh in the uh uh blog post drum roll please boom so the ROI so all we did there was do the conversion right we just converted kpis to money so what is the ROI and this is really really where it where it starts to you can start to estrap late this out also I want to caution you don't want to get too uh you want to be conservative with these numbers this is where the the you can really it's a merit based system right so you want to be held accountable for your predictions so you can say you know if we save this over two years right four semesters it could be something like this or sorry uh saved over I times it by four but let's assume this weird fake you as four semesters a year and then I said we'd reverse engineer this is the standard Roi calculation this is everyone around the world and it's what we just did the return on investment is equal to the savings divided by the cost times 100 times 100 is what's going to get us this ability to report calculations responsibly Brett I flew through that does this make sense you can see why this is where I lose people I I I think that's um you know going back to the conversion element right of liters to gallons uh we deal with that every day and and kind of when we speak to our stakeholders right whether that be clients if you're on agency side or internally you know your stakeholders of product engineering business uh you know marketing Etc um they're asking question one over here really they're asking this research question and we translate that so you know you said it's it's never our job but who's better than than a researcher to do that translation um and then you know to your point here uh a lot of people are in in the organizations I've seen uh you know they're just doing the cost savings right um they're just showing the the first layer of the problem of hey look at all the things we've saved well that's money not spent right so organizations say well that's great but then what are you doing right how are you moving that needle what are you doing to get us profit I think that second part of of what you showed is look at all the things we are doing to be able to produce where our money is being spent you know I think back to uh my my last role um and we were designing this POS this this point of sale inside stores we rolled it out to four Flagship stores at first uh just to do some initial testing right and it was supposed to increase the user experience where someone could just go up throw the stuff that they had and you know it would scan it and walk out right you didn't have to do anything with your credit card you would uh you didn't have to do anything with um you know a a checkout system it would do it all for you um it was terrible uh at that point in time right and and how we re-engineered that of now we don't have to do the engineering time we don't have to buy all the hardware and put those into the stores and do the installation fees and all that kind of stuff um and then end up saving the company multi-billion dollars uh and and translated into you know the the aspect of you know just just cost where my team was able to then go and spend the time and what we were able to do from that and show the translation of hey we've improved this experience and conversion went up but then this is also what we didn't work on and why we stopped it and look at all the saving so end of year came we had this you know the big saying was you know don't have squirrel heads have a moose head we had this big moose head of look at all the costs we saved but also what we did from them yeah excellent and and I really kind of gloss over the uh the equation here right because the times 100 that gives you a percentage and then you can go back but I dive much deeper into this uh in in the next section here and then in the blog post so if you want to learn more about it it's it's all here and then lastly I want to talk about reporting calculations responsibly go ahead one question we had that came through is is the cost internal external cost of the of redesign in this example it was calculating the internal uh cost savings but I do think there can be both right like oh yeah it can be either in this example it's talking about the uh and then we convert the time saved by the call center by the the the into the actual money that's what we did here yep and then what we we can do is then do the classic Roi calculation to get a percentage and then I mean I can show let me just show a real quick example from the article so like at the end you can report it in such a way I'm G to I'm going to eat my own dog food here like I'm GNA let's see here yeah so then once you do that you get this percentage and then you can say stuff like like for every dollar spent we gained 100 or sorry $1.75 in return that's how you get to the ROI component here does that make sense absolutely we're getting a lot of positive feedback this is so helpful um on on the comments oh great okay yeah I I don't you know I I kind of rifled through this quickly but I do it for a reason because it's not that complicated you know it's like pretty straightforward it only gets complicated when you when you layer these on top of one another um Qui quick like a little uh kind of down a rabbit hole um yeah obviously this is the process of we've went through this we've done the redesign we've done all the work um you know at times we're we're not given the affordability and we're we're told like hey we need to figure out how to make this better and you know what's going to be the return on investment before we do it right and that's not easy to do those are very uh speculation numbers um and you know that's a great area uh but in your experience you know what have you seen where are there ways to predict those changes or those numbers or those percentages roughly on like hey the team might spend this amount of time doing it and then it could translate to this but what are your thoughts on you know more predictive numbers than it is actual executional numbers yeah so I this goes back to benchmarking I really do believe I would hesitate to do any kind of predictive stuff unless you do some Advanced Predictive Analytics or something of that of that nature to uh to to use actuals I think I think it's it's not it's not hard to start doing this today for the time in which your boss asks you that question and then you can say you could be a rockstar and save the day and be like Oh I'm glad you asked because we have this method of actually looking at this right um but yeah I think the step one is to do the benchmarking today so that you can then turn them into kpis that match what your boss or whoever else wants and then you can do the conversion but if if you don't have the benchmarking if you don't have the current data um you can always go and look for secondary data that's lying around on that's the way to do it and say I you could calculate a past project and say our past project did this we predict a similar thing in this project or a past project did this um we believe this is this new project is different in these ways therefore we can predict this but those are my and I I talk about this next here in the in the in the slides but like you really want want a caution on the side of especially every time you start to strapl out when you take that number and then add four or multiply by four you take that number and you um or this predictive number and you're doing apples to Orange conversions and uh you really want to be conservative so my my recommendations are round down use ranges low to medium never even go to high and you know surprise them when the outcome happens because you've undervalued it you've under uh uh you know estimated your itions because nothing is worse than the opposite you want to tank that's actually a good point if you want to tank this initiative overestimate and then and then when the actuals come uh you know build all these unfounded expectations I I like your your point of the example right of of calculate the ROI on a past project um and it's almost that agile nature it's almost that fail forward component of what research and design and that space is is really meant for it's it's not for certainty all the time it's more a let's go experiment and figure it out and and you know getting into that world of certainty getting in that world of hey should we even spend this money um you know then it really starts putting that Cog in the machine and and does that set the precedent to create a culture that you know it's going to be difficult to do these experimentations that lead to the Innovation that lead to the newest technology or features that we might need right yeah exactly and and like I'm confident that like the handout itself is enough information to start doing this you know but I mean it is supplementary it's right the example I use in here is guess what that's a web form from you know I forget what I call there University's web form so there's the simple example and then there's a more complicated example down here where we go really really nutso that's more real world um these conversions are extremely powerful and it really is these four steps rinsed and repeated you know so you know the fifth step on there is just rinsing and repeating you can think of it like that but if you understand this framework then you'll understand that all you need to do is rinse and repeat to get these compounding um um kind of like benefits from your calculations so you have to remember a couple things and this is where it gets a little weird because as researchers we're usually talking about actuals these are not actuals and they are not Financial projections right these are extrapolations once you do that conversion and you start to extrapolate out it's based on projections so now we're in the world of strategy we're in the world of business analytics business strategy stuff like that just acknowledge that I want to put that out there because sometimes I get to this point with teams and the researchers are slightly hesitant and they're like not comfortable reporting this stuff and like no no let's just stick with time it's like no the conversion is critical and this again if you're concerned about it round down you know think of it like that um these don't have to be perfect and they won't be perfect they are projections again and just be transparent say every step of the way exactly what you did don't put it in your report but always have that hyperlink for that one stakeholder that doesn't like the outcome so they want to read up on it and you it's a virtual cycle it keeps you honest and if you do these four steps in a very rigorous and honest way like for instance I talked about like what what would this look like if it were easy if it were easy and you just started with some like secondary research or whatever be transparent in the report do not pretend I that your tendency will be to pretend that these are solid numbers right that they are like hard fast actuals like we like we deal with when we're looking at user Behavior data it is not actuals it's projections just know that and be transparent about it so last last time our last visit to my favorite University on earth fake you um so we have we we saw the reduction in time um now what are we going to do we have to assuming the average cost for 1 hour of the operator's time was like we said 2644 that's a projected SA Savings of x amount over one year X amount over 3 years you can keep us strapl liting out right again we're going to round down but this is how we might want to say it uh this isn't the slide this isn't the report this is the storytelling aspect so data as storytelling is a huge kind of theme this is kind of what how I think of it so you can say this he note the bolded words approximate right so the it cost us approximately this amount to make so you want to front load with that we estimate our return on investment will be around the number that we got over three by the way what what's that unit that unit is whatever resonates with your with your organization right it could be semesters it could be years it could be whatever resonates um and then that's that's really all I wanted to talk about here is that like at that final stage you want to make sure that you're being very very clear with a word approximate estimates they're not actuals I kind of got ahead of myself I I went on my rant before the slide but that's the slide that goes with the rant and then um so this is just the four-step recap right you want to make sure that you're using real ux metrics you want to make sure you have business facing kpis don't forget the user but make sure that they're business facing here's a hack it's going to be money you want to take that and convert convert all that stuff and then you want to report with 100% transparency if you follow those four steps like I said the fifth step is just rinse and repeat on more complicated um more complicated type calculations but this framework I've seen scale I've seen people upskill really quickly and you can start adopting it today so I hope this was kind of a helpful framework for thinking about it um and then I just want to leave us with like a final thought here it's like why why really even bother to do this you know I think we've probably convinced Everyone by talking through here and we have a sympathetic audience I'm guessing of people that understand this in general but like you know it's really really essential for us to not just lean on that McKenzie I think it was Mackenzie that study that says we can't just go into and say you need to invest in ux because some macro study proved that designed forward companies save up to 240% worth of efficiency that is true but I've never ever ever seen a macro overarching data point like that um move the needle when it comes with the real world in the real world we're going to need to really show our actuals or not actuals but we're going to need to show our our real impact using these kind of projected uh uh Str strategic type calculations by changing everything to money we're talking to them in their own language by using the the metrics that we have so you're going to have a whole bunch of metrics only keep the ones that we think resonate start there that's a really good I see so many people go down the wrong path when they pick the metric that that resonates the most with the design team you want to really think outside the box if you have to find a sympathetic internal business partner who's going to like help evangelize this idea across your work that that's a really good way to do it run it by them is this something that you think we care about usually you you you kind of know this but win and doubt run it by some people and that's all I got today so want to open up for Q&A and I want to give a little you you can sign up and download the handout here so go ahead and hold your phone up click the QR code and you'll get this this is for the folks that like are not going to deploy this right away and didn't take copious notes this is gives you everything all the cliff notes you don't you know you're not going to need all the uh you're not going to need all the all the fluff in there all the nonsense about fake you um that'll just give you the the uh the bare minimum that it will take to execute this I've I've literally seen ux research practitioner practitioners take nothing but that document and then run with it and be able to execute so I highly recommend you look at that in terms of your uh recap Trevor I think like the really the main things that I that resonated with me is like you create allies you you have these uh friends positioned throughout your organization right and it's kind of putting that fishing pole out there to tease them with hey is this a metric that's going to hit with your portion of the org like is this the way you guys do things um and and relying on those those allies right cross functionally um and making those translations uh but then also you know something that I've seen uh especially lately with an influx of you know more junior staff going into the research space has been the overindulgence of data right just because I collected the data I have to regurgitate it back um and what you're saying here is like that's not necessarily the case right like sometimes less is more but hitting on the right kind of data and hitting on on the stuff that is going to resonate that's going to land that is actually accurate in in to the estimation component that's what's going to build trust that's what's going to you know move the needle and be able to calculate the ROI of ux and actually be impactful versus just uh hey go do this and then go report it out and you know there's no impact from it right like that's the biggest thing that stuck with me from your call is the impact and the conservative nature of what we're we're recommending here uh to actually have something that that is beneficial and and helps you you know grow as a researcher right yeah like no one knows this but you but I did not read any of the presenter notes we were just riffing here and I totally missed that point that's a great point is that like we do not need the level of academic research rigor it is okay to start small the key point is remember the the the reporting is to be totally transparent about it you know I I really say doing something small today is much better than you know the big reveal and I don't know why I this just occurred to me I don't know if I ever I mean we said it in real time but like I'm definitely going to add this into the way I think about it is like yeah partner with the business minded people if if you don't know or at least they could help pick because you want to you want to focus on just one or two of these at once right you do not want to try to like boil the ocean like I said again so pick the number one um metric that converts best to a good kpi and run with that as your first point of uh of calculation and who's better to help pick that for you but one of your business partners hey here's a list of things that I can report on you know we track these over time and we can convert them to money don't you know what people do I've noticed that people want to kind of reverse engineer this and pick the one that equals the most dollar signs that's not always the right way to do it in fact many times that's not the right way to do it you know especially when you're thinking about making a practice from an Ops standpoint and you want to scale this so don't necessarily think about what's the biggest home run at the end of the calculation it's much more important that it resonates with more people yeah I'd Echo that right like I think um you know former roles I was over such a large portfolio and What mattered in the front of funnel compared to the back of funnel vastly different um being able to tailor customize adapt it to that specific audience where all they care about is having get conversion rate right so if I'm going to show them you know the the overall like last mile delivery or whatever we I was focused on it's not going to land it's not going to resonate right it's it's not really worth it uh versus like you said pick your audience know your audience um and and make friends right and that's a larger uh soft skill networking tool that I think us researchers are getting more fine-tuned at because of all the different hats we put on and all the different roles right it goes back to it's not our job but who better than a researcher to go figure that out yeah well and it's not not our job that's kind of my point like we are suited to do this very well and if someone's already doing it reach out to that person you know um I I guarantee you that the lens of the user Centric uh uh researcher can be laid on top of any one of these business metrics in a way that could really help clarify the the the return on investment that ux is making you know it's just like most businesses aren't just by default like design-minded user centered it's it's just it's just untrue and if they if they are then you can skip this all together but for the rest of us I think it's important to to kind of think about this awesome I questions yeah no I think it's another plug for your substack right I think you um you know you're producing stuff weekly and uh I've always enjoyed you know being able to read it and digest it I think your last was a recap of uh your year right but uh um love the love the aspect but yeah let's uh you know feel free to reach out to myself to Trevor both on LinkedIn uh through accelerant um you know we'll communicate with him if it's it's a question pertinent to him as well um but you know excited to to continue this yeah and I just want to leave this on a note like I just want to thank you for the opportunity to do this um when you reach out to me about this opportunity uh I I had no idea what we do I mean I remember the first conversation was like here's a bunch of stuff that I talk that people ask me about which one do you think and and the the ROI one stuck out uh in a big way so I just want to say thanks there's there's a a companion article on my substack it's also been republished uh on uh uh um userinterviews.com like a version of it uh it's it's slightly distilled down even further but the one on my substack is the one that's got all the nuts and bolts and and the craziness including an extended version of so the most complicated example in this presentation is the simple example in the substack article so if you really want to dive into it um I think it's a little more practical in that case but again I highly encourage everyone to download the the handout and hit me up on LinkedIn at any time I'm I'm happy to talk more about this kind of thing so I just want to say thanks that's what I want I want to say thank you for uh you know allowing me to have this time to talk to your audience and um and to just kind of like lay this out in a a really real world actionable way yeah absolutely yeah we always appreciate it and uh you know looking forward to to March as well and hopefully having you back so um yeah any questions feel free to reach out guys uh thanks for the time and oh wait are you are you are you I'm gonna put a version of this on your blog as well do you think yep yep we should be producing that uh later today uh so that will be coming out and then we'll also link that to Trevor substack um in the user interviews uh it's irrelevant to to for me where you which which of the blogs you read but it is a nice companion piece to the presentation and the handout yeah awesome uh thanks for your time everyone I know we are at the hour uh so want to be respectful of the time but um you know feel free to reach out and and uh our next one is uh February 11 so I'll see you there