Debunking A/B Testing Best Practices

hello everyone thank you for continuing with the quarantine conference today I'm just going to share my screen right here I hope everyone's having a good day and thank you for continuing into the afternoon we have had some a lot of interesting talk so far we still have good content to come so get your coffee and staying engaged and let's make this as opportunistic as we can so here are the rules again I won't go into too much detail because I think you all have heard these a number of times but don't remember to stay engaged use the hashtag get on LinkedIn get on Twitter and make the most of the conference and please remember to be respectful so next we are going to hear from chad sanderson he is the head of product and the data platform at convoy and he is going to be speaking to us about debunking a b testings best practices and so we are all very much looking forward to hearing from you today chad you can go ahead and and share your screen now yep you're unmuted now you should be ready to go all good all good okay perfect okay hi everybody let me go ahead and start this presentation here oh okay cool so hope everybody's been enjoying all of the other speakers and also enjoying your time in quarantine I am slowly losing my mind so this is a welcome break thank you for giving me the opportunity to do this a little bit about me I work at a company called convoy convoy is a digital freight brokerage it's a unicorn startup worth about two billion dollars based out of C and we do a ton of experimentation a lot of machine learning some really really cool stuff and experimentation is is really near and dear to my heart before I was at convo I worked on Microsoft's experimentation platform team so that is the internal a be testing tool at Microsoft that groups like office and Skype and teams and Xbox all used to run their AV tests so experimentation is something that I love a lot although something I've noticed recently that's been magnified a lot in the wake of koban 19 is that a lot of business leaders and stakeholders who at one point we're really bullish about experimentation and optimization and AV testing are now starting to question its value a little bit I've even had a few of those conversations myself over the past few months luckily not so much at convo more so at Microsoft so what I wanted to do with this presentation was take a look at some of the best practices that I feel may have led to these sort of questions coming up and how much value the common best practices in the AV testing industry actually add so Before we jump into talking about best practices I actually want to talk about kind of how we arrived at this situation and how business leaders and stakeholders have been sold AV testing okay so a lot of you have probably seen some charts like this if you're familiar with AV testing there's usually some x-axis which is time and then on the y-axis we have some metric that we care about like revenue or app downloads and to gradually diverging lines which represents a counterfactual world or a different universe where in red we have a universe where we didn't do a B testing and changes and improvements just sort of slowly sputter along and don't really go anywhere and with a/b testing we have these you know masses massive improvements and really huge jumps and at the end of the day were two hundred twenty percent better than where we would have been a lot of these charts were shown to business leaders and stakeholders by third-party experimentation platforms and of course your eyes would bug out of your head if you saw this right like six months you're telling me I'm gonna be making you know two hundred percent more revenue that's amazing however what I've noticed with a lot of these companies that get sold on the hype of a be testing is that at the end of the day the results actually look closer to this chart so you see the the a B testing results or at least the reported results and the improvements that features deliver at this gradual linear climb over time whereas the actual business results or what somebody's looking at in a P&L sheet or what your financial team might be looking at is relatively flat and how do you actually rectify these two things so so I'd let sort of continue down this road of like what were people sold and what were our business business leaders sold about a/b testing so I went back in time on Google and I was looking for some of the ways that experimentation or randomized controlled trials were described and there's a few common headlines like these are some examples of what you might see right the gold standard the rise of the randomized control trial and experiment design the gold standard of evidence the randomized controlled trial so if you're not familiar with the term gold standard it basically means the best possible way of doing something oh the best possible method if you have a gold standard for some particular field or activity and you want to do that activity if you use anything less than the gold standard you're essentially doing it in an incomplete or unoptimized way so the question is what exactly is the gold standard of a be testing talking about what does it encompass and here's a few comments that I found from some of these articles where it mentions just that experimentation is the gold standard for learning cause and effect in scientific research okay that seems pretty reasonable it's also the gold standard for measuring the effectiveness of changes to a website of online optimization strategic decision-making evaluating new product strategies and even eliminating biases so again you imagine that you're a business leader and you're seeing these amazing charts and then you're reading about a/b testing and it's saying that it's the gold standard the best possible method of strategic dissent decision-making and eliminating biases of course it sounds like a really incredible and important investment to make for your business and a lot of the experimentation platform so the tools themselves would essentially back up these beliefs right this is a common output of a test result so test be converted 34 percent better than test a and I'm 99% certain that the changes in test B will improve your conversion rate well that's pretty awesome right you basically have a tool that's telling you I know exactly how much this feature is better than another feature and I am extremely extremely certain that that change is real and you can see this reflected in a variety of other tools as well that are all making similar claims about what they know on the data and the likelihood that the changes they've seen are are true and real so there is something really important to understand about all the data that we just saw and is that it's derived from experimental statistics ok so experimental statists anytime you run an a/b test experimental statistics are performed and the output is usually translated into a significance value or a p-value or a confidence level or something like that so the critical piece here is that those types of experimental statistics only interpret the past they don't predict the future and that's actually a really really important caveat to add when when understanding what these types of results are telling you so for example here's a here's a quick visualization what we can know about the impact of an experiment over time there's sort of two time periods that we care about when it comes to an a/b test the first time period is the time that an experiment ran so that might be two weeks or three weeks or four weeks or whatever it is and then there's the point in time where we then need to make a decision we need to analyze our results and figure out if this feature is good or not and then we can push it into production or onto the website and then there's the future time right like then there's everything that hasn't happened yet but we're gonna assume that the experiment is probably going to deliver a similar value over that time period right so this is what we know we have an experiment it resulted in a 10% change we had a 10,000 people in that experiment and we got 95% significance on that well that's really great so what does that tell us about the future well the answer is we don't really know right because like I said before experimental statistics are only descriptive of the past they make no statements about the future at all and that brings us to sort of the first important point which is that the questions a B testing can answer are not actually the question the business leaders really care about so if you're a business leader and someone tells you that a feature delivering a 10% improvement across a week to two week time period I imagine that your first reaction is going to be okay that's cool but what about its improvement over the quarter what about its improvement over the year that's what I really care about not the small gains that happen from week to week that's way too variable so you might be saying yourself okay Chad so I understand that experimental statistics describe the past and we can't make these predictions but that doesn't really matter because I'm following the best practices and I've read online that if I long as I follow the best practices and I do that consistently then my experiment results are all gonna turn out okay and eventually I'm gonna see this nice aggregate improvement to the bottom-line business value so let's talk through a through a few of these business best practices and actually sort of digest them so the first one I want to refer to is probably the most common best practice and a/b testing which is that you should run your for two-week intervals so this is Google optimize which is a quite large third-party experimentation service and it says keep an experiment running until either two weeks is passed or at least one variant has a 95 percent probability to beat the baseline another example from another tool you can see in this middle section is set a minimum time the experiment will run with a default threshold of two weeks and then finally this is during the data gathering phase keep your experiment running we recommend running most experiments for at least two weeks so as you can see this is not an uncommon thing to believe the tools and the platforms that we use are recommending this two-week threshold so why are they doing that what is actually so special about running an experiment for two weeks well this is from an article from C Excel which is a really great website and I strongly recommend checking it out if you haven't heard of it and so I'm just gonna read this middle section here we need to rule out seasonality and test four full weeks did you start the test on Monday then you need it to end on a Monday as well and that's because your conversion rate can vary greatly depending on the day of the week if you don't test a full week at a time you're skewing your results so that makes sense right if I'm running a test and I started on Monday and I want to end it on Wednesday I'm not accounting for Thursday Friday Saturday and Sunday which is the point at which visitor behavior could change so maybe my data won't actually be representative and I won't see the results I expect when I push that into production there's another reason for running tests across two weeks and this is to capture the business cycle or the customer lifecycle so when a user comes to your application whether it's a website or a mobile app or or whatever it is then they usually go through a at least some version of this life cycle as they interact with your with your website they will sort of check out your site check out the offerings maybe they'll buy something then leave maybe they won't buy it maybe they'll leave maybe they'll come back maybe they'll buy it again maybe they'll buy it you know a couple weeks later things like that every single website has a different customer lifecycle and so what the two-week experiment is meant to do is to capture as much of that lifecycle as possible so for example let's say that you have a five week lead time on a customer they come to your website they browser or a five day lead time they browse around for a couple days and then four or five days later they return and they buy well what you don't want to do is only run your test for three days because then you're not actually capturing the point that they're gonna buy your product right you want it to at least go for that long so that's a sort of a high-level summary of why the best practice is to run your experiment for two weeks it's to capture the full time period and capture business cycles the reason that we want to do this is for representation essentially we want to make sure that the data that we've captured in our experiment is representative of what's going to happen in the future I don't want to just run my test on Monday because other things could happen there are other days in a week and I want to account for that I want the data to be representative but if you remember back to what I was saying a little while ago about experimental statistics and how they can be used you might recall that I said experiment experimental statistics are actually not predictive so if they're not predictive what's the purpose of having representation how much there's two weeks of data actually represent in the first place for example if I run my tests for two weeks does that mean that it's representative of the next two weeks or the next four weeks the next month maybe or the next six months or it can get pretty confusing understanding this whole business of represent representation and statistics so here's kind of the bottom line on that in this case we're using two weeks of data as a predictive metric despite it not having any inherent predictive capabilities so the shortcoming of experimental statistics which is we can't use it as a as a predictive metric we can only use it to look at the past we're dealing trying to get around that problem by saying well if we run out for two weeks then it's kind of predictive right we think maybe we can extrapolate this into the future so the one of the important points of this whole presentation here is that if we really care about being predictive that means if we want our experiments to be a signal on what's going to happen in our data six months from now or a year from now or eight months from now or or whatever it is then we don't really need to care about the two-week threshold what we really need to care about is creating the best possible predictive metric if we can do that in a day or if we can do that in an hour then the idea we need to run an experiment for two weeks is actually totally meaningless and this is something that we've done not only at convoy but at other companies I've worked with as well so if you want to talk about how to do that then we can certainly have a conversation after this presentation is over but that's a little in the weeds for what we're doing here today so you might be saying to yourself okay Chad I get that I understand maybe the two weeks is not completely predictive maybe it's not like the best possible indicator that we could use of what's going to happen in the future but fundamentally we've got to start somewhere right like we can't just do nothing and two weeks is pretty easy to remember for beginners and you know it's probably a good barometer it's not the worst thing in the world so can it really cause any harm is it really that big of a deal is it really that big of a problem if if we we used two week experiments so I like browsing I spend a lot of time browsing silly websites and and wasting my time on the Internet especially during this quarantine because I have almost nothing better to do then scroll through Reddit and here is an example of a pop-up I got while I was doing just that so I was just scrolling through the homepage and I get a pop-up that says hey download our app it's 50% faster it has infinite scroll and you can auto play gifts not really sure why that that last one is super appealing but okay the interesting thing here is that I really only have two options I can either click the giant glowing orange button in the center of the page or I can click that teeny tiny little text at the bottom that says or go to mobile site by the way I have pretty big thumbs so it's really hard for me to do that without actually hitting that giant orange button frankly this is a terrible experience right I was already enjoying my content I was already looking at things I didn't need a an app here popping this up while I'm enjoying content seems really counterintuitive to making a good user experience let me look at another example Yelp right similar thing I'm on Yelp I'm looking looking for photos of an establishment Omar's exotic birds and for me to see the photos I have to download the app this seems like core functionality to the Yelp product but I'm not allowed to do that my experience is once again interrupted and this time I can't even click away the pop-up and see the photos like I could do and read it and still look at content I actually just have to go back like that part of the app is now shut down for me I'm confused on how that could possibly be a good experience for anyone and here's one final example Cora and I used to love Cora I answered a lot of questions on Cora actually and now when I'm browsing through cute Cora for more than like 20 or 30 seconds I get this but there's more in the app unlock the full core experience this time they don't even bother telling me what's good about it they just say there is more I guess and if you look closely you'll notice there's actually no way to close this pop-up there's no exit button the only options I have are to either download the free app or to completely leave the website and then go back and at which point I'll get another 20 or 25 seconds and I'll get this pop-up again the reason I know this is because I've tried it multiple times so there's an interesting question asked here which is how do you experience the experiences like this at very tech savvy companies come to exist this is obviously bad this is obviously not enjoyable why why is something like this presented with such regularity and the answer that I would assume and have personally seen at other companies is because of Av tests and experiments that result in metrics that look something like this hey are we ran this experiment with the pop-up that prevents the user from doing anything except downloading the app surprisingly that increased our app download rate by 8% it didn't really hurt revenue that much didn't really impact visits that Muhsin sure it might have decreased our content sharing capabilities per user by 5% but guess what we're ok with taking that loss if we get more app downloads so if you're a product manager especially if you're a product manager for the mobile application and you've taken quarterly goals to increase app downloads by a certain percentage and you see a result like this that is a massive massive win now keep in mind that these experiment results happen over the course of a couple weeks two weeks or three weeks something like that what do you think would happen if we were able to look at more important metrics more customer centric metrics across the duration of a year something like this so instead of just looking at downloads per week or downloads per two weeks we look at the total amount of downloads per year so yeah we see a big jump in downloads career 8% but we might also see a decrease in satisfaction by 10% we might also see a decrease in retention by 6% and a decrease in yearly revenue by 5% now of course I just made these numbers up I have no idea what Cora's long term metric deltas actually look like but as someone who has recorded these long term changes for other features these are similar similar metrics that I have seen in the past frankly speaking it is incredibly hard to measure long term metrics unless you really take the you put the data science effort and the analyst effort into doing so it is a very very difficult thing to do and frankly speaking if you're a product manager or you're working in quarters or you're working in semesters these types of yearly metrics don't really matter to you that much it's more about what's happening today that I can report up to leadership and management so let me give you a first-hand account of how this happened to me I was telling you that I actually founded the experimentation program at Subway this is the first AV test that we ever ran on the subway website this is the homepage for Subway comm you can order online you can see that in the small order online button up at the top right corner and we decided to change the color of the cart in the order online icon to yellow now we didn't really have a great hypothesis here we didn't really even think that it would do anything this was more like we were testing the validity of the system than anything else however when we ran this experiment and you can see the variation B on the right we actually saw a lift of about 3% in orders for new visitors in a list of about 2% and click-through rate both of which were significant so for the orders we had a significant level of 99.5 percent that correlates to a p-value of 0.05 if you're familiar was sort of the the best practices around p-values usually point 0 5 is the threshold that you would declare a winner and for a click-through rate it was 97 points 97.5% significant so what did we do in this experiment did we follow did we do everything right did we follow all the rules all the answer is yes we ran the test for two weeks each variation had about a hundred and ninety thousand unique visitors and about fourteen thousand orders we had a confidence interval that didn't cross zero so again that's a statistical concept but if your confidence interval doesn't cross zero then that basically means the potential improvement is anywhere between 1% and 5% so and with a p-value of 0.02 0.03 Lytton's and purposes we had a winner right very exciting although I thought this was a bit weird right like why would it be the case that changing the color of an icon would result in so many more orders so we decided to do something called a reverse flight and a reverse flight is where you run the experiment again in the reverse order so now our control is the new version of the website and our treatment is the old version of the website and we did that about two weeks later what was the result of that experiment no improvement right so what happened we ran an a/b test we ran it exactly the way that everybody in the on the internet would tell you to run it we ran it for two weeks we had a ton of people in it we had a significant improvement we had a p-value of at least point zero five in fact it was ten times that that's significant and after two weeks we reran the thing and there was zero improvement at all so why did this happen what was the reasoning for it well part of the reason was novelty effect so if you're familiar with the idea of a novelty effect basically it means that if you introduce a new feature or you change things in some way then there's a really good chance that people will start interacting with it not because it delivers real value to them but because it's different because it's new so even if you're talking about something that at least in my opinion for that last experiment probably increase the cot the the contrast on the order now button it was probably slightly more noticeable for people even if they click that it doesn't fundamentally change their behavior of buying things and that's what we saw right so we saw a spike in people sort of using it and interacting with it and buying things very very early and then as time went on the performance of that metric stabilized now there are some businesses out there that account for novelty effects in their analysis a lot of people don't do this but a good amount of businesses actually do account for novelty effect in their analysis but even this is incredibly challenging because you have to ask the question of well how much of a novelty effect period should we account for should we only look forward over one week and then we start calculating results should it be two weeks or a month or six months at what point does the novelty start wearing off that's actually something that's really really hard to know and to answer that question to any degree of competence requires a very very large data science effort okay so quick summary here if we're looking at short term metrics only one-week metrics met two-week metrics then it's very unlikely that those are going to predict the future we don't know if those experiments are possibly predicted we have we have no idea if they're predictive until we do the due diligence using data science to actually measure that these short-term experiments can absolutely harm a business you can harm the user experience and you can build bad things for your customers by focusing on things that are very hard to measure through your sort of traditional experimentation software and it's also really really hard to know for innovating in the right direction right so as an example from the subway experiment if we had just taken that experiment at face value and said hey yeah well obviously if we if we if we totally structure how people order and we jazz that up and we spend a lot of time making improvements to the order button then we're gonna see these additional gains right but but that would have been untrue and that would have been a complete waste of time because that's that's not an experiment that really moved the needle at all okay so how do we how do we fix this right this is there's a lot of there's a lot of doom and gloom in that presentation obviously I think experimentation is still extremely important I'm head of product manager had product manager for a sizeable company is making a huge investment into experimentation every single day so what I would say is this number one experimentation is extremely extremely valuable in my opinion it's one of the most important things that product teams can actually do as a validation tool there is no other method that is as good as experimentation when it comes to the measurement of business metrics and engineering metrics so if you're trying to measure those then nothing else even comes close and that's extremely extremely valuable to do well so let's talk a little bit about maybe how you should be thinking about experimentation or at least how I think about experimentation so the first one is to demand accountability demand accountability of any of your teams that are running a/b tests and demand accountability of your a be testing tool that you use so a lot of teams that are responsible for running tools or overseeing internal platforms they have program metrics like the number of experiments that are run in a quarter or the amount of conversion rate improvements that were gained through experiments and then these are reported back to leadership to me these are completely valueless metrics experiments run says nothing about value delivered and conversion rate improvements doesn't really take into account the counterfactual so what what what what would have life been like without and experimentation platform something that's way more concrete is dollars earned over time how much money how much more money did experimentation make you than if you had just deployed this feature without testing it how much more money would it have saved you if you just deployed this feature without testing it these are really really difficult things to measure I'm not gonna lie to you even at Microsoft where we had a huge amount of money and resources these were still difficult things to measure but I would strongly suggest making the investments to understand how your experimentation platform is contributing to bottom-line dollars before you spend a lot more time investing in it and this might be something that you do on an ongoing basis over time as you start to think about how to build better metrics for your platform and for your team next one of my favorite movies a do or do not there is no try and that's talking about making predictions right so there's a lot of ways we discussed making predictions and how experimental metrics are not predictive metrics there's actually quite a few ways to make predictions the right way or at least have some sort of pseudo prediction so for example holdout groups is is one option here where a holdout group is basically a certain percentage of your customer base that never receives an experiment they're never up they never have an updated website they never have an updated mobile app this is this is way easier in for for mobile apps it's very very it's much harder to do for websites because of the cookie churn but it this this is this is a potential route for you what you can do with a holdout group is to basically say all right I've made all these changes and so six months later I'm able to compare the performance of the people who got nothing to the people who have been have been seeing this feature over time and maybe have been seeing all the features over time this is actually also a really effective method for proving the value of experimentation to the finance team because maybe the finance team is looking at a metro like margin or revenue and that could fluctuate over time right there could be outside influences affecting that bottom line metric but if the product teams are improving it within their sphere of influence that's still something that they should be commended for and and appreciated for reverse flights is something that I already mentioned so that's just re running experiments again with the control and treatment switched you can do this any amount of time after you've actually determined that an experiment was good or not you can do it a week later you can do it six months later and then you have true predictive metrics so these are algorithmically based metrics built in built on built in data science we have true predictive metrics at convoy that takes in a short term input metric and essentially spits out a long term metric saying this is what we expect this value to the business to be over a three month period or a six month period and we can use that long term metric in our experiment results these are these are usually pretty difficult to do they're very complex they have to be retrained and they have to be validated over time so you might be saying well I don't have the resources to do a predictive metric and I actually don't have the resources to do the holdout groups or even the reverse flights at which point my feedback to you would be then stop making predictions stop looking to your experiments as if they can tell you what's going to happen in the future if you do that then you're gonna run into situations like we saw at the very beginning of the presentation where your experiment impacts are not matching your cumulative effects so so what if you're not if you're not making predictions then what should you actually be doing one of the ways that I think is extremely valuable to look at 80 tests is as insurance okay so if you go out and you buy insurance you're probably not looking for ROI right nobody expects their their insurance to make them money you are paying into an insurance policy so that in the event that a disastrous thing occurs you mitigate the damage to yourself or to your family and that's a really good way of thinking about experimentation I think so the focus could be stopping really bad experiments early or stopping features that have problems with them something that's extremely common in fact I would say in almost you know 80 to 90% of all the features I've seen built and deployed over the years is that they have some bugs at some level it could be something relatively benign like an instrumentation bug that just you you get like the wrong measurement to it or it could be something really major like a functionality bug that doesn't work the way that you expect it crashes a users experience it returns a whole bunch of error rates it slows the entire site down right these are all things that could happen and they happen very very very frequently if you're running an experiment for all of these features that are deployed then you're gonna be able to catch all of those bad things and fix them and not only can you fix them you can know to a pretty high degree of accuracy who are they impacting how much are they impacting your customers and and what is the thing that you should be fixing in the first place if you don't have any form of measurement around these features that are going out you might not even know that the bugs happening at all right it may be that some customers in Italy or some customers on a particular operating system that you didn't test happen to be running into error rates and the the impact of those is just too small to observe from your aggregate data so if you're able to do this it basically means that every single experiment that you deploy should be a good feature it should be working the way that it's expected and it shouldn't be negatively impacting site performance okay so if you're able to focus on coverage and that's what you really care about you could use something called a non the superiority test that's an it's a statistical test I'm not going to get into the details of it here you're absolutely free to follow up to me but with me but you can basically use about the sample size that you would normally use for a regular experiment so you can get results much faster than you normally would and you can do many more of these experiments than you normally could and this framework is called experimentation as continuous integration and continuous deployment meaning it's it becomes a part of your engineering deployment lifecycle it's just something that everybody does for every feature the date that they deploy regardless of if it's a big feature or if it's a small feature now I've talked about this before and some people have given me negative feedback like oh you know we can't afford to do that are you saying that well we don't care about customer experience anymore no actually in fact I'm saying the exact opposite of that I'm saying that if you run experiments in this way as opposed to using those two-week metrics as predictive metrics then it actually unlocks the type of features that you can deploy you can focus more on features that improve the customer experience that you can validate qualitatively that your customers say are really really good and really really valuable even if you're not able to explicitly measure a significant dollar and cents business impact and and sometimes that is an extremely extremely valuable thing to focus on and then of course in addition you can still measure your your revenue impacts as well but I've seen this framework really help some companies that we're struggling with some of the issues I mentioned before on reddit and in core and and yellow and so with that I have arrived at the end of my time that is a that is all of my presentation the the final word I'll say you know I said it during the during the presentation but I'll say it again just to drive the point home experimentation is absolutely not something that you should be abandoning is something that you should invest in doing right if you don't have the data science resources to make true predictions then I believe the best possible use of your time is to use experimentation as a final gate before the qualitatively validated features that you've built go out the door and and and that's it that's why time thank you so much Chad I'm super interesting we don't have any questions as of yet so everybody start putting all your questions in the chat box that you have for chad is really interesting talk so I'm sure a lot of people have some follow-ups everyone here this one is from Doug great presentation Thanks in this type of a be testing do you ever include a subset of interviews or custom surveys with users to validate findings and/or find out why it worked yeah yeah so normally I can speak to the way that I've done this in the past normally what we would do is validate that the feature worked qualitatively beforehand right so we would go there and say okay you know here's this idea like let's say on convoy we want to give them the ability to rate their shipment facilities so we would build that feature out in our in in our UX we would take it to them you'd say hey you know look through this feature use it tell us what you think and then if we got pretty good feedback from that then we would run it through an experiment to make sure that nothing was going off the rails we might see for example that the way that the feature is implemented not in the concept but then in the actual implementation of the feature is is buggy or it breaks something or if you've done like some particular workflow like hey I before I rated I rated this facility I logged out and I logged in again and then I go to rated my whole app crash those are things that are really really hard to QA for in in its entirety there's just too many potential use cases so experimentation can really really be good it can be good not just for testing those but also for for measuring the actual business impacts that we think the feature might have mm-hmm so for your qualitative methods you do a lot more qualitative than quantitative at this point we do a lot of qualitative I don't know if a lot more I would say it's probably pretty balanced it depends on the type of feature so if we're if we're if we're we're if we're if we're putting out a feature that is very that I'd say is customer-facing or that has customer experience affecting then there's there's a lot of qualitative ground work that goes into proving that that actually is valuable it makes sense if it's something that is not as noticeable to the customer so for example we have a pricing statistical model or an algorithm that we use to basically determine like at what price do we accept a bid like what's the most reasonable bid for us to accept should we let the should we let this auction go on for another three weeks or should we stop it here that's all done using machine learning and changes to that model are things we experiment with so we can actually see the impacts of making those changes in production it's a lot harder for us to qualitatively validate that because maybe it produces a good result for one user but it produces a bad result by one percent on average for you know $100 right okay anybody else have any questions for Chad throw them in the chat I know for me I'm definitely interested in I don't know if you have like another example I feel like your subway example was super interesting about how you kind of seen this in action as far as the two-week testing or some of these best practices you have any other examples of that type of testing that you ended up being not best practice yeah what's another good example yeah so so I'll say I'll say I can't think of any any examples that went out the door where it's like whoa you know we did something really wrong but what I have seen is sort of a cultural shift at a lot of companies away from improving the user experience and I can give some examples of that so so for example it's really hard there's almost a there's almost a console at times between like UX designers and and data scientists and like you know very tech heavy tech centric companies and the reason is because the the data scientist almost always wants to see that before we implement a feature you can prove to me that there is some business metric that has changed positively because of what we've done and when you're thinking about the customer experience it's not always possible to measure it so cleanly with a business metric so a good example of this is actually at Microsoft for Microsoft Word the navigation bar now those types of navigation bars are super common but back when the navigation bar microsoft word was first introduced everybody hated it and there wasn't really a clear signal from customers that that the a clear business metric signal that the navigation bar was going to be something that worked and in the long run but it turned out that because so much extensive user research had gone into this particular navigation bar that the lead the u.s. lead on that project knew that it was it was drastically more usable than the old system even if that wasn't a conscious if you couldn't make a conscious positive comment and so after that rolled out about six months later I think that they ran another test and they looked at the metrics and they saw that there was a this massive massive improvement and so those types of things happen all the time where features take a while before they get adopted so another example at convoy convoys kind of a unique business because we don't have those clean revenue metrics the way that convoy actually makes money is we work with a shipper so this is someone like Starbucks or Walmart and they have shipments that are going out all across the country and they will say hey we need the shipment to go from Seattle to Atlanta Georgia and we're willing to pay you five thousand dollars to ship to find a to basically ship this for us and as a freight brokerage will take that and put it in our auction and the trucker's will bid for it say oh yeah you know I'm already in Seattle I live in Georgia I'll take that for five hundred dollars a mile because I'm going back anyway and then we would make the difference between a so in that case we'd make forty five hundred dollars so oftentimes there's a really long lead time between the fact that we we made some money because it only comes from you know it starts with our shippers to something like UI improvements for the app for truckers right we wouldn't know that a like some UI feature to make it easier to find loads is going to make us a lot more money until more of these shipments more of these shipments actually start their tenders as we call them start coming in so we have to rely on the qualitative piece and and believe that we're making the customer experience better and then take a more longer-term approach to measurement mm-hm right so do have a question from Mary she said so many people think quant big data is more compelling than quell any advice for convincing people on the value of 12 alongside quant yeah yeah Mary I I totally agree with you I think that that is a problem and I think it's because I think it's because quant is it's it's more tangible right it's easier to hold in your hands and say like oh you know I can measure I can see the number of clicks that someone has has taken on a website I can see that that number has improved when I do something and typically I see those changes immediately when you have to go out and talk to another human being and and hear their feedback and then you have to sort of like digest their feedback and you can still you can still take qualitative data and present it in a quantitative way so like this many people ran into errors while they were you know doing something on your application I think that's probably a good start to convincing people of the value is try to quantify your qualitative data which is possible and presented in a way that those more like the the people who are very metrics minded would understand having a paragraph and saying like yeah you know this we've we've seen that X many people like we've talked to 10 people and they really really liked the application that probably is not gonna resonate very well with metrics minded people but if it's more something like hey here's a table here's the number of people in that table that ran into errors here's the number of errors that they ran into and when we tried this new thing here's here's the the reduced volume and in errors I think that's something that they would have a lot more less yeah that makes a lot of sense and we don't have any questions right now so I think we can probably though I have a quick question for you and maybe the the final question you obviously in your current role you are the one who is basically looked to to implement a process like this for someone who is maybe earlier in their career in you know a load or lower totem pole position or in an organization where you know there's just the way of doing things is so engrained and someone wishes to to implement or you know start doing this type of process advise what you get for them yeah that's a tough one because in my experience the impetus for better measurement and experimentation has typically come from an educated leadership team if the leadership doesn't care about measurements then it's gonna be really hard to implement something at a lower level that being said I think that people like pretty numbers and they like green and they like checkboxes so one thing that I used to do when I was in a similar situation as in running a writing experiments on on our particular unit and elsewhere in the organization they weren't doing it nearly as much is when it would come time for our weekly business reviews or our quarterly reviews the way that we presented our in Meishan was here's the feature that we did here's what it was here's the the metric before here's the metric after here's how much value that we added and because we ran an experiment then we can we can start talking about our team actually delivered this much value this quarter and a lot of the other product teams weren't able to do that they were making estimates they were saying like hey look at this look at this chart over time you can kind of see it going up and by comparison a product leader is gonna is usually gonna start asking questions if you present this new sort of worldview to them like hey how do you how do we know that that graph is that that you know improvement is because of your feature and unless you're running an experiment then you really don't have a very good answer to that so so I would say the the best way to do it is to start trying to implement some of this stuff locally there's really cheap ways that you can begin doing experimentation Google optimized for example is totally free and there are other free platforms out there as well there's other platforms that do free trials for a pretty extended amount of time and the the implementation costs are also quite low it's usually like a single line of JavaScript in the header of your page if you want to use something client-side if you want to do it more on the server side then yeah it's kind of a lot of work but you can get started you can actually get started with some of this stuff pretty cheaply another another way to do it is if you want to start on the qualitative side is if you're if you're designing a new feature and it's implemented you can actually run a statistical test in more of the conventional way like more than the academic style test without using all those online fancy you know doodads and gizmos and things like that where you just have this a column a these are the these are the users that are that are doing the using the site in the current way column B these are the users who who use the new feature this is the metric and then there are some really simple statistical testing packages and like Microsoft Excel that you can use or plugins for Microsoft Excel that can give you a statistical output fantastic applause on behalf of everyone else this is a great great presentation definitely appreciated thanks for having me I really appreciate it and I hope that it was valuable in a time where I was just bored out of my mind honestly I'm glad I'm glad I can speak to some like-minded people while I'm I'm trapped I'm trapped at the house well thank you Chad we've got one more presentation coming up in about five minutes this is our final one so so you know Chad if you can stick around and attend that would be great otherwise we'll catch you later this will be our last one so this is going to be our cocktail hour presentation so everybody run run and grab yourself a drink and we'll we'll clink glasses as we go into our final presentation hang tight and we'll be right back all right thanks guys I you