Why Can't We All Get Along: A Comparative Critique of Qual and Quant Evaluation Standards

Good afternoon everyone. Welcome to another virtual insights conference hosted by Accelerant Research. My name is Christa. I am vice president of operations and I am very happy that you have joined us today. I'm excited to introduce our next speaker. But before I do so, I just want to go into some ground rules. So, everyone is probably experiencing some type of nasty weather out there. I know I surely am. We're in the Charlotte metro area. So, um please just know that we're at the mercy of tech. If anything does happen, we will do our best to get back on track. Another thing to mention is to just use this opportunity to network, to engage. We will post questions at the end of Brandon's session. You can submit those in the chat um on YouTube or through the question and answer feature in your Zoom platform. And with that, I will move on to our next speaker. Brandon is going to be talking about qual and quant evaluation standards, critiquing the two as a mixed methods UX researcher. Brandon is joining us from United Health Group. So with that, I will stop sharing my screen. Brandon, thank you so much. And everyone who's joining, thank you. We're excited to have you and hear what Brandon has to share. Awesome. Thank you so much, Christa. Can you hear me? Okay, I sure can. Awesome. Excellent. Thank you so much, y'all. Uh, so my name is Brandon. I'm a mixed methods UX researcher. Excited to be here with you all, fellow researchers and people who do research. Um, it's great to engage with everyone. So today uh mostly be talking about you know why can't we all get along a comparative critique of different evaluation standards. Um we had a great balance today. Our previous uh presenter was very much chatting about the difference between tactical versus strategic research. Very much reminds me of Jared Spool along with politics of UX and all that stuff. This is a great addition to her her work as well and the fact that it really tries to bridge the divide between tactical and strategic in some ways. But yeah, so to kick things off, I have a provocative question for everyone if you would entertain me. So please, if you would like, go ahead and grab your phone and scan this QR code and really try to help us answer what makes something scientific. It's it's supposed to be a hard question. I'll also paste the link in the chat here for everyone if you're on a computer. I'll give everyone just a few minutes there just to see if they can log in and we'll see some live results. This is of course assuming some people engage. If not, no problem. I'll just give it a couple couple minutes. Academically based. Yeah. What does it mean to be academically? Yeah. Study phenomenon via scientific method. Yeah. Robust methodology. How do we define robust? Yes. Something that has logic. Testing and evaluation. Lots of scientific method processes. Numbers for assessment theory to evaluate. Rigorous standards. Yeah. something that starts with a theory that can be tested with multiple inputs or outputs to prove or disprove. Interesting. Yeah. So, very much kind of leaning into that hypothetical like hypothesis driven paradigm. Well-designed unbiased it's testable with a hypothesis, best guess, method, data, conclusions, something logical and hypothetical. Yeah, these are great. Y'all testing or explain exploring something new or unknown. Stated hypothesis based on conclus confidence intervals. Okay, so very going there. S systematic approach that's often hypothesis driven. Yeah. So this is a really hard question to answer and there's a whole discipline dedicated toward defining science and what makes something scientific. It's called the philosophy of science actually and it's there's still great contention and lack of consensus of what it means to be scientific. It's an evolving definition and so pat yourself on the back you know if y'all gave it a shot which is fantastic. Many of the many of these evaluation standards tend to lean towards a quantitative paradigm. So I want to kind of jump a little bit into why listen to this presentation. So main reason for this presentation is about resurfacing evaluation standards applied to qualitative research specifically to ensure it remains a quote unquote rigorous scientific approach as you all were doing very much. I really appreciate you engaging on the that what makes something scientific and the last uh just jumping into many of you had scientific method confidence intervals very much numerical paradigm hypothesis driven so here's some common critiques towards qualitative research and I'm sure many of you have heard this I myself have heard this many times as well sample size is too small it's not random the findings are anecdotal they're not generalizable what hypothesis are you testing? Great stories, but where's the supporting data? It's subjective. It's influenced by the researcher. It lacks rigor and systematic approaches. So, these are all common critiques that I've come across qualitative research. I'm sure many of you all have. The struggle is all too real for those that embrace qualitative research. Um, but and this is the last uh QR code interactive exercise that I have for you all. Don't want to create too much burden, but think's happening. What's the issue? And if you would like, go ahead and gather your phones there. And I'll also place the link inside of the chat as well. Let's give everyone a few seconds, then I'll slam over to the uh the results. Here we go. While research is undervalued. Yeah. How we got here? How did we get here? Why is it undervalued? Untrained researchers doing research. Contentious. Contentious idea. Doing our best with limited resources. So it's like a subpar it sounds like compared to what we could do with the quoteunquote scientific method. No quant guys involved. Defensive stakeholders. It's definitely hard. Lack of trust in current qual methods. Yeah. What? Well, let's define trust. What does it mean to be trustworthy in qual methods? Co impacted how qual research is conducted. Definitely. Definitely. And there's some evaluation standards that we can get into that kind of yield a little bit of signal error. So, I appreciate y'all entertaining me for going into those interactive approaches. This is all great to the presentation. So what I speculate is that these the issue really lies in applying evaluation standards that are rooted within different philosophical paradigms. They very much value or vary according to different philosophical paradigm frameworks. But why as a researcher is this important? Well, it's it's our responsibility in my eyes to know how to implement and critique research to be quote unquote rigorous to inform decision-m reducing uncertainty and managing risks. As the cliche quote comes from Einstein, if you judge a or supposedly Einstein, if you judge a fish by its ability to climb a tree, it will live its its whole life believing that it's stupid, which is true. I mean, you got to judge it by its according according uh evaluation standards. Evaluation standards are very much dictated by their philosophical paradigms. And I know many of you, this is review that you've had from previous academic reading. Um this is just an overview of the philosophical paradigms. I'll briefly go through ontology, epistemology, frameworks and most will be um for the presentation will be on evaluation standards. So jumping into on so what is on this is quite heavy still so bear with me y'all but it's about the nature of being that answers questions See there. Can you all hear me now? Thank you so much for Yes, we can hear you. Awesome. Thank you for speaking up, y'all. Um, all right. So, quickly identifying ontology. Ontology is really about understanding the nature of what's being um answers questions like to what degree do things exist independent of human perception. The old adage for instance is like if a tree falls in the woods do you hear or is there does it make sound? Of course we know the answer now and so forth. So this is not an exhaustive list of different offlogies. This is very much a simplification just to contrast the two between quant and qual. In quant it's very much about realism which is one reality exists the subjective world so to speak. A great example here as user experience researcher, usability tends to be a common construct we're interested in is assuming that u usability problems exist whether or not uh users are aware of them. So if several users for instance are having difficulty completing a task, navigating a specific feature as an example um a realistic a realist realist excuse me will consider a usability issue to be independent of the person's experiences. So as an example loading time loading time would be a great example of realism. Relativism is very much about multiple realities existing. So in this case it'd be contrasting to focusing on how different users experience the app based on their background or their context or expectations. You know one one participant may be techsavvy and see is like it being easy to use another one not being techsavvy so to speak. So next going into epistemology just going to make my way up that ladder. So epistemology is really about uh critiquing a statement on what how do we define truth? What makes a statement to be true which is a really hard thing as you all are very much aware of. Um on the quantitative side it very much ascribes to what's called the correspondence theory of truth. It's essentially how well a statement aligns with this one reality that exists. Like it's very much hypothesis driven. As an example, 90% of users experience slow load times on an app or website. So if you collect data through usability tests or actual server logs that confirm that slow load time to be true uh quote unquote true because it corresponds to the observed facts that are happening. So it's about the singular truth. On the qualitative side, it's about I mean these truths can be interchangeably for quantum qual, but very much the qualitative side adheres to the coherence criterion of truth. So this is how much a statement is logical and integrated into the context of the situation. Great example would be uh believing that users prefer simple interfaces because of prior research. Similar products with fewer features had higher engagement. So when conducting interviews say with users for a new product you are able to um they express for instance that they pre they have a preference for that simplicity. So it aligns with previous findings that the conclusion is considered to be true based on past research. Another theory of truth is pragmatism. Pragmatic theory of truth. This is less about being objective or subjective. It's really about the utility of the research finding. A great example is like adding a one-click uh purchase button. So, this will quote unquote improve sales e-commerce app as an example. So, after the feature is implemented, we see the sales increase by 20% and according to the pragmatic theory, the observation is true because it works. It produces some type of utility to the organization. So teasing apart these nuances between what makes a statement true. Yeah, exactly. App loads slowly. Yeah, definitely. Um, but understanding and teasing apart the nuances of what is considered true allows us to constitute what research findings are quote unquote rigorous. So again, as researchers, it's our responsibility to know how to implement and critique research to to be rigorous quoteunquote and inform decision- making within product or marketing within the organization to reduce uncertainty in managing risks. Uh moving along here, just going up to different theoretical frameworks. All right, let's see. So in the diff different theoretical frameworks on the quant side very much about positivism. This is by August Kunt and the goal of positivism is really about prediction and generalizing laws out to the general population or cause and effect. You know running an AB test and example one group users having a certain feature the other group of users not having a certain feature and trying to measure the outcomes of like time spent and number of interactions that yield signal into user engagement. On the qualitative side, an example would be interpretivism. And the goal here is less about prediction and more about understanding and interpreting the meanings of individuals. So a great example would be, you know, understanding how customers, users emotionally connect with a new product. So instead of focusing on the numbers, it's really about exploring personal meanings that can inform, for instance, marketing claims or marketing campaign that the language is used. There's also deconstructivism. Lots of heady stuff here, y'all. bear with me. Um, the goal is really here to reveal underlying contradictions or biases. So, you might argue that for instance, user satisfaction is a term shaped by societal norms. Um, and measuring it reflects like a diverse user experience or just enforce narrow views of success. And then critical theory which can be want is wall kind of go it's primarily kal at least but it's goal is really to emancipate or liberate great examples here would be marks or vapor flu co for those that are interested in critical social theory and then you know after going through the ontology epistemology and u theoretical frameworks um really talking about the evaluation standards from the perspect perspective of social science and human subjects. All right. Uh I think I've got a little too ahead of myself here. So let's let's compare critique uh jump to the meat and potatoes of the presentation. The qual evaluation standards. So both methods quant and quall can be viewed through these two different paradigms which is confidence and relevance. Confidence is really about the degree to which the research findings accurately represent the participants realities. So in qualitative research an example would be you know why qual researchers probe for recent examples about a theme that's surfacing to ground the participant in what actually happened. It's really about having that trustworthiness and in quant an example could be for instance um internal validity. Then on the flip side there's relevance and relevance is really about what and how the research contributes meaning meaningfully to the existing theory or problem at hand as you can say. It's really about the utility of the research finding. And then so when thinking about evaluation standards, coming back to what we were talking about earlier with what makes something scientific, the quant evaluation standards tend to be very well known. It's kind of considered and perpetuated in society as like the gold standard, but less is really known about the qualitative evaluation standards. So I'll briefly go through the quant evaluation standards just really quickly. But as a whole, this is what the quant the equivalence of for instance reliability measure and how it's um equivalable to a qual evaluation standards of reflexivity and triangulation. Words are hard right now, y'all. Words are hard. So thank you for your patience. All right, thank you. Recovering from a cough, so I think I'm having a dry throat, so to speak. But you can see each of these standards are being viewed through the lenses of both confidence and and relevance, which is what denoted by the C or the R as an example. So briefly going to the quant evaluation standards, as many of you already know, won't go too far into it. uh validity, reliability generalizability quoteunquote objectivity. And an author Cavale talks about how quant evaluation standards have become the holy trinity. And a great quote or paraphrase for instance that the author mentions is they've nearly achieved a sacred status approaching the reverence of a holy trinity and are worshiped by all deers and followers of science. So it's really established this type of uh uh prestige, I guess you could say, amongst our society. that has a historical influence as well. Won't go into the historical influence for this presentation. But jumping into the qual evaluation standards. So the qual component also has an equivalent to what was considered the holy trinity when that is accuracy as an example which is based on close observations. So not remote indicators. A great example would be thinking of participant observations. you're there in the moment with the participant versus interviews, you're talking about someone's past experiences or hearsay. So, it's really how close you are to the phenomenon that's happening as it's happening and you can start gathering that information reliably, quote unquote. Another one is precision and precision is really about capturing the fine grain detail of things that are happening in C2 in the moment. And then breadth, which is really about gathering a diverse amount of perspectives or context lattering up to the context of um the situation at hand. So it's like someone's not just purchasing through a phone for instance or viewing their options through a phone. They're maybe at home after work as an example that they've in a certain socioeconomic status, a different gender norm as an example and the like. any meaningful kind of demographic characteristics. So I've commonly seen you know in evaluation standard at least for myself I'm sure you all have as well with qualitative research that saturation is tends to be viewed as generalization. This is not the case. So saturation is very much about the completeness of understanding of a phenomenon under study. There's no new information that kind of surfaces. So the the um initial term was was coined for thoroughess and depth of understanding not about making like broad generalizations to the other populations. Excuse. But what what are those evaluation standards beyond saturation? And this kind of comes back to that those that table that I was talking about between quant evaluation standards and qual evaluation standards. So I'm going to bullet through those uh each of those qual evaluation standards. They'll mostly be the the meat of this presentation. Starting with reflexivity. Uh so reflexivity is really about an awareness or sensitivity of the researcher's presence in shaping that data, interpreting that data and analyzing it and sharing it amongst the society. So great examples here would be of course you know the Hawthorne effect or even considering one's own perspective that they're bringing into the the interpretation or their presence on how that's managed and the like. There's a type of uh co-construction that's happening in in the data formation. Another one is triangulation. This is just this goes beyond data sources and methods but also different perspectives and observers. observers meaning a peer a fellow re UX researcher as an example looking at um the data that I'm having and the processes in which it's lattering up to the themes I'm arising another there's many different kinds of triangulation especially in qualitative research data triangulation investigator triangulation theory triangulation and methodological triangulation a big point of triangulation is searching for contradicting data and this helps bring out the complexity and nuances of the um the phenomena that's under study. I made sure to have all the resources here for you all if you're interested in learning more. So I know this is a very brief uh touch of the iceberg so to speak. Going to the next one, corpus destruction. So corpus is really about uh having that set of data. Think of a data set in qual or in quant excuse me this is analogous to that in qualitative research. So you're building a structured ideally a very diverse comprehensive set of data different perspectives and representations to kind of uh to eventually start having that process of interpretation. So here corpus really aims to represent the complexity of the research at topic. Great example would be, you know, a collection of medical texts, research articles, um, transcripts, a clinical notes, and used to like help understand terminology or language used within a certain phenomena. Just drilling it out for you all. Uh so thick descriptions, thick descriptions is really about both describing the phenomena on your study when you're watching the participant as an example or talking with them, but also simultaneously interpreting those social social interactions within their specific context for is the purchase person purchasing on their way home from work on a public transit? Are they at home? Are they with friends? Um and the like. really just trying to understand not only their thoughts and feelings and and relationships, but also the context, the rich interpretations of how these things are are coming about and interacting with each other. And doing so really helps bridge, you know, helps our stakeholders when communicating things that are really have thick descriptions and help um in UX research, it's very much about bridging the divide between the product team and the users, developing empathy and helping guide informed decisions. So, it's really about helping our stakeholders, as an example, really sit with those findings and have that type of empathy. Ideally, it be actionable. So actionable insights that are empathetic in nature if it's qualitative. Next one is local surprises. So local surprises is a concept that's defined by unexpected findings. So things that kind of challenge common um common held beliefs within your organization as a great example or even theoretical expectations that you have especially if you are coming from a certain academic framework like psychology, sociology or what have you. You can ladder that interpretation up to a theory and understand for instance how it's contradicting that theory or what have you. Trying to have that comp complexity of the nature. So it's really about diversity of phenomenon. So this really just pushes the researcher into reconsidering their initial interpretations and assumptions and having a deeper understanding and nuances of the phenomena under study. Bear with me y'all. I know I'm going with uh drilling it drilling it all down with the evaluation criteria for qualitative research. Next one is transparency. So transparency is the thoroughess and the rigorous documentation. This is similar lading up to thick descriptions and corpus construction, but in this way it's having all of that uh not the cor the corpus, the processes, the interpretation, all of that stuff being documented and also disclosing that information to your peers as an example for challenge and making sure you're also surfacing any things that were unexpected that happen or any challenges that you experienced and so forth. really just trying to promote the integrity of trust of research. Going into the next one here, communication validation. It's really broken down into two concepts called peer debriefing and member checking. So pure pure debriefing is similar to what I was talking about earlier in that I'm having a colleague for instance come in and take an audit of the corpus the processes and the themes that are coming in making sure that it's honest and makes that coherence criterion of truth is making logical sense then membering a common one now is engaging participants in the validation of research findings. So you bring them of course without any jargon synthesis of your research findings along with the data that happens of course depending on what you're willing to disclose to that participant um and have them give feedback for instance. This does not mean however that the researcher's role is not in is like a reproducing the participants view of themselves. It's not regurgitating their information. The researcher's role is still very much about translating that information and understanding its meaning for instance to the organization making it actionable. The last one here is transferability. This is similar to generalization you can think of in quant but in this regard it's not about generalization. It's about understanding the research findings on how it may or may not be applicable to a certain context. So it's really up to another researcher for instance to decide and inspect whether or not for instance these findings may be relevant to their context and the like. So it's providing contextual details for other people other researchers your stakeholders really emphasizing the importance of context how findings may be relevant in different settings. So gone through all those evaluation standards for qual. I appreciate your patience in going through them. Again, the reason why I think it's important is because as researchers, it's our responsibility to know how to implement and critique research to be quote unquote rigorous to inform decision-m ultimately reducing uncertainty and managing risks appropriately. And the solution here is very much in in my eyes unifying that approach through mixed methods UXR. I'm sure many of us here are familiar with mixed methods. The reason why is because pragmatism especially in industry the utility of research really calls for a mixed methods research or quote unquote the uber science as my partner likes to call it. This is not an exhaustive list of mixed methods paradigms. This is just a few that's quite common. Um one being convergent parallel design doing quant then qual or quant and qual separately then coming together interpreting that data and sharing out explanatory sequential design which is very much about doing quant work. So say for instance do experiments or surveys about the willingness to pay or perceived usefulness of a certain feature that you're hoping to launch then from those research from that quantitative research it informs the objectives for the qualitative research. Why do people choose that certain feature? What's their hesitancy for pain? And why is that happening? Which really understands and brings complexity and strengthens the the data. And then of course exploratory sequential design, which is where we do uh qualitative research first as an example, participant observations or ethnographies. Uh say we're developing a new product. you're um trying to understand how people are navigating the health care space, what challenges they're experiencing, what's not working, the quoteunquote unmet needs, and then following that up, for instance, with surveys to generalize out, you know, how how prominent are these things amongst the general population because in uh industry, we very much tends to value uh quote unquote market level problems, not just small use cases. as an example. So if we able to generalize out for instance which of those themes are most prominent amongst the population, it can help the industry or the company decide what features are for instance worth worthy quote unquote of of um of tackling. So really seeking unification here with mixed methods research. It's because leverage it leverages the strength of both quant and qual creating robust insights and the call to action that I'm advocating for is for researchers to understand and use proper evaluation criteria when critiquing the other side understanding its limitations and making sure you're not taking things out of context because once you can understand those strengths and weaknesses especially as if you uh combine your project with another stakeholder that has complimentary skill sets qual or vice versa then you're able to develop a full story that helps really mitigate risk and manage uncertainty when making decisions in in the organization or you could upskill yourself if you're interested in learning about different the other paradigm and the like. So the main takeaways here is evaluation standards are very much dictated dictated by their philosophical paradigms. on the left thinking about quantitative research. It takes a realism ontology has a correspondence theory of truth epistemology and ascribes to a positivist theoretical framework which in turn dictates its theor evaluation standards. On the qualitative side, it's very much about relativism as its ontology, embracing a coherence criteria of truth along with pragmatic theory of truth, epistemology, and then the theoretical frameworks of interpretivism deconstructivism and critical theory, which all determine the evaluation standards of the qualitative research. So, they're very different. It's very different. We don't use the other to compare the other. And I know I've said it a couple times, but I'm just going to reiterate it again. So the main takeaway here is why it's important to understand these different evaluation criteria is because as researchers we must understand how to implement and critique research based on appropriate evaluation standards because doing so really enhances scientific ri rigor ultimately increasing the utility within the organization to reduce uncertainty manage risks and inform decisions in the industry. And this is just a quick recap for instance of each of the evaluation criteria within quant and call and how they may compare with each other through the frameworks of confidence and relevance. And with that, I know I kind of went a little fast here, but I want to thank you. Really appreciate you all listening in. Um and yeah I if you would like I would love to have everyone's if you're open to it have some feedback a survey being a UX researcher would love to know it's anonymous as well uh just feedback for the survey what works what didn't work things that you think would be useful for changing just to make it more uh useful for making pivots to make things more useful for you all in general Yeah, with that I'll open it up to Q&A. I see some things popping in the Q&A box, but I will digress for the moment. Thanks so much, Brandon. We do have a few comments in the box and also a question. One question was, "It seems like common sense. Why is mixed methods research not popular in marketing research?" That's a great question. I can only speculate because I'm in uh user experience research. My colleagues over in market research, I'm sure, will have some some really great insights there as well. From my perspective, I think it's a historical influence. You know, the scientific revolution, for instance, very much embraced and dictated for instance that what was considered scientific to embrace a positivist paradigm, which is not what qualitative research ascribes to be. And even in market research, we tend to be more invol uh more valuable or it tends to value market level problems. So problems that exist amongst a population not really interested in for instance 10 people. It's really unless it's about for instance changing uh you know marketing campaigns or ascribes to a qualitative framework and the like. uh in my opinion I think it's very much has a historical influence and its utility within capitalism because the more people that you're able to understand for instance of a certain problem or certain more people experience theme A versus theme B you have that knowledge then it tends to involve more transactions more money keeping the business alive when things really comes down to it I hope that's making some sense and feel free to reach out to me on LinkedIn as well I'm more than happy tussle it out with everyone if that's helpful. Thanks, Brandon. Another question is, is there any common criteria for establishing when saturation has been reached in qual research? Yes. Uh, say that one more time. I'm so sorry. Making sure I understand that question. Yep. Is there any common criteria for establishing when saturation has been reached in qual research? Yeah. Uh there's different processes, different standards that I'm personally aware of. Um it does it's the common theme is it's very much idiosyncratic and up to the researcher to define when things have gotten enough. Really big importance is having a code book as an example. But it's not just viewed in isolation of each other. I think that's really important. So you don't just take saturation out and say this is what the not to say you're doing this whatsoever but the reason why I say it is because with saturation thinking of transparency and bringing in your peer for instance and seeing if they agree with you you know does does it make sense that we are reaching saturation that kind of practice will really help define the rigor and trustworthiness of that data yes there are frameworks I'm familiar with them I don't have I think I have it one-sided inside of this presentation But most of them are on my personal computer and I can um I can add that to the presentation or reach out to me through LinkedIn. I'm more than happy to provide some information there. I have a question and my question is what are some of your best practices for working in those local surprises when you're shifting into analysis and reporting? Yeah. Best practices for identifying local surprises. Yes. And how you work them into analysis and reporting. I'm assuming that you'll have to somewhat pivot a little bit and rework maybe some initial findings that you prioritized. How do you go through that? What's your mindset and your approach? Yeah, absolutely. So, in qualitative research, unlike quantitative research, it's uh it qualitative research tends to be very linear. It's even the papers tend to be very linear hypothesis, methods, results, etc. follows each other. And qual qualitative research, it's very an iterative design. Not to say quant can't be that, but qualitative research, it's um the reason why I bring it up is because as you're constructing your corpus, as you're constructing your data and gathering, for instance, information, you may figure out that for instance, there's this part that's missing that you want to uh probe for more. you get that you go back to your research build up your corpus again and that's when things may arise. Um big big uh in my experience something that really helps develop local surprises is the diversity of your data set. If you can improve the diversity of your data set with different perspectives of course across different meaningful criteria then it's going to complex complex comp complexify if that's a word it's going to make the data more complex allowing for local surprises to occur when it comes to synthesis and analysis. So hopefully you have all that stuff. Um you're going back back and forth between analysis developing your corpus analysis developing your corpus. So two participants as an example start developing corpus start doing analysis go back for instance develop your corpus anymore. It's that iterative process that's really the standard for qualitative research. Um I hope that provides some signal into what you're asking. If not please let me know. I'm more than happy to speak to it more. That was great. Thanks. I don't see any more questions in the chat. Um, just feedback about how great your presentation was. So, thanks again, Brandon, for taking the time to walk us through this. I'll turn my camera on as well, so I'm not speaking um to a screen and not everyone. But again, thank you all of the participants, everyone who joined. This video session is recorded so it will be available for you to view. For those who want to share. Also, if you have colleagues or friends who weren't able to join us live, just know this is recorded. You will be able to access everything that was presented today, including what Brandon had to share with us. So, with that, um, thank you again, Brandon. You did a wonderful job. A great topic, very interesting with some awesome examples. And to all of you who were with us live, thanks for your time. Feel free to take the extra time that you can to look Brandon up on LinkedIn and uh reach out to him, reach out to us at Accelerant as well. Thanks again and stay safe out there. Awesome. Thank you. Would it be okay, Christa? I see Sherry I think Sherry mentioned a little thing in the question. Is it okay? Yeah. So um there's a question that's there are some standards for implied saturation that are used in industry that are largely revolved around numbers of participants such as five plus or minus two for usability testing. Yeah. So I had a recent conversation with Jeff Soro and Lewis who are from measuring you um and very much worked with Jacob Nielsen the original author of the study for the five usability test or five people for usability tests when operating iteratively. It's very much uh it's not about generalization. So if for instance five people experienced um I'm not so sure that's what you're meaning here but I'm just kind of speaking to the point. Um but for instance four out of five people experience problem A versus two out of three five people experience problem B. It's not a an indicator of generalization. It's an indicator like you said of saturation. It's think of it as like a rock for instance. You want to make sure that this that there's a there there that the rock is a um ex it's it exists or whether it doesn't exist. We don't know whether or not that rock is big or small generalizability. It's very much about understanding which of those problems occur. Even the if you are familiar with quantifying user experience that book by Jeff Soro Lewis they talk about um the binomial probability formula which is where the the the recommendation for five people in usability testing is recommended. Um and that's derived around specific parameters. One of those parameters being the probability of an a probability of occurrence of seeing a problem occur at least once. It doesn't say anything about for instance having it be four or five. That could be a sampling error for instance. But yeah, but really wanted to echo that as well because uh again reach out to me on LinkedIn. I would love to have conversations more about this because I think there's gaps in our industry which aren't really talked about too often and is commonly misunderstood and maybe I have a misunderstanding on some things as well. So I would love to, you know, dish it out with everyone peerto-peer and and help strengthen each other. Iron sharpens iron, as my kiddos like to say. They learned that from Ninjago. Thank you so much, y'all. Thanks, Brandon. Everyone, stay with us. Our final presentation will be at 3:00. So, although Brandon and I are signing off and um will not be on your screen, our next speaker will and we're very happy to invite them as well. Thank you so