For high-consideration durable goods, do consumers actually think more carefully before buying, versus low-cost packaged goods?

Yes, meaningfully so. The milliseconds decision Tara described at the grocery shelf applies to low-cost, low-risk items like ketchup or a Pop-Tart. For a car, a washing machine, or even a lipstick, the investment level and the cost of a bad choice are much higher, so more deliberate funnel stages, including awareness, consideration, and trial, are genuinely relevant. The caution is against applying a funnel framework universally without asking whether it fits the specific category and consumer.

Screening for primary household purchasers or decision-makers can skew gender or demographic composition. How should agency researchers raise this with clients without putting them on the defensive?

It depends on what the study is trying to learn. If the goal is to understand in-store shelf behavior, talking to the primary household purchaser is defensible because that person is actually making the decision with a cart in hand. But if the goal shifts, say, to understanding how carts get built on a shared Amazon account with multiple household members contributing, then the same screening logic may exclude key influencers. Tara's advice is to keep asking whether the screening assumption is still true for this specific study objective, rather than applying a default rule across all studies.

Clients sometimes want to dismiss qualitative participants who do not fit their expected profile. What is the right way to handle that?

Tara's position is to stay with the participant unless they are genuinely disruptive to the session. Wanting to remove someone because they do not match the assumed image of the target consumer is itself a form of confirmation bias. She gave an example of a low-income consumer who meticulously collected abandoned shopping cart quarters but only bought Apple computers, saving for two years to afford one. Had that session been an Apple study, a client might have dismissed her as the wrong respondent and missed an insight about the perceived cost of switching operating systems. Unexpected participants often surface the most useful signals.

Do you have an example from healthcare where focusing only on the provider underestimates the role of the patient in treatment decisions?

Tara noted that healthcare has multiple layers of influence: formulary coverage decisions, physicians, nurses, patients, and caregivers. Companies that research only physicians get a tidy picture of protocol and treatment plans that often diverges sharply from what patients actually do. Patient compliance studies frequently reveal a large gap between what the doctor believes is happening and the patient's actual understanding or adherence. When in-market results do not track to strategy, she recommends asking whether an influencer in the ecosystem was missed, not just whether the physician-facing message is working.

Do you ever use projective techniques to surface unconscious bias?

Yes. Tara sees projective techniques, image elicitation, shop-alongs, photograph sorts, and storytelling exercises, as tools specifically designed to get around the rational self-presentation that direct questioning tends to produce. When respondents pick an image rather than answer a direct question, they have less sense of what the correct answer should be and are more likely to surface emotional, habitual, or irrational drivers. She also noted that some of these approaches can be adapted for quantitative surveys using technology that allows image sorts or open-ended visual prompts. She recommended the book Moderating to the Max as a practical reference for projective technique options.

Is writing out stakeholder hypotheses before a study useful when working with clients?

Tara strongly agreed. She said asking clients what they expect to find, without necessarily using the word hypothesis, helps surface the assumptions they are bringing to the study. Even framing it as 'what do you think the answer will be?' prompts useful reflection. She applies the same practice to her own team, from junior analysts to senior researchers, arguing that no one is immune to ingoing assumptions and that making them explicit is the most reliable way to keep them from distorting the analysis.

How Logic Fails Us: Confirmation Bias in Research and Insights

Terah Weese walks through the mechanics of confirmation bias and its adjacent fallacies, showing how easily they surface in everyday research design, analysis, and stakeholder conversations. She provides concrete examples from CPG, healthcare, and qualitative fieldwork, then offers practical steps researchers can take to stay objective. The talk closes with a discussion of how AI and social media are amplifying these risks in ways that are harder for researchers to detect and control.

Key Takeaways

Name your hypothesis out loud before every study. Writing down what you expect to find makes unconscious assumptions visible and easier to challenge throughout the research process.
Reframe the research goal as testing a hypothesis, not proving one. The null hypothesis is that the relationship or difference does not exist; your job is to look for evidence on both sides.
Screen assumptions in study design before fielding. The CPG example shows that a single unchallenged age cutoff (under 45 only) would have hidden that people over 45 represent 14% of category consumption and 24% of the competitive frame.
Survey construction can bake in the answer. A snack motivation question stacked with health attributes will overstate health as a driver simply because non-health options like habit, nostalgia, or fun were never offered.
AI does not automatically correct for bias and introduces new risks. Models can magnify errors in training data, hallucinate facts with apparent authority, and produce outputs that exclude groups or oversimplify, often in a black box researchers cannot easily audit.
Projective and observational techniques reduce response and observer bias. Methods like image elicitation, shop-alongs, and storytelling give respondents space to surface non-rational, habitual, and emotional drivers that direct questioning tends to suppress.

Questions & Answers

For high-consideration durable goods, do consumers actually think more carefully before buying, versus low-cost packaged goods?: Yes, meaningfully so. The milliseconds decision Tara described at the grocery shelf applies to low-cost, low-risk items like ketchup or a Pop-Tart. For a car, a washing machine, or even a lipstick, the investment level and the cost of a bad choice are much higher, so more deliberate funnel stages, including awareness, consideration, and trial, are genuinely relevant. The caution is against applying a funnel framework universally without asking whether it fits the specific category and consumer.
Screening for primary household purchasers or decision-makers can skew gender or demographic composition. How should agency researchers raise this with clients without putting them on the defensive?: It depends on what the study is trying to learn. If the goal is to understand in-store shelf behavior, talking to the primary household purchaser is defensible because that person is actually making the decision with a cart in hand. But if the goal shifts, say, to understanding how carts get built on a shared Amazon account with multiple household members contributing, then the same screening logic may exclude key influencers. Tara's advice is to keep asking whether the screening assumption is still true for this specific study objective, rather than applying a default rule across all studies.
Clients sometimes want to dismiss qualitative participants who do not fit their expected profile. What is the right way to handle that?: Tara's position is to stay with the participant unless they are genuinely disruptive to the session. Wanting to remove someone because they do not match the assumed image of the target consumer is itself a form of confirmation bias. She gave an example of a low-income consumer who meticulously collected abandoned shopping cart quarters but only bought Apple computers, saving for two years to afford one. Had that session been an Apple study, a client might have dismissed her as the wrong respondent and missed an insight about the perceived cost of switching operating systems. Unexpected participants often surface the most useful signals.
Do you have an example from healthcare where focusing only on the provider underestimates the role of the patient in treatment decisions?: Tara noted that healthcare has multiple layers of influence: formulary coverage decisions, physicians, nurses, patients, and caregivers. Companies that research only physicians get a tidy picture of protocol and treatment plans that often diverges sharply from what patients actually do. Patient compliance studies frequently reveal a large gap between what the doctor believes is happening and the patient's actual understanding or adherence. When in-market results do not track to strategy, she recommends asking whether an influencer in the ecosystem was missed, not just whether the physician-facing message is working.
Do you ever use projective techniques to surface unconscious bias?: Yes. Tara sees projective techniques, image elicitation, shop-alongs, photograph sorts, and storytelling exercises, as tools specifically designed to get around the rational self-presentation that direct questioning tends to produce. When respondents pick an image rather than answer a direct question, they have less sense of what the correct answer should be and are more likely to surface emotional, habitual, or irrational drivers. She also noted that some of these approaches can be adapted for quantitative surveys using technology that allows image sorts or open-ended visual prompts. She recommended the book Moderating to the Max as a practical reference for projective technique options.
Is writing out stakeholder hypotheses before a study useful when working with clients?: Tara strongly agreed. She said asking clients what they expect to find, without necessarily using the word hypothesis, helps surface the assumptions they are bringing to the study. Even framing it as 'what do you think the answer will be?' prompts useful reflection. She applies the same practice to her own team, from junior analysts to senior researchers, arguing that no one is immune to ingoing assumptions and that making them explicit is the most reliable way to keep them from distorting the analysis.

Session Notes

Why Confirmation Bias Deserves Ongoing Attention

Terah Weese opened by grounding the topic in the null hypothesis, a concept she had long associated with statistics but has come to see as a practical thinking tool for every stage of research. The core risk is that researchers, often without realizing it, structure studies to confirm what they already believe rather than to test whether it is true. She argued this risk is growing as the sources of data and the tools used to analyze them become more varied and less transparent.

Three Adjacent Forms of Bias

Selection Bias

Any error introduced through how you represent the population you are studying. Terah shared a live example: a study about a product traditionally bought by parents was screened to exclude anyone over age 45, on the assumption the product was irrelevant to older consumers. When she checked existing data, she found that people over 45 account for 14% of category consumption and 24% of the competitive frame the team was trying to source volume from. Leaving those respondents out would have invisibly confirmed a narrow view of who the consumer is.

Observer Bias

When your ingoing expectations unconsciously filter what you notice in the data. A survey asking what motivated a snack purchase listed almost entirely health-related attributes, reflecting the team's health-focused lens, with almost no representation of habit, nostalgia, fun, or reward. The result would overstate health as a driver not because it is dominant in reality but because the question never made room for anything else.

Response Bias

Less about respondents lying and more about how research design creates pressure to give the socially acceptable answer. Two examples illustrated this: people consistently say they would consider a store brand when asked directly, but market share and shelf behavior tell a different story. Similarly, fewer than 10% of shoppers turn a package over to read the nutrition panel in an observed study, even though the vast majority claim they do when asked. The gap between claimed and actual behavior tends to be widest when the category team is deeply invested in the product and assumes consumers are equally engaged.

Confirmation Bias: The Central Problem

Confirmation bias is the pattern of seeking data that supports a hypothesis and dismissing, reducing, or simply not noticing data that contradicts it. Terah described it as a sneaky, often hidden risk because researchers are frequently not personally aware it is happening.

Practical Steps to Reduce Confirmation Bias

Name your hypothesis. Ask your team, before fieldwork begins, what they think the outcome will be. Articulating assumptions out loud, even informally, makes them easier to monitor and challenge during analysis.
Reframe the objective as testing, not proving. The goal is to find evidence that the hypothesis is true, not to collect data that confirms it. The starting assumption should be that the relationship or difference does not exist, and you are trying to see whether you can refute that.
Watch for logical fallacies. Bias often enters through recognizable thinking traps rather than through deliberate distortion.

Logical Fallacies to Watch For

Jumping to conclusions (hasty generalization). Drawing a large conclusion from a single data point, one study, one respondent, one market observation. Treat early signals as hypotheses to test at scale, not as confirmed truths.
Appeals to authority. Accepting a finding because a consultant, thought leader, or prestigious firm said it, without examining their source or methodology. Experts can carry the same biases and fallacies as anyone else.
Bandwagon effect. Assuming something is true because everyone is talking about it. The protein trend was offered as an example: pervasive attention to protein can make it feel universal without the data to back that up.
Post hoc fallacy. Assuming that because A preceded B, A caused B. Spurious correlation charts (such as NHL goal assists appearing to track consumer sentiment toward a pharmacy chain) illustrate how two unrelated data series can look tightly linked.
False cause fallacy. A close relative of post hoc reasoning: assuming that because two things are correlated, one is causing the other, sometimes even when statistical correlation exists.
Anecdotal evidence. Using personal experience as proof of a broader truth. Terah noted this is easy to spot when a stakeholder does it but requires equal vigilance when researchers themselves use their own experience to infer how consumers think.
Wishful thinking. Organizational investment in a product or idea creates pressure to find data that keeps it alive. The longer a team has worked on something, the harder it is to be the voice that questions whether the evidence actually holds.
Oversimplification. Human behavior is rarely driven by one factor. Distilling complex, interrelated drivers to a single slide point is useful communication, but it risks creating a false rule: do X and Y will follow.

AI and the Amplification of Bias

Terah described AI as an unavoidable force in research and analysis, noting that analysts and researchers are consistently listed among the roles most affected. The concern is not that AI is useless but that the bias-control skills researchers have developed over careers are not automatically replicated by machine learning systems, and the analysis often happens in a black box where human review is limited.

Errors in training data can compound through the model, producing results that are neither replicable nor representative of the full population.
AI can hallucinate facts and present them with the kind of authority that triggers the appeal-to-authority fallacy, especially among stakeholders who are inclined to trust AI outputs at face value.
When analysis happens inside an agent, researchers lose direct visibility into where a logical fallacy or exclusion may have occurred.
Social media algorithms are a related risk: they are optimized for engagement, not proportional representation, which means polarizing and emotionally triggering content gets amplified. Social feeds also function as self-reinforcing loops, showing users more of what they already believe and reducing exposure to contradictory evidence.

Social Listening and Review Data

Terah flagged over-reliance on social listening and consumer reviews as a specific application of these risks. Vocal minorities, both enthusiasts and critics, tend to dominate these data sources. Algorithms further skew perception by prioritizing what generates engagement. Using these inputs without accounting for their structural biases can lead to overestimating the size or importance of particular opinions.

Transcript

Read the full transcript

Hello everyone. We are about to begin again with our next speaker. Um I would love and happy to introduce you all to Terah Weese. She is going to be presenting for us. But before we dive into her presentation, just want to make sure that we uh you know all understand the ground rules. Um of course technology always loves to play tricks on us um and can create all kinds of problems. So we are at its mercy. Hopefully everything goes uh smoothly but just in case I want to put out that that disclaimer. Um obviously in research sometimes we attack the same problems from different perspectives. So make sure that you are being respectful in any questions or engagement that you have with one another. And then obviously we want you to engage. Use the chat as a great opportunity to ask questions, engage with one another. And then of course go to LinkedIn, find new connections. You never know uh when a new connection might be made. So without further ado, I would love to introduce Terah. She will be presenting um on how logic fails us confirmation bias. So I will turn it over to her. Thank you very much. Great Molly. How do I sound? Good. Thumbs up. Awesome. Okay. So, let's make sure I can get my slides up here. And they're looking good. Are you able to share your screen, Terah? >> Yep. I thought I was, but it it says it's screen sharing. It doesn't look like you're seeing it, though. >> There it is. It was because I was still sharing. You're all set. Okay. Um, so if my slides are up, what I'd like to do is just kind of ground us in how I got to this topic starting out. Um, for me, confirmation bias is sort of a a passion point that I come to again and again with different members of my team because I feel like it's um something that actually comes up more recently than it did once upon a time. And whoops. Okay, that's not what we wanted. Hold please while you look at my very unattractive screen. Ollie, I think you jinxed me. Um, >> I'm sorry. >> No, that's okay. My PowerPoint just crashed. You know, as >> because we gave the whole preface about technology, >> right? Let's try this again. That was a very um fascinating table of numbers and I'm sure you're all wrapped by the details there. Um I'm screen sharing. >> I see it. >> Okay, awesome. We're going to make a second attempt here, guys. So, um confirmation bias I feel like is a really important foundation and it's just a good reminder because I think it's coming up more and more as the data that we're exposed to and the sources of our information change. It's something that's really important to come top of mind as we're thinking about the approach to answering the business questions. Uh so, you already did my intro. I won't belabor that. I'll get right into why I think this is such an important topic for us to be talking about as research and insights professionals. And it's because for me the null hypothesis for a long time was this concept that I learned in school that was about stats testing and sort of a technical detail of how statistics are built. And I think what's renewed my interest in thinking about the work that we do in this way is I think thinking about the null hypothes null hypothesis in each part of our research makes us remind ourselves to question um what we believe to be true and recognize the bias that we might be bringing into the research that we're doing. Because in the end, what we're here to do is prove with objective information that there is a relationship, that there is a difference, there's meaning in this information. And when we introduce bias, it can sometimes cloud the clarity of that outcome. So, um I think bias and confirmation bias in particular is sort of a sneaky um almost hidden risk in a lot of the research that we do because uh often we're not personally aware of our own bias. And I think one of the things that really brought this home to me is when I was at the Reva Institute learning to be a moderator and you really just live that journey over and over again. When you interview consumers uh face to face, you have to come face to face with the the assumptions you're making about people, the assumptions you made about the way the conversation was going to go. And it is really foundational to being a great moderator, being able to recognize what what you're bringing into the conversation from your own humanity in terms of the assumptions and the background and the biases that you bring. And I think as researchers, we are better when we're forcing ourselves to be consciously aware of some of these things. It opens us up to being more objective and it opens us up to thinking about how we research problems in a fresh new light. And so, you know, bias is our enemy as researchers. We don't want bias in our data. We don't want anything that would call into question our conclusions, make them invalid, make them unreliable, uh, or make them something we can't replicate again in the future. That was a one-time effect that we don't feel like is really real. And so, I'm going to go through a couple of different, I would say, adjacent forms of bias. Confirmation bias to me is the central idea of this presentation. in the idea that um it's very easy to mistakenly structure research in a way that confirms what you already believe to be true and I think often our research partners uh are you know crossunctional teams can sometimes encourage us to go say you know tell us why this is a good idea or uh you're you're vested as a team that you've taken an idea so far and you you kind of want it to continue to succeed so bias creeps in really easily um And there's a couple different types of bias that I think are really related to this idea of a bias that would help you um sort of confirm without really challenging your beliefs. And one of them is selection bias. Selection bias in its purest sense is any sort of error that's introduced through the way you represent the population you're studying. And this is a real life example of something that happened just in the last few weeks in my team where it feels like a very innocuous question. It's a standard screening question that asks how old people are and qualifies or disqualifies them from the study based on the age that they are. And you'll notice here that uh we the choice the the recommendation that I was given was that we were going to terminate anyone over the age of 45. And I think this was just a really great example of how bias can so easily sneak in because this was a study about a product that's traditionally thought of as something that parents buy for small children. So it was an assumption that the team made that by the time you're 45 to 65, this product's probably not relevant to you anymore. We don't really need to talk to those per people, certainly not anyone over the age of 66. and me trying to identify any potential way for us to be making assumptions that may or may not be true. I said, "Does that make sense? Do we know that that's true?" And I looked to the data that we had elsewhere and actually uncovered somewhat surprisingly to the team's um learning that they didn't realize this was true. 14% of the category, well, relatively a small amount, still more than they would have expected, um is being consumed by people over the age of 45. And probably even more problematic had we excluded anyone over the age of 45. Uh 24% of the competitive frame of where they're trying to source volume from is consumed by people over the age of 45. So, it's just a real life example of how this selection bias is interrelated with confirmation bias because had we left out all of the older people, then the learnings would have confirmed in some way that what we're doing is only relevant to maybe parents under the age of 40. And so it's just a really interesting example of how it might have um gotten missed but it it really is impactful because in the end people over the age of 40 are a huge growing cohort and we might have missed an entire opportunity with that older group. Uh another example of an adjacent bias to confirmation bias is observer bias. And I think this is the the classic qualitative moderator example where as researchers, our brains are just like any other human brain. We have a little bit of a filtering going on. So much stimulus your brain sort of intakes and recognizes information that it thinks you think is relevant. And observer bias is when your expectations unconsciously influence your collection of the data. what you notice, what you pick up on, what you see, what catches your attention, what seems meaningful is being influenced by the bias that you're bringing into the analysis. Any sort of ingoing expectations that you have can filter unconsciously what you're noticing. And so this is a step where in order to control for obser observer bias, you go back to that broader application of the concept of the null hypothes null hypothesis and ask yourself, is this proving what I believe or is there something here that's disproving what I believe? Am I noticing a pattern that reinforces what I believe? Can I see any sort of pattern that would call into question what I believe? It's almost challenging yourself to prove to yourself that what you believe is true and not allowing yourself to fall into that trap of of just noticing the things that reinforce what you want or already think and missing something that might be contradictory. And a again live example recently just in the last week or so um where I came across another survey and the question was which of the following motivated you to buy this snack seemingly innoculous question has a nice long list of things. I think the miss here the observer bias here was they were so focused on trying to make sure all of the attributes that they were interested in were in the list. What they didn't miss is that only 10% of the choices in this category are actually health motivated and everything on this list by and large outside of maybe packaging and price and flavor is very health motivated. So the question construct itself is biasing because it's so heavily weighted toward health motivations and doesn't really have a balance of other non-healthrelated motivations. And so the learning, you know, consumers will answer this question. They'll give us feedback, but it may perpetuate this idea that health is more important than it actually is because we didn't ask the question in a balanced way. So I think had we put in attributes like habits or nostalgia or fun and reward, it would have created a little bit more balance to what we know is true. Um, but again, it's because this team is very health focused. The brands they're working on is very are very health oriented. And so that's that's sort of the the ingoing hypothesis that they automatically have is that everyone's very health motivated when they're choosing those products. And I think that's just another example of how this observer bias sneaks in in unexpected ways. And then response bias. And I know we talk about this as an industry a lot, especially lately when there's so much concern over sample quality and knowing that your respondents are actually who they say they are and representing um the true people you want to talk to. And response bias can be that it can be that people aren't giving you honest answers. But I find more often than people directly lying to you. I think a lot of times the way we construct research sort of sets up the situation where consumers want to or feel they should rationalize their behavior. They're going to explain to you what they think motivated them or they're going to explain to you what they think the process they went through to make a decision is uh or what the right answer should be, especially in person. Often there's a pressure, societal pressure to to sort of answer correctly and be a good parent or be a good person or be healthy or whatever the case may be. And so sometimes we're introducing bias in just even the methodology that we choose and the way that we set up the survey and um the sort of expectation that then the respondent reads out of maybe what we're looking for or what we're hoping to hear from them and their desire to give us what we want essentially. Um so these are some examples of how I've seen that come up. Again, a lot of studies going on these days, so lots of examples to draw from. Uh where a seemingly simple question like would you be willing to pay more for a national brand than a store brand seems very straightforward yes or no question. But the thing here is what we find time and again is that people believe that they should buy the best value and so of course the store brand should be considered. But in reality, the claimed behavior we'll get out of a question like this does not match the reality of market share or the behavior that would actually happen in shelf and an observational study. So it's just a caution again to that idea of how can we set up questions in a way that enable people to answer as accurately as possible and aren't confirming some sort of bias or assumption that we've made about how people make decisions or how rational even their decisions are. Um, and then like I was mentioning earlier, I think looking at an example when u we ask people to explain how they go about doing a process that may be only semi- rational or conscious. And a lot of times if you've ever been on a shopalong study in a grocery store, I work in CPG so we do a lot of that type of research. What you find is that um if you just observe someone without intervening, their purchase decision is seconds, maybe fractions of seconds. It's very automated, very um habitualized. And then if you stop and talk to them and start to to try to explore what's motivating it, they'll give you answers about what they were thinking about. But at some point there may be a little bit of bias being introduced there about what they want to believe their decision- making was or what they think they were thinking that's not really real. So um in this case again fairly straightforward question that asks were you looking at the ingredients on the side? Were you comparing prices? Were you looking for deals? All the things that people might be doing uh or none of those. And I think one of the things we continue to learn that really seems to shock people is that when you ask a question like this, people will say, "Of course, I read the ingredients. Of course, I'm looking at the sugar, the fiber, the protein." Um, but the actual behavior of that in an observed study is just fractions of the claimed level. Less than 10% of people will actually turn a box over and look at the side panel in real life versus how much they would tell you that they do that behavior. So again, a form of bias that may be a little bit built in by our assumption that people are thinking about our categories and products way more than they really are because we live it day in and day out and so we're really close to it and thinking about it all the time. And it brings this kind of expectation that our consumers also are are much more engaged um when in reality they actually may not be. Which leads me to the real meat of what we want to talk about today which is confirmation bias. And this is a form of bias recognized in the research that says when you are looking for data that supports your hypothesis and you accept that data and when anything that's contradictory to your ingoing assumptions or bias is maybe dismissed or reduced or ignored or even just not noticed like I was saying earlier in terms of how sometimes you're just not even conscious that you're you're not really seeing that. And so what are ways to avoid this sort of confirmation bias? What do we do about this? I wanted to give you some practical tips about how you can avoid this slippery slope of confirmation bias. The first one is name your hypothesis. It seems kind of silly and basic, but the actual act that I've asked my team to do every time is to say, what do you think the outcome of this will be? What do you think the answer will be? And sort of like call your bet in a way. Because then what you're doing is you're saying out loud, I'm recognizing that I have some sort of inkling that I think it's this or I think it's going to go this way or I think this is what's going to happen or how this works. And by naming it out loud, you can be more conscious about looking objectively for data that does support that. And also making sure you're poking around that there isn't any data that actually refutes it and proves it wrong. So, it's almost like taking something that's a little bit of a nebulous concept and writing it down and and articulating it so that you recognize and it's a little bit more conscious and front of mind. And then remember that you're there to test it. It's not about finding information to prove it correct. It's about testing to see if it does in effect exist. So, that's a reframing of the question. It's not I'm going out to find out why people make healthy health healthy decisions. It's I'm going out to find out. I think they make healthy decisions. That's my that's my hypothesis. People want to be healthy. Of course they do. Um but I'm going to go out looking to see can I find evidence to prove that people are looking to be healthy. So you're trying to prove that your hypothesis is true. Um and that it does exist, but your assumption, you have to kind of reframe your mind to think the assumption is that it that it's not real. The null hypothesis is that there isn't a difference or that that's not a driving factor. and you're trying to look at both sides of that coin more actively. Uh, and then look look out for logical fallacies because this is really the tricky part. And the first two, it's like, okay, I can do that. I can name my hypothesis. I can make sure that I'm looking at both sides of the coin and and looking at is it really true? Can I defend it? Is there any evidence against it? Um, but logical fallacies can be really tricky traps that you fall in that can trip you up along the way. Um, these are kind of where the unconscious bias comes in. So, I'm going to walk through some of those uh logical fallacies. Maybe some of them you've heard of before, maybe some of them are new, but I think sometimes just giving it a name makes it a little easier to remember and watch out for. So, logical fallacy number one, jumping to conclusions. Uh, this is something we say, it's it's a phrase you've heard surely. Uh, it's a hasty generalization. It's when you draw a very big conclusion from very little evidence. And this is really easy for teams to do. Um they get one study, they hear one respondent in a in a focus group. Um they see a product performing in the marketplace and they take that one point of data and they say okay now this giant concept is true. This is this is successful. This is what people are looking for and therefore it's a universal law and it's true. And I'm sure many of you can think of examples where this has happened before. But this is a case where you say, "Okay, it was true in one case, in one situation. One one person said it. Let's go see. Let's test if that's actually a thing that's bigger and has scale and we can put data behind because you're you're concerned or looking to see if it's an anomaly. Is sort of an outlier? Is it a one-off?" And so we want to really be careful to not overblow small information and jump to conclusions that are bigger than the data warrants. Number two, appeals to authority. This one happens all the time. It's uh something's true because an expert said it. We hired a consultant. Uh a bigname uh thought leader said it. It came from a really fancy company that everyone respects. There's lots of ways that this can show up in our day-to-day life, but it's where we sort of just taken information that's presented to us or published by a source or said by anybody in authority and we're just like, well, they said it's true, so it's true. And people take it at face value. And I think what we need to do, and I'll get to this in a little bit, especially in a day and age where AI is such a big influence over the information collection, information reporting, is you really got to question what's their source? Where do they get that information from? Can I see the reference? Can I see the information? Um, can I see the detail? Uh, because maybe the expert is also jumping to a conclusion or maybe the expert also has a bias. So just because they're an expert doesn't mean that they're not also potentially um vulnerable to these same biases and logical fallacies. The bandwagon, I mean any corporate environment has a definite bandwagon effect, but the logical fallacy of bandwagon effect is really about when you just kind of get caught up in the energy of everything else. Uh it's caught up in the group. So if in our world right now, protein, everybody's talking about protein, you're seeing it everywhere, everyone's innovating on it. So it just starts to gain this momentum in life of its own where um all of a sudden people want protein in everything because we're putting protein in everything and everyone's talking about protein. And so we're all just jumping in the bandwagon, right? Um, and that's not to say that it's not true and it's not real, but again, I think this sometimes can lead to uh just assuming something to be true and accepting it is true without really interrogating if you have confidence in the data behind it. Uh, and just taking it at face value with inter without interrogation. So, bandwagon fallacy definitely one to look out for. And then the classic chicken in the egg, the post hawk fallacy, the because A happened before B, uh, certainly A cause B, you know, spurious correlation, that type of a thing where you say, I I'm assuming there's a relationship between these things, but I didn't really interrogate and test to confirm that that relationship is real and is moving in the direction that I think it is, that something else isn't an influence or that it's step three in a process and I'm missing the first two steps. Um, so this post hawk analysis is really just o a different form of oversimplification where you're you're jumping to a conclusion and a logical fallacy that says there's a simplified relationship that may or may not be the whole relationship. And then I love any I don't know if anyone else here has seen these sperious correlation charts, but I just find them so comical. Um, so the references are in in the deck and on screen, so you can go find some of them. Super fun. Um, but they basically what they do is take two completely unrelated data sets and show how they can look like they're completely related based on them following the same pattern. So this is a great example of false false cause fallacy. And the idea that because two things are correlated, because they move together, whether you did the statistics or not, honestly, a lot of people anecdotally are just like, well, this is growing while that's growing. So clearly they're being driven by the same thing in in a less than statistically scientific way. But even sometimes statistical correlation can lead to this kind of fallacy where you believe that because they're correlated, they must be causing one's causing the other. And it this is just a great example of how NHL goal assists clearly has a lot to do with how people feel about CVS. I mean logically. So um but this the the bars look like they moved the lines look like they moved together. So um it's kind of a funny example but there are many real life examples of this and it's just another opportunity for us to go back to that thinking of saying okay what are we assuming to be true? What data do I have that really supports that? And can I refute the null hypothesis? Can I prove that there is an actual meaningful relationship or am I just kind of jumping to this false cause and falling into this logical trap? And then anecdotal evidence. I this one really probably doesn't need an explanation because we've all been in a meeting, I'm sure, at some point when somebody says, "But in my house or my sister or my mom or whatever it is." Um, and they use their own personal experience as a proof point for something being real. Uh, so I think anecdotal evidence when someone else is saying it is a little easier to recognize as a researcher is like that's that's great. That's one person end of one. Okay. But it's also a great reminder for us not to do this for ourselves as the researchers. Not to say because I know how this works for me, I know how this works for everyone else. Um because that's another form of this, right? You're saying, well, because my experience is that this this is the way I think about it or this is the way I make that decision or this is the process I work through can become a form of bias that you're bringing into your analysis and you're looking for other people to be like you. by nature of of how humans work, that's very common. So, we have to continue to question ourselves and say, "Is everyone else like me? Are they thinking about it the same way that I would think about it?" And just being really aware, front of mind, of that potential fallacy you could fall into that could lead to confirmation bias. Uh, wishful thinking. Wishful thinking, that's a great one because, uh, in business, I think this is a really, really easy one to fall into. uh where the confirmation bias is pressurized and is driven by this idea that we as an organization want this to succeed. This is our baby. This is our idea. We spent so much time, developing it, researching it, thinking about it, um building it as a brand or a product or whatever it is. And it's almost so easy to go hunting for data to continue giving us hope and to continue reinforcing what we want to be true. And it's really organizationally hard to be that one voice that says, "Hey guys, actually, it's easier as an outsider, but when you're deep in the organization, it's a lot harder to say, let's take a step back. Is this really is this really working? Did we really see the whole picture? Is there something we've missed? um is there another point of view that contradicts what we want to be true? Uh it's it's a challenging thing to navigate, but I think it's an important thing for us to continue to remind ourselves because otherwise we're we're falling right into the depths of confirmation bias. We're just saying because I want it to be true, I'm going to look for information that reinforces what I believe. I'm going to try to ignore, dismiss, discount, diminish any information that would contradict it. um oversimplification. This is you know generally like taking a really complex basically anything with humans is a pretty complex um system of how things work. Generally I found it's it's very rare that you can be like people do this because of this one thing and it's just that simple. It's usually um a complex system of interrelated influences or interrelated drivers and business people want you to distill it onto one pretty PowerPoint slide and make it as simple as possible which is great and an important part of our job but um there is a a risk there. You oversimplify to the point where you miss the detail. That's where the real nuance is. And you create a bias where people think as long as I do X, Y will happen. As long as we deliver this one thing, this is what's going to happen next. And you've oversimplified to the point that you've injected some some confirmation bias into that learning or into the the key takeaways for the business. So, I think this has always been true. It's always existed as a as a concern in our industry or as something we're all trying to be more and more aware of. And I think the new challenge for me that's been unlocked in the last five to six years is the increasing use of machine learning and AI um has introduced even more potential for these biases to be outside of our direct influence and control. Um, the logical traps can be happening in teams that are analyzing data for themselves without an insights person directly in the mix or an AI agent can introduce their own bias in the same way that a researcher could, but now we're a step removed because we're trying to identify the the machine's bias in addition to our own bias. Um, so I thought it was interesting and I don't expect you to read all of this content, but for a discussion, um, that when we think about the industries that are predicted to be most affected by AI, I find research and and analysts are always on the list. This is just one particular source, but there are many sources that point to that same outcome. It makes sense. You know, it AI is very efficient in analyzing a bunch of data, and that's a big part of what we do. So I think um it's an opportunity that's unescapable in many ways. But what that means is all of the training that we've gone through to learn how to construct studies in a way to not have bias to be aware of our tendency to have bias to think about bias in the way that we set up our analysis in the way that we draw conclusions. Now, we're handing a lot of that over to a machine who may or may not actually be I don't know who is the right way to phrase it, but they they may not be programmed. They not may not be o able to control for bias in the same way that we had been as human analysts before. And so, when I think about the influence of AI in the desire to minimize confirmation bias, I think there's a couple of really important risks we need to keep in mind. Um, we know we know for a fact that AI models have a tendency to have like a snowball effect where if there's error in the the source data, it can get magnified within the analysis. If it's in the training data, if it's somewhere in the foundations of the analysis, it can result in this bias. And the bias could come in the results not being again replicatable, valid, reliable, but also it can come in ways that would exclude groups of people. It could bias the data against certain consumer types or certain outcomes. So we really have to think about how bias can be embedded in the data and the analysis based on the way machine learning works. And then we also recognize that AI uh can and does sometimes hallucinate and have inaccurate data and it'll it'll own it when you ask it, but there could be false facts. They can pres present them very convincingly. It comes across with authority like we talked about earlier as a as a potential logical fallacy is trust trust the expert. AI, you know, AI is so smart. It did all this analysis and I'm just going to take the answer at face value. Um so this is another place where you can see some of those uh oversimplifications. Some of the the tendency to jump to conclusions could come into the analysis through the way the AI tends to maybe put a little art in its interpretation or hallucinate or misrepresent the information. And then there's this black box. And this is what I'm talking about where if it were 10 years ago, I would say I can spend time with my team. I can practice this. We can train this. They can um they can learn to be better and reduce the risk of introducing bias into their research. But now if the research or the analysis is happening sort of in this black box, if you will, inside of the agent or the machine, it's a little harder for us to see where those things might have happened, right? because we're not in the depths of the data to say actually I saw some things that refuted that or that doesn't really jive and and make sense. Um so sometimes it's it's harder for us to pressure test and identify some of these potential logical fallacies or biases that are coming in. And then this goes back to sort of the expert value this over reliance that sometimes um parts of the organization may just overrust this information. they don't do their own human review. There's not enough human in the loop, if you will, in the analysis and therefore we're missing we're missing the opportunity to identify sources of bias. Um, another place I feel like this has come up a lot recently is when we get into social listening or even over reliance on using consumer reviews as a point of data and um the analytics that are built off of social media listening and reviews as data sources. What we've always known to be true for any company that's had like a consumer call center back in the day when people called on the phone uh is that it's not representative, right? The people who talk on social media uh about topics can be kind of the polarization of the people who really love it and the people are really angry about it. Um there can be a very loud minority and so it can kind of swing uh your perception of how big or small or how important issues or points of view or reflections are. And then the algorithm can also skew the perception of that information because algorithms are motivated by what serves the platform not necessarily by representing proportional reality. Right? So engines and AI can uh with the algorithms prioritize things that people are engaging with more engaging with longer that might be the more controversial the more emotionally triggering things. they can really amplify more polarizing and uh sensational material and make it feel like maybe it's bigger than it is, which makes it easy to jump to a conclusion or maybe oversimplify. Um, and then I think what's really interesting and I'd love to discuss um, because this is kind of the the end of the bulk of my content is the idea that the way that social media and social networks work is people tend to find their people, find their passion areas, find their topics, and um, what they engage with is what they see more of. So then social media in and of itself becomes a little bit of a referential self-reerential reinforcing form of confirmation bias. If this is what you believe and this is what you're into, you're going to see more and more and more of that kind of content. And so it starts to feel like, well, this is what everyone's seeing. This is what everyone's talking about. And um it can help really drive people to lack of recognition of how they're essentially being confirmed in their biases and um a lack of empathy for people who are of a different point of view or are down their own separate rabbit hole of of what they're engaging with and and getting more and more of in their feed. And I think um while that's more of a societal question of how that's affecting our culture and our society at a large, I think it does have implications for business both in the way that we target people, the way that we talk to people and engage with them in media and also just how it translates to then the viewpoints people bring in their own personal viewpoints that they bring in to business meetings and conversations and how confirmation bias can show up in those environments. So, at this point, I would love if anyone has questions or they want to debate and discuss how we handle this this challenge if it's something they're seeing themselves. Um, just wanted to give time for that because I think this is an interesting thing um to talk through as an industry. Well, what other people may be typing. We did get one question kind of towards the beginning of your discussion. Um, we were talking about uh grocery items and like are consumers, you know, really giving attention. Um, and we had a question about like durable goods and do you think that consumers do actually maybe think more about a product that's a durable good as opposed to, you know, produce or things that that expire quickly? >> Absolutely. So my comment about how decisions itself shelf can be so milliseconds probably applies a lot more to catch up than it does to a car obviously. So I think that's where again um there's so many facets of this like I'll see data or sit in meetings where people say well we have information we've seen from our agency that the funnel needs to work like this and that's probably true if you're selling washing machines or maybe even cosmetics um where people do carefully consider and research the things that they're buying first because of the investment level being so much higher right you're dropping a lot more money on a car than a bottle of ketchup and um the risk factor of what if I don't like it or what if it doesn't work for me or what if it's not safe, you know, it's like those are much higher investment decisions in many ways than the typical consumer package goods where it's like, oh, if I don't like it, it was a couple bucks, whatever. I won't buy it again. Um they're not as ruffled by something that dissatisfies them as being stuck with a car you don't like or a washing machine or even a lipstick is kind of annoying, frankly. Um, so I think that is a very big difference and that goes back to just this sort of like stopping and pausing and thinking about yes funnels are great. You know this idea that you first have to make them aware and then you have to come back and communicate what your point of difference is so they'll consider you and then you have to come back and generate a trial. It's like, you know, if I'm selling a a Pop-Tart, um, that probably happens in in a couple of seconds. And frankly, it probably happened when they were like 12 years old the first time. Um, but if I'm selling, you know, a a new computer, that's a whole different ballgame of like multi-touch, funnel, driving, um, communications. And so, we just have to look at that information that comes to us and say, "Okay, this makes sense. I understand what you're saying, but does it apply in my industry? what how would it be different? Um how might this change and evolve given the circumstances of what we know about our consumer in our marketplace? And so it just goes back to not taking things at face value, I guess, and just really reminding ourselves to interrogate um whether that really makes sense or if there's something that we're missing that we haven't thought of. I don't know if that really answers the question fully. >> Yeah. something that you were saying just about like high value items and kind of sparked this other thought in my mind like I often think of representation bias is more like demographic but we have been seeing >> across industries more and more people focusing on like decision makers and primary account holders or they have their payment method saved >> and that ends up skewing gender- wise but it's framed deemed as behavioral. So, how can maybe people on the agency side talk to clients about that without undermining them or, you know, kind of putting them on the spot in a way that, you know, maybe puts them on a defense? >> Yeah. Yeah. It's a it's a great question. I love I love these kind of questions and having these debates in our team because I think the the complicated answer is it somewhat depends, right? So, if I'm doing a study that's for Meyer about how people shop their shelf, then maybe it does make sense, even if it's not gender representative or age representative, to talk to the primary household purchaser who's the actual one with the shopping cart in their hand. You know, that I can argue that that is representative of what we're trying to learn, but it's not a universal law. And I think that's where sometimes we fall into these traps where it's like, well, we always talk to primary household purchasers. you always talk to the card holder or the you know the decision maker whatever the term is for your industry but if I'm in the next study actually trying to understand what motivates someone to engage on a retail media platform do I know that that's still the case that I only need to talk to the primary household purchaser I don't know I think we need to question how do carts get built online if I'm shopping on Amazon is it like my husband and I share an account? Do my kids share the account? Are they sending me links to put things in my cart? Like maybe we need to take a step back and and ask ourselves how do these things get built and what are the points of influence when you're not limited to like who's physically in the store, you know, and it's a an environment where it could be different influencers playing a bigger more direct role and request whether that's still true or not, >> right? I mean, and then kind of building on that, um, I'm sure if we are, if you're in the qualitative world, we've all had this experience where a client is observing and they're like, I don't like this participant. Maybe uh they're older or uh maybe like not as articulate as some other participants had been. Yep. >> How do how do what's what advice I guess do we have around there to ensure that you know we're giving a good participant experience but also not reinforcing biases from our client? >> Yeah, such a hard one. Um I I have a pretty strong point of view on this. may be unpopular, but I feel like um absent absent overt like destructiveness to the research, you know, if there's a person in there that's just insulting other people and being disruptive and whatever, like, okay, there's a point where you got to go. I'm sorry. Like, this is not working. But the tendency to want to remove respondents because they don't fit the expected profile is the definition of what we were just talking about, right? It's like, well, that person is I wanted people who were XYZ. You know, they were supposed to be brand loyalists and they were supposed to be high income or they were supposed to be whatever they were supposed to be. And this person doesn't fit the image of what they think that that was, you know, in their mind of what that person would be like or what they'd be motivated by. That's not a great reason to to boot someone out of research or to exclude them because there's probably something to be learned there. Either they answered the question honestly and misunderstood. Well, then maybe maybe we need to learn about how to ask that question better. Or um maybe they really are high income, but they're super frugal on this one thing. I can remember for example, it's kind of down the lane of what you're asking. We recently were doing some call and we were supposed to be talking to lowinccome consumer like super incompressed not just low earnings but like low disposable income let's say like real incompressed people and this woman was like I walk around the parking lot picking up the quarters that people leave behind if they leave their cart oldie I have to assume leave their cart out there with a quarter in it I'll pull all those carts back and take all those quarters like you're thinking my gosh this woman is really pressed like for a couple quarters she's put in a lot of work, right? And then 2 minutes later, she was like, "I know I just told you all that stuff, but when it comes to computers, I only buy Apple." She's like, "I know. I know. They're so much more expensive." She's like, "But I just I will save for two years to afford an Apple computer because I just don't even want to learn a new operating system." So, that's where I mean, I feel like had that been reversed and it was an Apple research study, people would have been like, "Oh my gosh, why is this woman here? this isn't our consumer. And so it's just like you can't jump to those conclusions about people's motivation. There's so much to be learned by and especially in qual I have a strong opinion of like that one thing somebody says can unlock a whole new opportunity for exploration. Do I take it as fact and run with it just based on one person saying it? No. But what she just articulated was a really interesting insight about the value of the burden of learning a new operating system. like it's a really extreme example of it, right? So, I think I always try to encourage and as a moderator when I was a moderator for years, um I I would stick with people and the people in the back room like, I can't believe you didn't send that guy. Like I was like, no, there's always something to learn from everyone. And I I'm I'm like, I'm happy even if we just learned one thing, it was still worth it, you know? So, I just think it's a matter of reframing uh what's important and helping people through those experiences to recognize that we may be jumping to stereotypes and conclusions and we need to be more empathetic and sensitive. And I guess my last point on that topic is um and this just came up yesterday so I'm still a little sweaty over it is uh um you know most people in our industry we're relatively well educated. we make really good money on the whole versus the rest of the population, right? So, we're we're in a peer group operating with other people who are more like us and there are people in the population who are less like us many cases. And so, I I really find um there's learning opportunities and growth opportunities when you're sitting in a boardroom and people are talking about the Dollar General shopper or the people who buy food at convenience stores and they talk about them in sort of disparaging ways. Um, you the term bubble was used and I was like, "Okay, learning opportunity. Let's stop and talk about this." Because that's you're making an assumption that the only people who go to a convenience store are like some guy who is a construction worker or whatever you mean by that term. I'm not even really sure. And then you're making assumptions about what they care about and what they're motivated by. You don't know. You don't you don't know. That guy could work out at the gym and be the biggest health nut ever. You don't know. So, I think it's just an opportunity for us to to be good researchers and help other people understand those kinds of things. >> Yeah, we we've run into the lowinccome thing and have started kind of approaching it from um like a financial strain perspective because $100,000 a year if you've got six kids and live in San Francisco is a lot different than $100,000 income in Tulsa, Oklahoma. >> Right. Okay, so we've got some other questions, comments. Uh, we've got a question that came in from Lynn. Do you have an example from health care where health care providers are considered representative of the impression value willingness to adopt a therapy that underestimates the role of the healthcare consumer in that decision process? >> Yes, I did work in farm healthcare for a couple of years. So, I have a little bit of experience um in this area. And I think what's really challenging about the healthcare marketplace is um it's almost like you have intermediary buyers, right? You've got you've got the insurance formulary coverage decision maker and what's going to be covered and not covered in tier one and tier two and all these things. And then you've got the primary health care provider, you know, the doctor, the PHP, the specialist, whatever they are. And then you've got maybe even a nurse in the equation that's having a bout of influence over the patient and or the doctor. Um, and then the patient themselves and the patient's caregiver or, you know, like there's a lot of influence points in that decision- making. And I think um it it can become a little bit reductionist when companies want to only talk to the doctors because then they're missing this larger ecosystem of influence and you can go talk to the doctor and the doctor will oversimplify. It's like well this is this is the protocol and this is the drug treatment plan and this is what they're supposed to be doing and and it seems all like oh that's very neat and tidy. Um that's what's going to happen. And then you finally get the money to do like an actual patient study and you find out that there's a giant rift and a gap between what the doctor thinks is happening and the patient's actual compliance or experience or understanding of what's supposed to be happening. Like the doctor said it and the patient did not hear their, you know, Venus and Mars or whatever. Like they're not speaking the same language. They're not understanding each other. And I think a lot of times when when you have a situation where your strategy was built on research and you have confidence in your research, you understand why you came to the strategy you did, but then the in market doesn't track. That's a great opportunity for you to say, did we miss something in this ecosystem? Uh did we miss an influencer? Because maybe it's working for the doctor, but it's breaking down in a different part of the system or a different part of the system is is thinking about it differently. And so um it's challenging because you know not we don't all have the money and the time and all the things to do all the things we want to do obviously but um I do think that beyond the doctor are there even influences over the doctor what conferences are they going to who are they listening to what are they reading what are you know those things so it's just again it's like that one extra question of what else am I assuming to be true that may not be true what could I be missing There's several comments about uh in the chat of people just with really bad experiences with vendors and and like this representation and and participant selection. So, you know, it's crazy that we we all have experienced this and we are still encountering it um constantly. >> Yeah, it's just I mean confirmation bias is human human nature. We're never going to escape it, right? It's just built in as humans and that's why I think it's something to continue to talk about because it seems really simplistic, >> but it's inescapable. It never goes away. Um, you know, the the beter minehoff theory that like you're you're only going to see the things that your brain thinks are important for you to see. Um, it's just like a really great scientific principle that goes back to how easy it is for even the most rigorous researcher to fall into this trap of unconsciously filtering things that don't confirm what they already believe to be true. And I have just found personally that the best defense against that behavior is making yourself super uber aware of what you believe to be true. Like like lay it out. What do I really believe to be true? so that you can be aware that you do have a preconceived notion. And I really thank Naomi Henderson for that training because she was so adamant about it and picky about it. Um she's like, "But in the guide, what have you assumed when you wrote this thing? You made an assumption. You were assuming that something was going to happen. What is that assumption?" And challenge it and make sure that you have a way to check yourself and confirm that that's true. And it's just a really good reminder that I bring with me into into every day. And I just find more and more again with all the things I said about AI, how it just it's almost seems to be more and more important to think about today than it even was 10 years ago. >> Yeah, I really appreciated your discussion on AI. I mean, everyone's talking about AI, but I've not seen as much content talking about how bias um inserts itself. Like I was just using an agent yesterday and every participant became male like despite their name, despite demographics entered. Um I'm reading through the output and every participant was he or his. Um, and so that's the type of stuff that, you know, when you talk about replacing jobs, um, like no, uh, we we do have that human touch that can still bring bring our research to life and and kind of can counteract some of that automated stuff that happens in the background that we may miss. So, >> I really appreciate it. >> Of course. >> Yeah. Any other questions? Oh, I see some coming in. Um, not so much a question, uh, but I love the idea of having your team write out their hypotheses. I try to do this with our business stakeholders when I have the opportunity ahead of a study. >> That's a great great important thing. I I and I have found that often asking clients that allows them to maybe think stronger. But I would love to hear your perspective on that. Yeah, I think um it's it's just we're all busy. We're all under pressure and it's just such an easy thing to say like, "Do I really need this?" I know. I know what we're trying to do. And um it's just I found an invaluable small step to say, "What are you expecting to be true? What do you think the outcome's going to be?" Um what are you expecting to hear? Those kinds of questions. Even if you're not using the technical terminology of like hypothesis, which kind of puts people um a little off at times, it just helps them think through what bias they might be bringing to the analysis. And you know, all the way from your most junior analyst, you know, as they're crunching numbers and building models or whatever they're doing, like do you have what do you think? What would what would be your guess? Do you have an incoming hypothesis? and just reinforcing uh especially in the research team to recognize it and challenge it. And you know, it's it's very sort of high and mighty to say that we could train ourselves to be completely objective and say, "No, I don't have I'm just here to learn. I have no preconceived notions." Like that's ideal, but it's not realistic. I think we all get biased over over the days, over the years. You spend too long in a company, you start to kind of like lose touch with reality in some ways. Um, but at least putting it down on paper forces you to think it through and say, "Do I have a preconceived notion about this? Do I have a stereotype? Do I have any sort of assumption that I need to check and and confirm?" And it just helps us avoid missing things. Like the silly little examples I gave, it's like it would have been easy to be like, "Yeah, yeah, it's a mom for kids snack. Why would I need to talk to 66-year-olds? It doesn't make sense." And then you go poke around and find out, oh actually they make up a pretty sizable amount of the consumption. That's weird. What's going on with that? You know, it's like if I had never asked that question, we never would have connected those dots. >> Yeah. Another question that came in was, do you ever use any type of like projective techniques to help identify these unconscious biases? >> Yeah. I think a lot of the fallacies you probably if you're a quali would have noticed that a lot of the less direct questioning techniques were designed in many ways to address some of these inherent risks of of logical fallacies and bias introduction. So um you know direct questioning can lead to this like oh I think I know the answer this is what I do or this is what I would do or this is how I think about that and those aren't bad but when we over rely on pick lists and choose all that apply and rank these and score these it simplifies the decision- making in a way that it underrepresents that whole like bi system one thinking type of you know a big part of our decisions aren't super rational aren't super thought out are more emotionally driven than logic driven. And so projective techniques, direct observation, these things allow us to give space for that other piece of the equation to come to life because people feel more free to be a little silly and somewhat illogical and more complicated and messy if they're picking a picture that describes how they feel because they don't know what the right answer should be. And so you can have a more interesting conversation with them about that and what it means in a way that enables them to kind of get out of their system to like I know the right answer. Of course I should be looking at how many calories it has obviously. Um and we should I'm not saying we shouldn't, but like there's more to the there's more to the decision than that. Let's be honest. Sometimes you're just really tired or like you have 4 seconds and you just got to pick what's there. and you know like those things don't always come out right away and the projective enables you to kind of like pick up on some of those other signals. So I agree. Yes. Um you know direct observation, photographs, um image elicitation, storytelling, there's so many great techniques um in that area that really help you get around these things. There's a book called Moderating to the Max that I love. is like my little go-to resource of, oh, I need a a sort of projective technique and just flip through there and find, oh, there's some good ones in here. I forgot about that one. So, if you haven't heard of that book, I recommend it's still sold on Amazon. I tell people about it all the time. It's just a really good like here's an example of something you could do. Here's an example you something you can do um that apply. I use them in quantitative studies sometimes even. It's traditionally qualitative, but there's ways with technology that you can have image sorts in surveys and you can do um find an image online that answers this question. There's, you know, some of those tools are available now to us. >> All right. >> Okay. Great. >> Well, I think that was it. Thank you so much for joining us today, Terah. Um and everybody in general, thank you for joining us. Um, as a reminder, after this concludes, uh, we will be having all of these recordings available on our website, so you can come back, uh, listen, uh, pick up anything that you missed or or use it as a good tool going forward. So, thank you everyone again. I appreciate your time and thank you again, Terah, for your lovely insights and great discussion. >> Thanks. Thanks for having me. I really appreciate it. Come by.