What about synthetic data as a solution to the sample quality problem? (raised from audience/chat)

The panel is largely skeptical. Carrie prefers calling it 'modeled human behavior' rather than synthetic data. She believes it may work for observable, publicly recorded behaviors but cannot reliably model attitudinal or private thinking. There is also a finite data supply: models will eventually exhaust the existing survey data they are trained on. The narrow area showing some promise is missing-data imputation to shorten survey length. The core concern is that modeling bad data produces more bad data.

Is there pressure from internal decision-makers to get research turned around so quickly that quality suffers? (raised from chat by Julie)

Yes. Rosanna and Carrie both acknowledged this as a real and ongoing tension. Decision-makers sometimes bypass research teams entirely using DIY tools, producing poorly designed surveys on top of questionable sample. Carrie argued that not every decision genuinely requires quick turnaround, and that researchers need to recover the ability to discern when speed is truly necessary versus when it is an artificial pressure.

The State of Research Sample Quality: A Cross-Industry Panel Discussion

Three research industry veterans, representing a sample provider, a research agency, and a client-side insights leader, discuss the deteriorating state of participant data quality in market research. They explore root causes including private equity ownership structures, technology-driven shortcuts, and lack of transparency in sample sourcing, while sharing practical experiences and cautious optimism about emerging solutions. The conversation spans fraud detection, survey design quality, synthetic data limitations, and the hidden costs of bad data.

Key Takeaways

Four out of five researchers rate the current state of participant quality as poor or worse, and three out of four say quality has declined over the past five years, with some buyers discarding 70 to 80 percent of purchased sample.
Private equity and venture capital now dominate ownership of research and sampling firms, creating a financial incentive to scale and sell quickly rather than invest in data quality innovation, which is an 'innovate and sustain' exercise not a 'grow and sell' one.
Client-side researchers must treat sample quality oversight as a core part of their value proposition, ensuring 'decision-grade research' and having hard internal conversations about the speed-quality-cost tradeoff.
Detecting fraud requires deep subject-matter expertise in the category being studied; junior researchers and clients lacking that knowledge are especially vulnerable to missing subtle data anomalies.
Synthetic data is viewed skeptically by the panelists as an overpromise for most use cases, particularly for attitudinal or future-oriented questions, because it models existing data and risks compounding existing quality problems with bad inputs.
Glimmers of hope include growing industry attention to data quality, the Insights Association's participant Bill of Rights, large clients like P&G setting their own standards, and a potential boomerang back to classic panel-based research.

Questions & Answers

What about synthetic data as a solution to the sample quality problem? (raised from audience/chat): The panel is largely skeptical. Carrie prefers calling it 'modeled human behavior' rather than synthetic data. She believes it may work for observable, publicly recorded behaviors but cannot reliably model attitudinal or private thinking. There is also a finite data supply: models will eventually exhaust the existing survey data they are trained on. The narrow area showing some promise is missing-data imputation to shorten survey length. The core concern is that modeling bad data produces more bad data.
Is there pressure from internal decision-makers to get research turned around so quickly that quality suffers? (raised from chat by Julie): Yes. Rosanna and Carrie both acknowledged this as a real and ongoing tension. Decision-makers sometimes bypass research teams entirely using DIY tools, producing poorly designed surveys on top of questionable sample. Carrie argued that not every decision genuinely requires quick turnaround, and that researchers need to recover the ability to discern when speed is truly necessary versus when it is an artificial pressure.

Session Notes

Setting the Stage: How Bad Is Sample Quality Right Now?

Accelerant ran a poll of several hundred researchers across suppliers, clients, consultants, academics, and UX/CX/MRX practitioners to gauge current sentiment on participant quality.

4 in 5 researchers rated the state of participant quality as poor or worse.
3 in 4 researchers said quality has declined over the past five years.
Year-over-year, quality may have stabilized somewhat, but it is not improving.
Open-ended responses described cleaning exercises removing 25 to 80 percent of purchased completes, with some buyers discarding the majority of what they paid for.

I go to Bojangles, I buy my four-piece chicken tenders, I take them home and I'm going to be able to eat somewhere between one and three of them. In what world does that make sense?

Root Cause: The Ownership Structure of the Research Industry

Carrie Edelstein presented analysis originally developed by SC Consulting and shown at the Insights Association CEO Summit. The data compared industry ownership structures roughly 20 years ago versus 2022.

Twenty years ago, the industry was a mix of public companies, corporate-owned firms, and privately held businesses, with very little private equity and no venture capital.
By 2022, private equity and venture capital had become the dominant ownership model.
Private equity operates on a 3 to 7 year ownership and exit cycle, which is too short to make the long-term investments that data quality requires.
This creates a fundamental misalignment: PE and VC owners are motivated to grow and sell, while data quality demands innovation and sustainability.

Data quality is an innovate and sustain exercise. It's not a grow and sell exercise. And I think there's just a misalignment there with the organizations that own sample companies and what we need out of data quality.

Carrie argued that when sampling companies must spend money on fraud-fighting technology, they are spending on a problem their ownership model created, leaving little left for genuine innovation such as partnerships with HBCUs or other non-traditional recruitment channels.

The Client-Side Perspective: Owning 'Decision-Grade Research'

Rosanna K described how, over 14 years at Procter & Gamble and Duracell, she came to see sample quality oversight as a core part of her value as a strategic insights leader, not just a cost of entry.

The job now includes ensuring the organization is partnering with the right suppliers and that all quality processes are in place before a study launches.
She coined the phrase 'decision-grade research' to describe the standard client-side researchers should be protecting.
Without that stewardship, business partners can easily end up making million-dollar decisions on bad data without knowing it.
The hidden costs of a wrong decision are large, and so is the opportunity cost of researchers spending time on DIY cleanup instead of strategic analysis.

Do you realize how much you would be making decisions on bad data if I weren't sitting in this seat?

The Speed-Quality-Cost Tradeoff and Tech Intimidation

A recurring theme was pressure to deliver research faster, often driven by what Carrie called 'tech intimidation': technology companies implying researchers are missing something when, in fact, the technology itself is falling short.

Tech intimidation manifests as jargon, condescension, and the message that everything needs to be agile and instant.
In reality, not every decision needs to be made quickly. A two-year platform buildout does not require data by Thursday.
Some brands lack the internal bandwidth to act on daily or weekly research outputs, making nightly-fielded studies impractical regardless of data quality.
DIY tools like Survey Monkey lack critical functionality: nested quotas, block randomization, and proper rotation are often absent, introducing structural quality problems on top of sample problems.
Some tech platforms sell to marketing and product teams, bypassing research departments entirely, compounding poor survey design with fraudulent sample.

Detecting Fraud Requires Category Expertise

Carrie shared a concrete case study to illustrate how difficult fraud detection can be without deep domain knowledge.

A study on media consumption suddenly surfaced roughly 200 Native American men aged 18 to 34 with college degrees overnight.
Because Carrie had recently completed work with a Native American media company in Alaska, she knew this demographic is less likely to hold college degrees, not more likely, making the data clearly incorrect.
A junior researcher without that contextual knowledge would likely have celebrated the diversity and moved forward with bad data.
The implication: researchers must build subject-matter expertise before fielding, and agencies and clients must maintain open communication about anomalies without blame.

I've coached my team to assume everything is wrong and then be delighted when it's right.

Quality Beyond Fraud: Engagement, Relevance, and Survey Design

The panel stressed that sample fraud is only one dimension of data quality. Even real, correctly recruited respondents can produce poor data if the research instrument is poorly designed.

Engagement quality: Are real respondents actually reading and thoughtfully answering questions?
Relevance and category fit: Surveys must match how consumers actually think about a category. Rosanna cited the Disposable batteries example: the average consumer thinks about batteries for roughly 30 seconds at the shelf, so a 20-minute brand tracking survey is misaligned with that mental engagement.
Soft-launch discipline: The industry largely abandoned the practice of soft-launching surveys to stress-test question clarity and eliminate correlated items, a quality step that used to be standard in early online research.
Qualitative iteration: When doing focus groups, debriefing and refining guides between sessions is increasingly squeezed out by compressed timelines.

Participant Incentives and the Respect Problem

Bill raised the ethical dimension of how participants are compensated relative to the time they contribute.

Sample can currently be purchased for roughly $2 per complete for a 30-minute survey, far below the US minimum wage of $7.25 per hour.
Only a fraction of that $2 reaches the actual respondent after platform margins.
By contrast, a 30-minute qualitative IDI participant might receive $100.
This disparity is disrespectful to the consumers whose opinions drive billion-dollar business decisions, and it contributes to recruiting low-quality or dishonest respondents.

Skepticism About Synthetic Data

An audience question about synthetic data prompted the panel to share a cautious and largely skeptical view.

Carrie prefers the term 'modeled human behavior' over 'synthetic data,' arguing that the latter dehumanizes the customer and obscures what the technology actually does.
Modeled data can work for observable behaviors recorded publicly online, but it cannot reliably model attitudinal or future-oriented thinking that people do not express publicly.
There is a finite data supply problem: models trained on existing surveys will eventually exhaust their input baseline.
A specific, narrow use case that may show promise is missing-data imputation, such as predicting responses to a few additional questions so that surveys can be shorter.
The broader concern: if the underlying data fed into models is already low quality, modeling it produces more bad data, classic garbage in, garbage out.

I think it's an overpromise that will likely be underdelivered.

The Case for Transparency in Sample Sourcing

The panel called for radical transparency about where sample originates, comparing the current opacity to a time when panels like E-Rewards were explicit about their sourcing from frequent-flyer programs.

Today, account managers at many sample companies genuinely cannot answer where their respondents were originally recruited.
Knowing the source allowed researchers to strategically match panels to study needs, for example choosing E-Rewards when targeting six-figure households interested in travel.
Rosanna suggested data labeling, similar to food labeling, as a practical mechanism for transparency across all data types including modeled datasets.
Bill noted the blockchain concept as an intriguing, if not yet practical, path toward authenticated respondent provenance.
Carrie suggested asking respondents directly in-survey how they were recruited as a low-tech transparency measure.

Glimmers of Hope and What Comes Next

The Insights Association Participant Bill of Rights has been published, establishing minimum standards for the respondent experience.
Large clients like Procter & Gamble are developing their own data quality standards and using purchasing power to enforce them.
Industry conferences such as Sample Con are now actively platforming data quality skeptics who were previously dismissed, signaling a shift in the conversation.
A potential boomerang back to classic panel-based research may occur as the true cost of fraud mitigation makes cheap sample no longer actually cheap.
Accelerant and others are building proprietary panels out of necessity; Accelerant's panel has grown to be one of the larger proprietary panels in existence, which Bill described as concerning rather than encouraging.
The brightest spot remains the real consumer who shares a genuine story, whether quantitative or qualitative, that drives meaningful business action.

Transcript

Read the full transcript

yeah and we are just going to jump straight into our next session um I am Bill McDow hi y'all I am joined by Rosanna K and Carrie Edelstein uh who will both introduce themselves in a moment um I'm going to start because my company has the name on the banner so I I get to do that um no uh we are going to be talking about what I think is a fun topic um so Trevor just shared you know pure method you know these these exercises that we can give to participants cool analyses that we can do um on the information that they share but what if you know those participants don't exist or are really bad or you know are robots pretending to be real people like that's you know kind of what we're up against as research and insights folks um and that's the whole reason for wanting to have this this um this talk so we call it the show that is research sample quality but we'll just call it quality in general um you know this is what we're going to do so I got you got myself I'm going to play the role of sample provider or operations I got the Short Straw in this uh in this sort of panel array um because I'm the one who who we're going to be fussing at and talking junk about uh we're joined by Carrie who is going to be our research agency uh participant or or representative and then Rosanna who's going to be our client cider um kind of bringing all these three perspectives I'm certain that there's a joke about if the three of us were to walk into a bar if somebody could drop that into the chat or the Q&A you get the door prize for the day I don't know um but yeah actually so speaking of chat um what I'd really like for this to be is not your typical panel discussion right so you know if we're at an inperson and this is honestly I think you know a good advantage of a virtual conference in this case uh if you're in person you're sitting in the audience you're watching on stage the experts talk and do their thing you get an opportunity for maybe some Q&A at the end but it isn't all that interactive we can do that in this case um we've got the chat we've got the Q&A let's all kind of weigh in and talk about what we've experienced what we've seen um what we're up against cuz we are all in this together um this whole data quality thing let's treat this as a big therapy session right you know we'll walk out of this this will be cathartic we're going to feel good we're going to laugh we're going to cry um but no holds bar let's just you know talk about the reality of what we're up against and see if we can't you know at least draw attention to it raise Consciousness know that we are not alone that you know some of the hair pulling out that I especially have been doing over the past several years um isn't isolated to just ourselves right so like I said I'm going to kick us off uh real quick I'm going to walk through some I don't know State of the Union or level setting uh accelerant we ran a panel or I'm sorry a poll last week because you know surveys we we don't give participants enough we don't do enough uh but basically just to ask researchers um and that's suppliers clients Consultants uh academics ux CX M RX all the different X's uh you know what they thought about the state of of of data quality right now um so a couple hundred of them were gracious enough to give us some quick opinions and it ain't pretty y'all um you know what's the moral of the story I mean you've you look at the what's in front of you here and what four out of five researchers or research users are saying that the state of participant quality is eh at best and and and it kind of goes down from there um we ask folks you know how has this changed over the past you know several years right so you know past five years has the state of of you know participant quality improved stayed the same or declined and yikes so past five years three out of four saying that it has declined um in the past year I don't know if there's a glimmer of hope to be had that you know it's it it got bad and we'll call it even terrible during and postco maybe it's stabilized izing it's certainly not getting better but I don't know that I want to hang my hat on that um we also asked a lot of just open-ended like yeah share your experiences tell us your feedback um and what we got a whole lot of was just you know the talk about you know purchasing sample or purchasing participants for for research studies and the cleaning exercises that must go after having received you know we'll call them complete right um and in some cases we're talking about you know folks that are doing this 25% of the sample that they are purchasing they are removing for you know a myriad of you know different bad quality reasons um some that are talking more in the 30 40% range we've got folks talking about like 70 to 80% of the sample that they are buying is unusable and my word people um you know I think about like I go to the you know Chick-fil-A Drive through War I'm in North Carolina so let's go Bojangles but I buy my four piece you know chicken tendies I take them home and I'm going to be able to eat you know somewhere between one and three of them like that you know in what world does that make sense um that's the whole point of what we're what we're were looking to talk about here um next up is going to be Carrie so Carrie did a recent presentation at sample con um which is you know just a very very compelling in I don't know we'll call it Theory or or you know idea that she has as to some of the root causes for what what we're seeing here from a quality slippage standpoint um I'm gonna let Carrie introduce herself and and kind of talk through awesome thank you Bill and that that was really interesting I feel like I'm in the group that would have said poor but I was actually just asked when we were at sample con I was on a panel after my presentation and we were asked um or the four of us who were on the panel were asked if we believe that this state of sample and this issue with quality has gotten better over the past year or not and it was interesting because I think people who work in sampling were like yes clearly it's getting better we're addressing this and then those of us who are buyers were like no or I had the thought as you were talking bill of like we've stabilized the show we've solved it you know or we've acknowledged it we're trying to fix it but you know I'm not observing in my own data that I have to do any less work to make sure we have good quality data and trustworthy data so and that is thing I just you know 20 years ago I didn't really review my data to make sure it was right I I reviewed it for mistakes in the you know in programming things like that but I didn't review it to make sure that people were actually humans like that's just kind of a that would have been a crazy thing to think about in 2005 you know but that is a thing we pay very very careful attention to today that um I was at a a different presentation that was given at the CEO Summit that the insights Association puts on back in I think this was in the 2020 2 or 2023 Sumit I forget which year and I can't take credit for the couple slides I'm about to share they come from C Consulting so those of you who know Simon Chadwick this is his brainchild not mine but as these were being presented I had this kind of aha what's been happening in our industry so I'll share these couple slides with you see and I'm gonna they're not going to be in full screen mode because I want to be able to see you and my screen at the same time but can you give me a thumbs up that you're seeing the screen so this is what what they showed was essentially the the ownership structure of the research industry so not the brand side but you know sampling companies research agencies Etc this is who owned them 20 years ago so there were some that were public companies I used to work for one of them Harrison are active went went public while I worked there um and then several of them were corporate corporate owned so big corporations and then many like the one I own now and Bill I think you're in the same boat were privately owned and just a tiny bit of private Equity was starting to come into the industry and no Venture Capital at that point in time and then they showed us the same distribution for 2022 and it looks quite different you start to see that it's it's mostly private equity and Venture Capital that now own re the research industry and there's still there's actually quite a bit more public but there's still some corporate very little privately owned industry anymore that's really just the small businesses that remain so I started to think to to myself like wow that is such a profound shift in ownership structure in our industry and you know really about the the financial incentives of that and this is one of the other slides I I showed I'm toggling through here but um one of the other slides I should add uh sample con was the definition of private Equity from the American Investment Council and it talks very specifically about the idea of private Equity ownership intentionally having a 3 to seveny year ownership cycle now I've owned my company since 2011 so we're in year 14 and I sometimes still feel like we're start up and Bill I know you've been at this even longer than I have so to me like when I had the seven-year Mark we were just finally like getting started and like knew it was up and kind of had you know we're felt like we were off to the races at the three-year mark it was still kind of like what are we doing here and so that's that's very very early stage to take on a company as an owner and expect to flip at that quickly and so one of the things I believe has happened in our field is that this ownership structure in particular private Equity but Al also two degree venture capital is motivated to grow and sell not innovate and sustain and data quality is an innovate and sustain exercise it's not a grow and cell exercise and and I think there's just like a there's a misalignment there with the organizations that own sample companies and what we need out of data quality that I I don't actually know how we reconcile that under that ownership model my my whole presentation was actually about we need to go back to owning ourselves you know can we even solve that which we don't own because the goals of private Equity are or operational efficiency it's not Innovation it's not coming up with new ways of finding people you know I floated a question at sample con last year to another panel that I was not part of but just from the audience like how can we innovate in sampling to be more represent ative without technology and nobody could answer the question four senior Executives on stage and no one could answer the question three of them used the word technology and answering the question which drove me a little bunkers and there are some really obvious answers like Partnerships right you need more young black men partner with hbcus that's huge here in the Southeast you know why are we not thinking a little bit outside the box and the answer is that's a more expensive way to do it it's more operational overhead it's not as efficient but it would lead to representation it would lead to certainty that these are real people and would lead to data quality and we're just not doing some of these things because the ownership structure of our industry now in my estimation is totally at odds with what what the ownership structure needs to be in order to make those Investments that are not going to be as high margin right it's going to be a little lower margin but it's going to be more sustainable and I think that's just kind of a fundamental disconnect that we need to reconcile as an industry I shop stop sharing my screen that's that's kind of my high level of feeling on what I think is happening in our industry and um you know we can spend all this money to fix fraud but it's you know in a world where we you know owners want to spend money on Tech and they have to spend money to fix fraud that means they're not spending money on real Innovation on doing things differently on you know thinking about how they could bring new people into the mix and and how they could be more representative and less duplicative of the same panelist taking your surveys over and over again they're putting it that that investment money to fixing a problem rather than harnessing an opportunity and so I think our money is being spent I mean it's being spent on a necessary thing but ultimately we are spending money to fix the problem that that ownership structure caused in the first place that's my mic drop hey drop it go for it um yeah Rosanna uh all right so you're the representative of the the corporate side you've gone through your career in since at what Ty in PNG Duracell um what have you seen over those years yeah um awesome I'm so excited to be here and join the conversation with especially with Carrie who's so passionate about it I love it um I'm just over here soaking it in and learning um but great to be here guys my name is Rosanna K I've spent 14 years in kind of client side consumer insights work um at Proctor and Gamble and most recently at Duracell I'm now doing some independent Consulting um but this is um a really interesting topic for me because I feel like over the years it's not the part of strategic insights work that we've talked about a lot on the client side but over the past few years I've come to see it as like part of my value proposition as a strategic insights leader right like it's not necessarily the cool flashy stuff um like that we bring to the business team right that like ensuring we're asking the right business questions and executing with foresight and uh driving insights to implications and action right those are the things we'd like to talk about on the client side as far as the value we bring to our business teams but over the past few years I've come to realize like part of my value is just making sure that we are executing what I call decision grade research like are we making sure that we're partnering with the right suppliers that all the right processes are in place to ensure that we're making decisions on data that is good uh and valid and we don't talk about it a lot because it at one point I think it just seems like a cost of Entry right of course we're doing quality work right like of course we work in insights and we're doing quality work um but it's a huge part of what we do and kind of the value that we add uh making sure the data is good making sure that every study is designed kind of with an eye toward our category or kind of our business need um and I you know I've joked with some senior leaders in the past kind kind of joked but like a lot of it is reality of like do you realize how much you would be making decisions on bad data if I weren't sitting in this seat and making sure that one we're partnering with the right suppliers that have that eye towards sample quality or that we're coaching our supplier teams on the nuances of our category that necessitate kind of changes in methodology um so I think it's something I'm excited to talk about it like I think it is truly like a piece of the value equation that we bring as client side researchers I love that and you know it's funny I WR I wrote a piece for the insights Association blog I think is where it ended up being published I should share with you that was all about um sort of Brands side researchers like owning that power that you talked about and and not kind of you know I know you and I have talked about software as a service kind of being like that's the client side job just you know get subscriptions and do everything yourself but it's like well okay but then you don't have the time and the bandwidth to really review everything in that level of depth to make sure it is decision grade I wrote that down by the way I'm goingon to totally Co-op that from you just just in grade research I think is a phrase we need to just socialize out right and that actually takes work to really monitor and I think that gets a little obfuscated in a DIY environment where there's just stuff coming through and you don't necessarily know where the sample is coming from a lot of those uh software companies those Tech platforms will tell people that they they have a panel they do not have a panel that's not even true like it literally is not an asset on their books you know it's just it's a partnership like anyone else would have just like we have as an agency and you know I bill I know you and I have talked about this behind the scenes but I actually had an intern a few years ago do a secret shopping exercise in the sampling space just to see how many people were lying to him it's like most of them were just totally misrepresenting what they could do and they were just taking other exchange you know well sample from exchanges and things like that and marking it up to like eight to10 cpis but they were paying like 50 cents you know and then they're like we have our own panel and it's really well beted and that's why it's eight to $10 but like no they don't they're just making a th% markup on this stuff so it's kind of nuts what's quietly happening behind the scenes and I think to your point I really love the attention that you put towards recognizing that you know the job of a brand site researcher is to make sure that that's not quietly happening behind the scenes so that you have defensible research to make big business decisions right if you're off by a couple percentage points let alone five or 10 or 15 percentage points which can happen in these scenarios you know that changes Marketplace outcome to the tune of millions of dollars sometimes absolutely we can't be off by that level of error and you know it's one thing to have sampling error it's another thing for the sample to be erroneous you know and I I I feel like sometimes we are not making clear the table Stakes of getting that wrong and I really appreciate that your worldview has been you know defend that ground and make sure that the organization can trust consumer insights because I think that is something that I see happening less and less at different Fortune 500 companies they don't realize how bad it is and aren't giving it the attention you gave it so I I appreciate that you're carrying that torch yeah absolutely no I think it's you know it's having hard conversations sometimes right because we always talk about right the trade-offs between speed quality and cost right they're all moving uh you know in continual triangulation um and and yeah I mean with with technological advances I mean Carrie you'll you do a great job talking about like Tech timid and asking the right questions just to make sure that we as researchers understand exactly what's going on like it's like you need as a researcher to understand exactly how this is working right like um so asking those questions and understanding that but but you know a lot of times it is an internal conversation for me around kind of speed quality and cost right um and just staying up todate on what is available because we are continually improving right there's continual stuff that's coming out right on the best ways to learn with consumers um and so so you got to stay a breast of that but there there is going to continually be this speed quality cost tradeoff and that's a conversation I have with business partners all the time which is like yeah we can do it cheaply or we yes I can have you an answer tomorrow but I don't know that that's an answer I want to make a decision on um and you know part of it is rightsizing learning plans too risk reward um like what do we already know all of these things right as um a strong insights partner that you're filtering things through but it's also like what are the hidden costs right I think there's a lot of hidden costs to some of this stuff so you know you you brought the most important one a wrong decision is a massive hidden cost right it's not so hidden once it happens but it's hidden before it happens and and the other thing I think is like the time that it takes you to do some of these things so you know I remember being at a at a conference where there was a presentation on the cost of DIY and the presenter went through all these little hidden costs but none of them was Labor and and I was like when you factor in the cost of DIY do you factor in the time and the salaries of the people doing it themselves and the answer was no I was like do you think clientside researchers work for free their time is valuable they make a salary there are payroll taxes and office space costs and benefit costs and all these things on top of that and you're valuing their time at zero and you're also valuing the opportunity cost of what they would be doing in set at zero and that's such like those are two major hidden costs that I think sometimes we just don't calculate in all of this and I think it's it's so important to really like do that calculus correctly but in particular I think your point about you know the hidden cost of make getting just making the wrong judgment call on that is probably the one that backfires the quickest and can lead to long-term distrust in the insights industry overall I love that one on opportunity cost too though because I mean I think that's a big one as a client side researcher depending on you know the size of your team and the size of your scope right like if you're learning on one thing you're not learning on something else right and so we have to make sure we're going after the again the biggest most impactful business questions and decisions that we can influence and um and if they're big and impactful they need to be done rigorously right and write and you want to be talking to real consumers and um answering them in a in a rigorous way so and but that didn't used to be the role right I mean there there was there was a time when the corporate Insight professional could just trust that what they were purchasing I mean the quality you know piece of that you know quality call speed triangle was more about you know is this going to be a good report that's you know well written or am I going to have to redo this thing it was just an understanding that that you know it was just it was going to be real people taking part in my research and it wasn't going to be you know just completely wrong so I mean that skill has been you know a relatively new one that has to be to be polished and I mean I love the decision discussion because I mean ultimately that's you know there is going to be some sort of Reckoning unfortunately about all of this right like I mean it a decision a new Coke level decision is going to be made based on garbage research or Garbage data and there is going to be a down the chain sort of you know recking for it I personally have no interest in being you know part of that you know business school case study for for for decades to come but like that's you know that's what someone has to be tasked with you know kind of protecting for and Rosanna if if if an organization has been smart enough to bring in someone you know with your type of attention that's great but there are a lot of organizations who you know are bringing in insights folks who are less seasoned who maybe don't even know to even ask these questions and they're relying on suppliers who in some cases I mean when you hire your your supplier you have to not only be you know thinking about their quality but also the quality of those who they subcontract to right cuz I mean it can be a brilliant research team that you're working with but again if that raw material that they are purchasing is flawed it it's going to make its way to you terrifying yeah it is and um you know that's actually an observation I've made I feel like you know you're like this bright light Rosanna in an industry where a lot of people don't ha have your attention to to those specifics and you know we've seen that play out in a lot of ways and one of them to your point about it didn't always used to be this way you know I think the tech and and ren you mentioned Tech intimidation that's like a phrase I coined because Sor I GNA socialize it out let's make it a thing um but for those of you in the audience who don't know what that is it's this idea of like tech companies intimidating you into thinking you're missing something when in fact they are missing something and and so the thing I hear most often is like the condescension of like you just don't get it you just don't get we're trying to do and I'm like well I don't get your business model but I do get what you're trying to do and I think you don't get that it's not working you know and and and sometimes I'm just like you're right I don't get it I don't get why this is sustainable why it will work explain that to me and they can't you know and that's sort of the point I'm trying to make and or they talk over you in jargon or acronyms to make you feel stupid and you know make you think that you don't deserve to be part of that conversation you know which is just you know patently ridiculous when you're when when you're the one spending money to buy that Tech you know like actually your customer like that skepticism is a form of feedback you know and you should be listening to it if what you're saying isn't Landing with me so that's where the whole Tech intimidation thing came from in my mind but I think it just speaks to um you know sort of a bigger issue that we've been overlooking that part of tech intimidation in my mind is spreading this idea that every decision has to be made fast it's the agile culture and there are absolutely some decisions that need to be made by tomorrow like I used to do I do a ton of work in media and used to do a lot of work in the entertainment industry where we'd be test screening a show that was going live next week and so yeah they The Producers needed the results of a test screening the next morning because they were re-shooting or recutting that day you know to be live next week and so yeah that made sense right that some of this needed to be really quick turn but sometimes I'll see a company building you know a really elaborate e-commerce platform and they're like I need the data by Thursday I'm like why this is like a twoyear long buildout there is literally no consequence to you getting it in a month versus getting it in a week you know and it like this you're not going to get to the point in your buildout where you're using this you know before that point so why is everything a rush and I think we've a little bit lost that ability to discern what actually does need to be quick turn and what doesn't need to be quick turn and frankly what shouldn't be quick turn because you can't possibly cross all your tees and not all your eyes if you do things too quickly and also I don't think you know and Ros you can speak better to this than I can but I think some brand Brands actually don't have the bandwidth to act that quickly on things so when I hear about these like nightly fielded studies I'm like who is the bandwidth to make a different decision every day I'm like maybe weekly you know but more likely monthly or quarterly at best these things are going to be addressed yeah these companies are out there being like you need new data every day I'm like do you like could you possibly act that quickly and nimbly do your decision processes even allow for you to act that quickly or is a chain of command too slow or relying on people who are bottlenecks which is more often the case so I think there's been this Tech imidation of you need this thing that's agile and quick and that is sometimes true but it is not categorically true and and to your point I think we sometimes like we've lost sight of when it is and when it isn't or maybe the ability to discern when it's important and when it's not and sort of you know we're being told it's important all the time and I don't I don't really believe that as a business owner like not every decision I made you know I'm dealing with needs to be made tomorrow and in sometimes you some cases I'm like get back to me in May this is not a mission critical decision I've got other fish to fry sooner and so I know as a business owner it can't possibly be true that every decision has to be agile yeah and there Carrie there was a heyday of you know purchasing sample where I mean I could be spending you know two bucks a complete I could be getting you know whatever audience I desired and I could feel the you know very high sample siiz survey over a weekend be back Monday morning and have you know some pretty clean good-look data in front of me that has right and we are now to your point on time having to not only you not be able to achieve those quick turnaround goals for data collection but we are adding on you know the need to now take our data and you know analyze them to be able to analyze them right I determine first if it's even you know what we intended to to collect or if you know again to the you know the results of our survey like you know am I having to throw out you know 5060 70% of of my results go back to field and get more that weekend is is gone at that point so you know we've gotten spoiled you know it's setting in that you know the actual reality right now and much of it is our own doing and our industry is doing but you know where do we go from here right yeah and and also like that you know that aspect whether it's 30 40 or 50% takes time to identify who it's not always super obvious right and you know we've we've run into that at our company that there's a level of expertise in the the subject matter you're studying really required before you do the study to even identify who's faking it in the study right which can get really tricky so an example I'll give on that is um like I said we do mostly media research and we had a study about a year ago maybe year and a half ago where all of a sudden overnight we had like 200 Native American men 18 to 34 with college degrees come into our study now just so happens that I'm also a consultant for a Native American Media company out in Alaska and we had just done a bunch of work with them on their listener base and like who's in it and so I learned a lot of information about the indigenous community in America and what their demographics are and one of the things I learned is that they are less likely to be college educated not more likely and that's actually something they're working trying to work through and um and so I'm looking at this data I'm like why would this group over index on college education they should under index and also Native American men are very very small incidence of the population how do we get that many in that quickly that doesn't seem logical and my team in you know not knowing all of the things I had just learned the week before that this indigenous client where I happen to be you know doing a retreat with them they were like oh this is amazing we got all these you know indigenous respon this is so great and I was like that's definitely wrong right and but you don't have the experience of work working in that Community to know that data is absolutely Incorrect and so not only do we have to be knowledgeable about media we had to know every demographic how they consume media and what their profile looks like in America that's a a lot of layers of expertise I had to have to flag that right away and you know I started to think to myself like what do you do if you're a junior researcher like I have 25 years or more of experence almost 30 now of experience in this so I start to flag that stuff pretty intuitively but somebody who's who has two years isn't necessarily going to know all that so how do you even flag these things and I I would say I actually rely on my clients a little bit to flag when something looks off because you know Rosen you're going to know much more about the battery Market or the cpg market than I would and you're gonna flag when something's like that is way too different from my other 10 studies to be right you know and that's and and so then there has to be this very open relationship between agency and client about the magnet to fraud so that we can have a conversation about it without an agency feeling guilty for that that being the way it is when we're not actually in control of that so we're we're bold into it too so I think it there just needs to be better communication so that we can do that but we got to build it into timelines I now build fraud fixing and and data cleaning into my timelines including the possibility of I need to ref field to to get some more data if that's wrong and that means to your point bill there's no no Fielding over the weekend right we're not getting that done over the weekend unless we're just lucky that we didn't get hit with a fraud attack but more often than not I'm like I want a full week because we could have Monday morning and realize this is all wrong we got to go back in the field Tuesday morning after we fixed it and now we don't have data till Thursday you know and so we're trying to build that into timelines but that's hard you know in a situation where people want Intel to make a decision you know next day two days later it's really hard to find that balance you know there there are other aspects I don't know if you want to go into this bill I don't want to go off discussion guide but um like there other we're talking a lot about like sample fraud right like are the people that we're talking to the people that we think they are right or are they Bots or you know whatever but I I mean I think that's only one aspect of quality right like when that and the other aspects of quality take time too maybe even more time right which is you know I think I think do I have am I talking to the people that I want to talk to like are they real people are they engaged like they might be real people that are the exact demographic or you know profile of who I want to talk to you but are they engaged and like giving me quality answers and reading the questions are they are they reading the questions are they am I designing my survey or you know whatever it is in a high quality way that's going to kind of give me the best possible chance of getting those high quality responses um and then related to that but I think of it in a separate bucket is sort of am I being realistic about what I can learn from these consumers right and and ensuring that that's quality checked too um and I'll give you an example on that one is you know I most recently worked on the Disposable batteries category which as you guys know is so fascinating right like I I'm sure you as consumers think about disposable batteries like all the time right like no you don't like the average consumer thinks about disposable batteries maybe for you know a minute every uh you know four times a year every three months right you're going to spend 30 seconds at the Shelf to buy some when you need them um and that's the amount of mental space that they're giving to the category so we put a lot of effort uh into like we recently revamped our brand tracking program and I was adamant that we are not asking consumers a 20- minute survey on disposable batteries because that is not like that is not a category that they think about for 20 minutes and I'm not bury a kpi that I'm going to make brand decisions on at the end of a 20 minute survey right so like the thought and thinking and category knowledge that goes into that I think is an important aspect of quality too and all of it takes time yeah that's such an important point because I think back I mean kind of to Triangle that with what bill was saying earlier too you know when we first when I've been working in online research since the late 90s because I was at Harris when they were starting to do this stuff with you know with the Harris interactive in like 98 99 that era and you know in that first decade or so of online research we used to soft launch every study or preest every study not just to make you know to get a sense of incidence rates and stuff which we still do now but in survey length but we would actually kind of stress test some of the questions to make sure they were clear and understandable and that or not correlated with each other do we need all these questions if they're coordinated to a level of like 0. n you know maybe we can come on a couple things or cut a couple things and we would really spend a day or two looking at soft launch data clean up the questionnaire and then full launch I like I haven't been given a window of time to do that in 15 years you know and to your point right and so that's a really interesting point that from a data quality standpoint that aspect of like attention when you do have a real person is real I would say attention and then personal relevance like you asking the question that's how they frame things or are you kind of going down wonky jargon Lane and that just like that's how you talk internally but it's not how people talk externally about that category yeah and yeah I sort of missed the days where I got to learn from the soft launch improve the quality of the survey instrument or if it's a qualitative study right maybe you're doing six focus groups after the first one you meet you debrief you clean up the guide sometimes we have the time to do that now but oftentimes it's like no we need all six groups in two days you know instead of over three or four days so that you can actually like iterate and kind of take take your time to make sure the guide's working um and I think in all forms of research we're just kind of rushing to field on stuff and not getting to spend time making sure that's a great point that like you're hitting the quality part that's actually more like attention and relevance and relatability so that you're getting good data back from people yeah and and fighting to be a part of the conversation even right so we talked about you know software as a service and and platforma ization but you know DIY survey tool RS exist and you know Julie dropped an interesting comment in the chat that's you know in line a lot of we're talking about but you know I know Rosanna talked about you know being the steward or the protector of you know these bad bits of information from making their way to your the folks that you're supporting internally the decision makers but you know Julie talks about the pressure that is coming from those decision makers that you know get this stuff turned around quickly get this stuff because I got to make these decisions yesterday so and if I am sitting in marketing strategy operations and I can just go grab myself a Survey Monkey free trial and zip out a survey written horribly but at least get some data back in the time that you know I'm unwilling to to to bend on what does that mean right so like you mean our job as researchers is to now like be everywhere and try to maintain our relevance and stay involved and it it is not an easy thing to do no and you bring up a good point that some of those platforms also don't have the functionality to enable quality so like Ser m is a perfect example like I won't use it on anything complicated because there are certain types of randomization or block randomization or question randomization that I can't do in that tool that something like well what they're not all forced to Brands but decipher and confirm it we built to do and and and so I know that if it's a really really basic thing I'm just trying to get you know a short list test from a newsletter list of one of our clients we just want to kind of get a sense of how often do you read the newsletter right yes you can use Survey Monkey for that but if we're really getting into things where you know there's order bias and things like that or we got long lists or competitor list things like that you've got to have something that has all those rotations and randomizations built in and it's like you got randomizations on your randomizations sometimes in order to do it right and not all of the tools enable that and not all the tools enable nested quotas either that's another thing that I run into um so for those who don't know nested quota would be like instead of saying you have quotas for age and gender and race and ethnicity it's age by gender by race and ethnicity knowing that those things have compounds so and and I started doing that you know all you need is one study we're like all of your people of color are women and all of your white people are men right or something like that that and I had that happen not to that extreme but to a pretty big degree on a study probably 15 20 years ago now like only has to happen once before you're like nope I'm not leaving that as a soft quota I have to Nest these things or I'm going to get wild swings that introduced introduced compounds that I don't know what to make of and they can't be controlled with data waiting because we actually don't have enough people in some of the cells to do data waiting and there are a lot of tools out there that cannot handle nested quotas like they just don't they don't build into a platform and then that could also not handle data waiting if you want to use their dashboard and so that kind of speaks your rosan of like another quality issue that even if you do get the sampling right you know or you have real people you may have the wrong distribution or the wrong representation of people for that you know Marketplace you know whatever that might whether it's batteries or TV shows you know um and I and I think that's part of it and one of the comments I made to a sampling company that was complaining about this was like well why are you selling into directly into those platforms instead of around them to agencies and and clients who will fix that why are you selling through those platforms when you know they're G to wreck your data you know why aren't you doing some training with them and holding them accountable and setting standards for who you'll sell to in order to maintain that integrity and the short answer is because my Venture Capital private Equity owner wants me to scale as fast as possible and would lose their money Minds if I did that and I go back to my first point of I don't know how you reconcile that when you are not owned by people who care about data quality but you're own by people who care only exclusively about scale to the exclusion of data quality at times but we just you know to your point both of your points we spend a lot of time just intervening on that as much as we can and you know I've coached my team to assume everything is wrong and then be delighted when it's right and just look for what's wrong and be really Vigilant about it every day that we're field so we catch it before it comes out of field and to only use platforms where we can set that up up front but I think when you know some of these software companies are selling direct into not just end clients and research but like in our marketing teams and our product development teams and so on who don't have that rigorous training and they're going around research teams you know all of a sudden research pops up in a different discipline and it hasn't been designed properly it's got you bad sample fraudulent people or not real people and then it's also architected poorly and so you get this layer on layer of all the data quality issues in one place speaking of rightly named presentation Bill well and like all right I'll throw another one at you but that is going to be the you know we talk a lot about you know our own selfish needs and and you know what we desire to do with these data and and how we're applying and what processes we're putting in place to to to try to fix but you just also the quick timeout to think about this from the you know the research participants say Point like the participant experience in our industry like it ain't pretty I mean it you know think about you know the journey that one takes to you know to get to a a data point right like I mean the and especially the Myriad like you know steps and platform and and rerouting that a you know a single participant can go through in order to even make it in to beginning a survey or or or or a bit of research and you know let's talk about incentives for a you know a hot second but like think about $2 sample for a minute there right like it it's been something that that can be purchased now that is what you would be paying for someone to complete your survey now I can go out and buy a half hour you know surve Sur complete among you know a lowish incidents um among the general population for two bucks right us minimum wage is what 725 these days half an hour for $2 how much of that $2 actually makes it to the the reward pocket of the person who's actually taking part in this research and man that's you know that's rough um you know a half hour qualitative IDI is going to get somebody in many cases a hundred bucks right yeah why would I be getting like you know Chuck-E-Cheese reward point or two for that same half hour if I'm sharing it in quantitative research um I mean that's something that it's going to have to be fixed at some point I mean these folks are effectively you know we talked about sub subject matter experts in our in our last presentation but you you're hiring these hundreds of many consultants to to give their time to you and you are paying them nothing in many cases and that's not right um I don't know it's disrespectful of your customers too right in the end right it's like it's kind of disparaging of the the customer you're trying to win over isn't it and and I was reading Ian's question in the in the chat that kind of you know to me is like the solution being offered to this now which is synthetic data which I don't personally believe in yet um you know I think the fundamental challenge of so first of all I want to back up and say I don't I hate the phrase synthetic data because it sort of dehumanizes the customer um but and it's also not synthetic it's just modeled human behavior right that's what synthetic data is and um you know we have to give everything a new sexy Tech name right Tech tation and my feeling on that is I think it will work in some areas where big data can model like so I saw a couple people presenting at sakon Big Data creating digital personas and then you ask a digital Persona something in much the way you might talk to a chat GPT and that I think at a certain point will be interesting for some very specific use cases that involve observable behaviors on the internet spoken about on the Internet or in other public forums and transcribed but I think when you get into things that are very future thinking and More attitudinal in nature more the kinds of things people don't speak up about and keep to themselves you know I don't know how you model that which isn't observable right that doesn't make any sense to me as a statistician like it's you know and so I think it's it's an overpromise that will likely be underd delivered and then I also think they're going to run out of data to feed in so at a certain point like you're using existing surveys and data to then feed this quote synthetic data but at a certain point you're going to run out and so that's kind of a talking point now it's like a year from now you're going to run out of data because you've eaten your own base line you've eaten your foundation you know and then what you know and then where does this go so I don't I think it fills a very specific area like I'm interested to see where synthetic data goes and maybe allowing us to predict five minutes of questions from 15 minutes so that you don't have to ask a 20-minute survey you can ask a 15minute survey I think that kind of missing data imputation computational work is starting to show some efficacy in certain areas and I'm kind of curious to see where that goes and I think modeling you know big data sets in certain areas could be really interesting as well but I don't think it applies everywhere so I don't think it's going to be a solution to this problem I think it's going to be kind of its own opportunity that harnesses big data in a more effective way but I don't know that I think it solves the data quality problem in what we're doing I think if anything you're modeling the bad data with more bad data right like I don't know how that really solves anything unless you have something super credible or is you you put a decision grade Rosanna to then model a pawn and most of that's confidential like I don't even know where you'd get that data from so uh I feel like it's an overpromise yeah know Carrie I was thinking the same piece um again like I love your reframe of synthetic to um modeled human behavior I'm stealing that one uh uh but I it it creates again the need for transparency right like as we get into these um kind of modeled behaviors modeled data sets like where what is it modeled off of right and how are we ensuring that the data is modeled off of is real data quality data exactly garbage in garbage out right yeah and you know to that point like I I think that's you know that visibility is where we kind of have to be headed as as as an industry like you know having the right as a purchaser of you know research participants or or you know research um sample or whatever having the right to know where and how they came to be right I mean you know there was a time when you could you know e rewards was one of the you know first panels that came into existence and you know much of their sample came from relationships with like frequent flyer programs and they were validated and they were real people they had their biases sure because of you know having been sourced from you know singular locations but what we would kill for something like that these days just to have you know real people so I mean you know we we demand it from our you know consumer goods that we buy but I think it's something that needs to begin happening with regard to the the research data we're purchasing and that's just understanding from once it came right like I mean how did this this stuff come to be come to make its way into my survey that's actually what I used to love about ewards is that they were so transparent about where it came from that when I knew I needed a sample base that was higher SCS that maybe sixf figure households and I had some clients that were literally travel clients trying to reach sixf fig households with you know cruise lines and things like that I was like oh we should definitely go to E rewards because they're going to have people who are interested in travel and make six figures you know and they're going to be easier to screen for because I know that's a bias you want in in the study that's your it's a nice parallelism there and so I could specifically reach out to them and work with them when I knew I needed that and it wasn't great if I wanted to do like a national sample like a polling kind of thing no they were totally the wrong panel to work with but for a lot of our clients they were actually the right one to work with and I and I think G it gave us as researchers a little bit of that empowerment to do that matching in a way and and to think strategically about it and I feel like we're just missing that transparency today like and I don't even think it's that our sample partners are trying to lie to us they do not know the answer if I ask them where their people came from they cannot tell me and you know I got the idea at sample con to ask it in the survey and I'm like I'm G start doing that like maybe consumers will tell me how they got into the survey and where they were recruited from and we can get an idea and whether or not there's bias to that and get some transparency there but I mean it really seems to be the case that account managers at sample companies do not know where their actual real people even started they can't tell you where they sourced them from which is hard you know it's terrifying for sure um and it's funny I mean I'll put my old head hat on for a minute as well like I can remember when online research became a thing and it's funny that you know the initial fears about doing research on this internet thing was not knowing who your participants would be right they could be lying about you know their Identity or or whatever and it's you know fast forward to today and we've actually like you know created some kind of self-fulfilling prophecy where we've actually like as an industry allowed for that very thing to to take shape now we're purchasing data that we've all now agreed is you know the way of doing research online that we can no longer trust that is genuine yeah good times and that's a happy note yeah all right so we we're coming up on time and and let's talk about happy notes I mean what's being done what do what do we see as the future you know what glimmers of Hope can we look for out there I mean I know a couple that I've seen you know for example are these you know I call them uprisings but it's you know just groups getting together and starting to demand better uh I know insights Association they've recently uh published this participant Bill of Rights is just a thing that honestly read through it it's quite sad because it's very low bar for what should be the experience that people taking part in research are experiencing and like I say read through it I I I highly recommend because it is you know it'll I don't know bring a tear to your eye um I know Rosanna you mentioned we we were talking earlier this week um you know things happening at at you know large end client purchases research like uh like PNG starting to create their own standards or you know data quality um just you know deal breakers that with an oomph like you and purchasing power like a PNG they can start to do something like this and hopefully the rest of the industry will will start to catch on yeah and I think you know one thing we talked about Bill is um the more on this space of synthetic data AI modeling um just data labels right like just like again this sort of idea of radical transparency on where the data is coming from and I think that can be applied across a wide variety of you know whether you're modeling creating um you know models based on a data set or um even what you're talking about Carri right just asking the question where those consumers come from um and just demanding kind of more transparency and understanding I mean that was honestly something that as blockchain has started to become a thing that got me most excited about the the technology and that's the the possibilities from a research standpoint to you know if my non-tech novice understanding is correct we would have the ability to actually you know authenticate people and whe where I'm purchasing them from you know I can know because they have that that tag that they are indeed genuine and wouldn't it be super cool if I didn't have to you know waste the first you know 5 10 minutes of every survey that I field asking the same screening questions over and over again which this person who's taking part of the research has been asked a zillion times like you know my birth year hasn't changed in a while I think right like I mean that's not something that that I should ever have to answer again but we do it because these are going to different sources and different data sets and it has to be asked again and again and again yeah I agree with all that um we can see it in our data too when that happens because people drop out at question one question two and that's how that's the indicator that you know I'm probably the fifth time they've been asked it um I will say I do feel like there's a lot of attention in Industry towards us now there's tons of data quality initiatives I really appreciate there there's a a flood light being shown on this in a way that didn't exist a couple years ago and so I feel encouraged in that direction I also think there's going to be a boomerang back to classic panel-based research like we used to do because it worked and it wasn't as cheap at now the alternative is actually no longer cheaper because we're spending so much money on fraud so um I think there's going to be some of that boomeranging back over time which is encouraging but I think also you know and clients are becoming hip to What's happen happening there and there's this demand as you said Rosanna for radical transparency that I think is going to start to become the norm it's not going to happen overnight it's going to take some time but I do feel encouraged that the people like myself and you Bill who've been speaking out on this and concerned about it for a while like I you know I got to Keynote at sample con and and bring this all on stage and I don't think that happens two three years ago I think we got to a Tipping Point where they want to be on stage because everybody's really nervous and we need to have an honest conversation and so I feel like that's the starting point right is that the people who were kind of dismissed as Skeptics are now being heard as maybe the early adops of understanding the writing on the wall and um and I so I think there's a little bit of a pivot that's taking place that I feel encourage by I mean sample Source like you know I started a research agency with absolutely no intent to to get into the the sample game um have been sort of forced into starting to build the panel just to be able to help with initially some some qualitative research but then kept building it because frustrations kept growing and always intended for that to be a s a one among many mixed method sources to to be able to do research and you know we've built a a nice siiz proprietary panel which you know until visiting sample con a year two years ago I thinking was ours was an adorable little you know panel that's that's that's great and it turns out it's one of the the largest at least proprietary in in existence and that's not that's not a good thing I don't I don't like that and we're certainly leaning into building more and more sample to be able to deliver real people into real research but man the sources of of getting these data are are getting scarcer so we got to figure this out are we coming up on time thank you yes and I have to go any parting shots no I mean last bright spot is like when you get a real consumer who's willing to share a real story whether it's you know qualitative or part of a Sur quantitative survey um man I mean that's why I do what I do so really encouraging we're continuing to find those real people that uh you know give us insights to drive action thank you thank you for having us thanks for your time I don't know if we solved anything but we certainly had some fun so that's good appreciate it y bye