Think Like a Fraudster: The Blind Spot in the Industry Fraud Conversation

All right, it is the top of the hour, so it's about that time. Um, I'm going to go ahead and get us started. Uh, welcome everyone to the June 17th installment of the Excel Research Virtual Insights Conference. Uh, my name is Bill McDow. I am the president at Accelerant Research and I'm going to get us started for today's festivities. Um you know for all those who have uh have attended our conferences in the past uh welcome back. Uh for anyone who's attending for the first time welcome. Um effectively what you have in store for you is just a series of backtoback all day webinars on the subject of research and insights in general. Um covering a broad array of topics. Uh we got some really good speakers for the day. So very excited about it. Um, I'm going to do a quick intro MC kind of welcome for everyone, but then I'm going to quickly get out of the way and let us get right into the uh the content at hand. Um, I'm going to first start by sharing a couple of ground rules that we've tended to live by with these conferences. Uh, so we began doing these virtual conferences actually during COVID uh shutdowns. Um, and we've just they were wellreceived at the time and we've just continued to do them. uh but you know I will warn that accelerate research we are a research company we are not an event planning or conference organizing organization so you know we tend to be a little less polished a little bit you know more raw but I think we we kind of like that um so you know that's that's the way that we we tend to operate um so I was warned that we are at the mercy of tech we are at the mercy of you know anything that could could and does happen at times. Um, we still got, you know, a lot of our our presenters, a lot of our our speakers are still working remotely. Um, and you know, sometimes stuff comes up. Uh, I call it the happens clause of these these conferences. Um, for example, uh, in my office, we've actually got a lot next door that is being cleared for for new construction. I've got bulldo bulldozers going by uh, every few minutes. Uh, so if you see anything shaking off the wall behind me or if it looks like, you know, you hear anything rumbling, no earthquake. It's just just me dealing with that. Um, you know, other rule we we like to live by is be respectful to one another, to to our speakers. Um, you know, the fact that this is a virtual conference, you run no risk of bumping into each other in the buffet line. So, you know, sometimes you have folks behind the keyboard that, you know, will let loose a little bit more. We tend not to have uh rudeness or disrespect uh with these conferences but we also always like to mention it out loud. Um you know that and that doesn't mean that you can't and shouldn't uh engage with presenters that you shouldn't uh you know ask questions um even to the extent possible network with one another in attendance. Um you have both on YouTube as well as um um on the uh the excuse me Zoom live stream that that we're on. You have a chat function, you have Q&A functions. Um, use those. Uh, network, engage with one another in attendance as as well as our our speakers. Like I say, ask compelling questions. Um, just quick housekeeping announcement. Uh, we've got actually another one of these conferences coming up on August 20th. Uh, that's going to be our next one. We're skipping the month of July for for summer vacations, but coming back in August and continuing monthly after that through the end of the year. So, if you haven't registered, please do so. Um, we're going to keep this keep this thing going. So, how to get to registration, just go to accelerarchearch.com. Uh, you can go to the resources dropdown and the conferences tab and you'll have everything you need there. Uh, you can also access prior conferences, all that we've done in the past for lots of engaging and fun content um to get lost in as it were. um quick overview of our presentations that you can expect for the day. Um so starting in an hour at the uh noon Eastern time slot uh we're going to have Gavin Johnston talking about hyperpersonalization tailored experiences in the age of AI uh followed by Dr. Kimma Singh and she's going to be talking to us about beyond research how AI is transfor transforming insights into decision engines and batting cleanup today is going to be Carrie Dugan and she's going to be talking about the price of insight the overlooked realities of conducting research in vulnerable communities. Um so you know we can't not talk about AI it's on everybody's mind it's you know key and hot topics but it's not all we're going to talk about today. Um, so again, we've got some really good really good presenters from different backgrounds and lots of different topics to to engage and sink your teeth into. And that's kind of what we're going for with these conferences. Um, but our first speaker, uh, without further ado, I'm going to present, uh, Terrence McCarron. He is the CEO and founder of Opinion Route. He's going to be talking about Think Like a Fraudster, the blind spot in the industry fraud conversation. I will say just shout out to Terrence. Um he's doing some really good work out there. He and his team um I mean if you haven't been living under a research and insights rock for the past several years um you are have been slapped in the face by the difficulty of just conducting research and getting quality participants to take part in research. Uh Terrence and his team do a really good job of you know trying to combat that. Um and it ain't easy. It's It's bad out there. And you know, I'm really interested to hear what Terrence has to talk about. I'm going to step out of the way, let him do his thing. I'll probably circle back in a little bit to uh handle any Q&A, but otherwise, I'm going to go dark. Terrence, it's all yours. >> Hey, thanks, Bill. Appreciate it. I'll jump in here to share here. All right. A lot of the conversation in the industry really comes through everyone's individual vantage points which makes a ton of sense. Um, so I want to introduce a different lens into an increasingly emotional conversation because it speaks directly to the heart of the integrity of the industry. So for me at opinion route, one of our charges is really building confidence amidst what feels like a really difficult, really highly complex topic. We want to bring a little bit of calm and confidence to your strategy and your equation. Before I talk about the missing perspective though, let me just share a little bit more why I'm the one sort of telling this story. Um, I started Opinion Route about 13 years ago, but this is now 27 years like many of you. I'm living in the research industry. I'm a lifer. I'm a big believer in the sector and its overall value proposition. And I've lived through a lot of waves of change from Katie to panel to programmatic. Um, and ultimately my mission or my purpose for starting opinion route really came down to this migration into high-tech. modern automation was really jumping as being a mainstream truth within the data collection side of quant surveys. Um I saw those risks were sort of emerging in my front row seat and I started getting nervous that there wouldn't be a stakeholder to protect the research process or the research company. So I built a company to do just that. And now about a dozen years later um we have scaled. We have hundreds of market research clients globally. We run thousands and thousands of projects a year. Um, we process a hundred million plus respondent surveys a year. Um, and we've invested over $10 million just into the topic of fraud prevention and data quality. Um, and the thing that I think makes our perspective a little different is a 100% of our clients were already cleaning data when they found us. And in fact, that's why they needed something deeper and more systemic. So our whole build really anchors around this partnership of of not necessarily trying to be smarter than the researchers but recognizing the various stakeholders in the process but ultimately producing data through our process through our service through our technology stack that can really meet the standard of the expert. So, I'd like to start with a scenario that plays out pretty commonly throughout um the industry, certainly in the US, but globally as well every single week. And and maybe this will resonate with some of you. Uh we're pretty far into the project schedule. The field is almost done. The researcher on their team starts digging into the data and with minutes it's obvious we got big problems. Um, so the researcher calls the account lead, gets on the phone with the vendors, starts demanding answers. How could this have happened? Why are we in this position? That moment is a high emotion moment. And, you know, generally speaking, that's a moment that doesn't have have a lot of hope. Um, it's a panic moment for sure. The one thing that's been hard to miss over the last year in the broader industry conversation, whether it's conferences or webinars or podcasts, is that the narratives that are brought to those presentations are often stemming from the emotion of this moment, the panic of this moment. And if you think about that, it makes total sense. It's actually extremely rational. And again we focus a lot on the individual perspectives of each stakeholder. The end clients who are sponsoring the research um they over the last two years have been forced to open their eyes to a truth. This battle that had been waging in the industry behind the scenes related to data quality. there was a long-standing belief that hey my vendor has it covered and that really doesn't feel like it's holding up very well for the insights buyer in many cases for the market research agency which we just shortened to MRA they also had trust they had their own process but because of the complexity associated with the modern sampling ecosystem they can't be and should not be expected to be experts in how all the automation works. So, they've had to have trust in the processes and the fraud prevention, all that that their sample supplier brings to the table on every project. But now they're becoming awake to the idea that if I pick the wrong vendor on this project, my most important client relationships can be at risk. Now, if you're a sample vendor with your embedded data quality tools, uh they have a lens as well. And their lens is basically like, hey, we we do mass scale. We deliver a lot of volume for a lot of surveys for a lot of companies. Um we know a thing or two. And that's true, but one of their fault lines is the fact that they literally have a lot of their completes that are going through projects that don't have a researcher reviewing data. So the standard that they're evaluating against is really set in large chunks by people who aren't measuring anything. I love talking about real people. Yes, there are still real people taking surveys in our ecosystem and those folks are getting burnt out. the the friction they feel just to get through a survey that's now filled with modern traps that are designed to catch the bad guy but get in the way of their user experience. And over the last year, I've been really fascinated to see the academia world really paying a lot of attention with papers that are studying the survey research ecosystem. through their lens in academia, they actually bring a lot of deep thought to the equation, but they also may be a little disconnected from the commercial norms in the business side of survey research. So, right now, one of the narratives that seems to be dominating the discourse in the industry is around this context of AI bots or synthetic respondents as a new fraud opportunity. So for me this is a really good time for to transition into the fact that there is a sixth seat six stakeholder in this ecosystem that is often a blind spot for most of the discourse. So I'd like to officially welcome you to the concept of fraud inc. So fraud at large but certainly also in our ecosystem of online surveys it's a business. It is definitely an enterprise. It has marketing. It has research and development. Has distribution. Yes, it has tech sprints. And thankfully, it has constraints and their own vulnerabilities. So, let's dig into the business model and then I'll pivot into how this can help us build confidence. But, Fraud, Inc. is pretty mature. It's been around for many, many years. Um, and one of its sort of hallmark characteristics is it uses mainstream technology for nefarious purposes, which is to Bill's point earlier why this gets to be so difficult. So, it uses things like Telegram channels and YouTube and it actually monetizes on YouTube. Um, it productizes exploits and it sells it like SAS does for subscriptions. They have their own communities. um they do on the tech side research and development sprints. They are like a lean startup in a business model on a lot of way lot of ways that they are not trying to just hack a a single oneoff survey, but they're trying to find something that will scale and work repeat on a repeat basis. So if they fail on a particular approach, they'll abandon it and they'll pivot and find something that's stickier. Um they are indeed about scale, not precision. I know I get this way, maybe you do too, that sometimes when I have a really important study um executing, I'm worried about that study like it has to go great. I need to really nail this. So, I start really thinking about what this one study may be vulnerable to. But the fraudsters in their business model, they don't think like that. They think about the collective of thousands of simultaneous studies in the whole industry ecosystem and what percent of them can they jailbreak. They want to be repeatable and yes they are managing to a profit. They have a P&L. They want to limit expenses and they want to make these exploits uh something they can sell to many parties. So for me, when I think of this as a business model, it calms me down a little bit too because my tendency is to think of the fraudsters as this phantom at midnight in a dark room with a bunch of screens. And while there are some that are like that, the overall scale of the fraud that we're trying to understand and combat is really an organized business model. So, I talk a lot about these different channels and technologies that they use. Um, this is just a little visual that talks about some of the tech that are built for this purpose or co-opted from mainstream purposes to do this. Um, and in my view, the thing that really strikes me when we put this all on one screen was how shockingly diverse it is. um spoofing services, geo masking, um the marketplaces, how about your your browser add-ons? A lot of mainstream things are all involved in fraud hacks. So, yes, it is complex. There are a lot of technical things we need to look for. So, I love talking about some of the ways that fraud has evolved technologically along with the advent of new capabilities in the mainstream tech landscape. So, if I were to take like a 10-year view, this is pretty much how the industry conversation has evolved. Um there were bots, then VPNs, then we started talking about click farms, then data centers, and then the concept of link jumping and ghost completes. Um a couple years ago, residential proxy proxy IPs became a real tough one to flag. And then of course AI, uh chat GPT, but now the more sophisticated synthetic respondents. The one thing that we have learned as we've tracked this over the decade is it does change it. it was things work for a while and then they the fraud prevention catches up. So what we have found is the cycle where the industry narrative tends to be six to nine months behind the actual R&D. Uh so what we're talking about now may be gone already or maybe it's sort of starting to it's already reached its peak and it's starting to come down. So the question I'm just going to offer up in the context of this presentation is should we be fixated on this synthetic respondent approach when we think of fraud inc as a scaled enterprise. So that was the question I I posed to our team internally led by our chief trust officer. You know papers started coming out addressing this threat. The fear took off on LinkedIn like crazy. Started jumping in on a lot of industry conversations. So I said, 'Look, go answer this question. Is is this something we need to worry about? Um, obviously the team already had a head start and I'll explain to you why. We have this methodology that I think is part of any holistic system that's combating fraud and data quality threats overall. And the the thing that really typifies what makes it work is we're not trying to guess, oh, we heard about this new thing. Now, let's go see if it's out there and and sort of chase our tail that way. Instead, we've built a system to be able to look through this through a multiple ways and stakeholders and then quantify is this really something that is working at scale. So, for us, we use these four techniques. We we do what's called ethical hacking. We're constantly evaluating different survey tools, survey sample platforms, um, research agencies, and we're able to look through the lens of a fraudster, how they may look. We don't obviously get into any survey data. We do this ethically. We don't get any payment through this. We're just hunting for doors that are open. And then when we identify a vulner vulnerability, we take it to that stakeholder to try and slam that door shut. Uh the second is white hats. Uh my favorite analogy for that is the movie with Leonardo DiCaprio and Tom Hanks called Catch Me If You Can. And that was the idea of a a bad guy that wound up being flipped to work for the FBI. Uh we employ this concept in the tech world. It's called white hating. And basically we hire former fraudsters that have reformed to try and become fraud preventtors. We have them on our payroll. They know the playbook. They've written pieces of it. So, we actually engage them to help do the R&D with us. We also have an internal red team that involves our developers, our internal hackers, white hats, and they try and hack our own technology. Where are our vulnerabilities? We'd rather find them in-house before they show up in the fraud marketplace. And then, yes, marketplace surveillance. We spend a lot of time trying to identify where they buy and sell their new fraud techniques, where they're communicating about things that have tried and failed or the new opportunity to jour. So for us, we have found it's really helpful when you know what you're looking for to actually protect a research project. So, let's dig into the marketplace surveillance aspect a little bit. And kind of in short, it's like we're reading their mail. Um, we are watching the conversations. We are scraping it. We are mining it. We are translating it in a lot of cases, but we are keeping our ear to the ground on what's emerging. So, we took the question, are the bad guys using this synthetic AI approach to cheat us? How do we answer that? We like data. Um, so we went over the past month and we scanned 136,000 fraudster conversation messages back and forth and we wanted to see well what percentage of this 136,000 messages were relating to this successful technique around synthetic bots or synthetic respond respondents automating answers through a survey. So what did we find? Only 625 which equates to less than 1 half of 1% of the fraud marketplace conversations are AI approaches of any kind. Now in this you can look at this through two sides of the coin. one side is well are we sure we're in all the marketplaces maybe we miss something and that's always true but more the more we do this the more confident I get that the reframe is very commonly more accurate and the reframe in this case is or maybe they tested this a long time ago and it failed modern protections so they walked away and they pivoted to something else. So the next step we took is we actually had our team build their own synthetic respondents and then we let them loose on our own internal studies with our protections to see how they would hold up. And that is where we really got our confidence because we were able to prove that we were catching this not just within live client projects but when we had designed something that was trying to trick everything. our protection mechanisms was still good enough to hold up. So that little exercise is is a little microcosm of I think what needs to happen given the complexity of the ecosystem, but it's also really empowering us to do this and invest this way because of the framing of this fraud inc. We understand it as a business. So let me share with you four truths that this broader program has really revealed to us and I think this is where we can really pivot to confidence building no matter how you go about it but it'll give you a more expansive view of the way the fraudster views the industry. So truth number one, um just because there's a new tech that makes fraud technically possible doesn't mean that this is probable at scale. So our imaginations run wild. I'm certainly have days like that too where we can imagine every threat. But fraudsters only ship that which can scale. So what's possible are things like custom AI agents, synthetic respondents, novel device farms, train chat GPT bots, and any of a hundred other things that we're not even talking about here. But right now, what is actually scaling in the fraud inc marketplace are things that are repeatably exploitable, right? That have no friction on automated tech. Um things that can be packaged in a friendly little kit and sold to new fraudsters emerging into the community. And also surveys that are not secure and that do not have researchers in the loop reviewing data. They just don't want anything in there at all. And the the less that they have terms of catching the techniques, the more scalable it is. Truth two, fraud is not random crime. It is a forprofit endeavor. So the antidote to all of this really starts with consider the economics. Make your studies expensive to attack. So what are profitable hacks for fraudsters? Insecure DIY tool utilization, automated handoffs. These are oxygen for the fraud marketplace. Um be aware of distribution vehicles Telegram YouTube mainstream tech that are not meant to be fraud but are being co-opted. um they growth hack their economy which means they are literally advertising what they're doing. They're trying to sell this uh the payment process. So this is crypto but they're looking for things that are the most attractive which is fast payments that are not secure. Now this is a lens to look at vendor vetting on the sample side or new panels that didn't exist two years ago. That could be a real indication of a vulnerable part of the ecosystem. Third truth, fraud starts as human assisted and if it can't automate, it doesn't scale. So why does the human in the loop matter? Because that's the most effective it'll ever be. Um, one of the features of this friction that we want to bring to the fraud world is how many different parties are involved in a single study. You have all these different access panel companies or sample companies. You have different survey tools. You have all different approaches in the market research agencies. Uh a new automated technology has to navigate all of the differences, all of the techniques that are embedded in all of those parties. So they expect to fail when they start something new, but they want to fail fast and then ultimately find something they can scale. So if they're not on the third sprint, you know, validating a new novel approach to get through all these different combinations, they'll give up. So this unique pairing of factors is a maze for the fraudster. That is desirable friction. I would recommend we all double down on that complexity. It is a value prop for us in the data quality battle. The fourth truth is fraud is not a single thing. I I think this is becoming mainstream understanding at this point. Um it's not just a bot. It's not just stealing an IP address. It is a combination of techniques that are device that are network that are behavioral. Um which is actually really good. It's it's complexity is also an opportunity for detection and mitigation. Um, so the I think broad understanding for people that are not directly in the industry every day is they tend to think that it's just basically a bot blocker at the beginning of a survey and then researchers manually scrubbing data and then if something goes wrong, they're in reactive land. Um, but for me, I think where we're really getting to from a mainstream standpoint, if this doesn't reflect your process, I encourage you to embrace this is be more holistic. Um, I understand the diversity of the threats and where they manifest. It's not just device fraud, but data quality threats include real people being fatigued by the experience. Yes, it is chat GPT answers and behavioral fraud. So we need to bring to every survey this cyber security grade um mitigation and if not that is part of the vulnerability. But through our sort of holistic proactive approach you have the opportunity to be real methodical in your upgrades and not reactive and certainly not panicked. So ultimately if you take a systems mindset that we really advocate for you have to define your core outcome like how do we know this worked. So an example would be for us is the data good enough for a researcher to keep and then letting the clients be the experts that grade us on that. That is the core outcome that we anchor the whole system against. What's yours? So, the playbook, I'll just give this to you free right now, and if you need any help um digging in and how this relates to your business, we could certainly do that later. But the number one overarching thing is, are you bringing enough friction to your project approach? Um for us, we think we have a real smart systemsbased thinking approach. We validate the researcher expertise. We have a very diverse large group of signals that identify potential threats and we measure all of that on a study level and all that can been accumulated to really tell a data quality story on a project and then ultimately the feedback loop every study makes the next one better at fraud prevention. We get um just absolute improvement is baked into the process overall. So the number one takeaway if you remember nothing else friction is your edge but so is your judgment. You have experience that really matters here. So depending on where you sit in the ecosystem, if you're a market research agency, I would say, you know, up the competitive advantage of friction. Um stop playing defense. Get ahead of the game. If you're a brandside researcher that's sponsoring projects, you know, don't run DIY experiments alone. bring a researcher either internally on your team or through a market research agency to the process. Apply the judgment, the expert eyes to cleaning all data. And overall, this is I would advocate for as being the best way to bring some friction to fraud ink. All right, Bill, kick it back to you and uh would love to answer if there's any questions. Absolutely. Uh yeah, so if you have questions, definitely drop those in the chat or the Q&A. Um looks like we've got one or two already waiting, so we'll get rolling on those. Um so question number one, does text stacking work better or worse for quality? Meaning the more quality platforms that are used can be better because they all specialize in certain areas, or does this just filter out too many good response? very important topic. I I write about this a lot and um the second half of that question sort of clarified an angle but tech stacking generally has two different meanings like one is I'm really afraid of device fraud um like case for quality demonstrated they all kind of give different scores they don't 100% align so I'm just going to put three of them on every study right uh I would absolutely recommend against that is an total totally ineffective approach that what it ends up being is is paralysis. It's it's indecision. You're trying to navigate deep, you know, flags and things you can't see to try and make it make sense, and it almost never does. So, I would highly recommend not doing that. But if you're talking about which we actually do, so I definitely advocate for the idea of having different technologies in a stack on a project that are all looking for different aspects of data quality threats and then they talk to each other through a data story. That is absolutely a wise way to go. I totally advocate for that. um that is the connective tissue throughout the beginning to the very end of the project and it ultimately tells because so much of this is so complex that you need a trail of breadcrumbs that collectively tell the broader story. So yeah, if by tech stacking you mean having fit for purpose at different phases of the project pre-ervey and survey and post survey 100% I think that's the right way to go. >> Looks like we have another one. Uh, every quality platform out there has their own type of quality rating system. Um, how do we know which one is right and ultimately how should we be judging a quality score for each project? Yeah, that's the outcome that that's the outcome question I had in there is uh it is a hopeless exercise if you're trying to bring all these in and then measure which one is right because guess what? Each vendor will tell you they're right. you know, like this is the right one. This and this is why. Um, and ultimately it's it's again part of that paralysis. So for me, I always let the survey data tell the story. Like >> the researcher vantage point is ultimately the outcome that we anchor against. So we shouldn't have to passionately advocate to you why our scoring system is right. And we're not going to tell you what it is, but you know, just trust us, right? This is not a trustbased, it's the trust but verify based error that we're in. So for me, the verification comes from the the survey data itself. And then quantifying across projects, is this really working for me? Look, the truth of the matter is if you get a dashboard that says we've blocked X number of intrusions, but I'm still spending 10 hours of project scrubbing out data manually, it isn't working. I don't care how elegant the scoring is, it isn't working for you. And ultimately that's all that matters. >> So I guess from a you know perfect world standpoint you know obviously the the goal of anybody involved with research is going to be to to eliminate um is elimination or you know a down the road point where fraud does not exist a possibility? Uh, Bill, I'm sorry to say, um, no, it's not going anywhere. We we will not eradicate this. And and this is not unique to market research either. It's not going away in banking. It's it's not going away in e-commerce sites. It's not going away in social media. Fraud is part of the digital economy. And we were lucky because they didn't really find us as an ecosystem at scale until COVID when everyone had time on their hands and a need for money, right? So, but that that bridge has been crossed. So, to me the there is some corner of the sector that does show frustration out loud about if there's it's zero tolerance in their mind. Um, that's just not reality. And I'm the guy that invests the eight figures to mitigate all this stuff, right? Um, but ultimately, I'm trying to be real and and from a humble standpoint, I'm just saying we're big business now. Survey research is big business. So, we should expect it will attract people who are trying to extract profit out of it through nefarious purposes. Um, that is a consequence of our wild growth as a sector. So, in a way, we should feel good about that. But it's also a warning call that do not send a survey out to that ecosystem without the proper protections. So I guess I know we we've talked about fraud, but I mean quality being even even broader conversation like if I am my if I'm looking at bad data, you know, do I just automatically assume that it is fraud or there other, you know, things that I should be on on the lookout for? Yeah, I I think quality and fraud tend to be used as synonyms and they're just not like there's data quality is the overarching theme. Fraud is a component. Um, this is becoming really mainstream as an as a conversation at least because the end clients are so much more aware. They're getting into the data. They're secondguessing everything their agencies do in a lot of the verticals. Um, but to me, things that may look like fraud are are actually could be a sign of a real person who was just checked out, right? On their cell phone doing 50 other things, just trying to race through to get their incentive. Um, so for me, real people who are fatigued in the survey often present as fraud. And to me, the distinction is not just semantics. like it's important because it's a different thing when you know you got a real person in there but they were checked out. We can do something about that. Like we can make the survey more engaging. We can add gamification. We could do some fun things to resolve that to mitigate that. Uh but if you have a vendor that's sending you just outright fraud, well the trust, you know, sort of issues go really deep now and you start questioning everything. So the nuance I believe is important to just understanding the broad landscape. But yeah, responded disengagement. Um certain settings on studies like someone being allowed back in the survey is often done without thinking twice about it because people haven't gone that deep in their process. There are all kinds of things that happen that are not fraud but get labeled that way. So yeah, for us data quality is a holistic thing. Fraud is what everyone wants to talk about. AI everyone wants to talk about, but it's actually way more than just that. And we try and address it all. >> Yeah. I mean, one of my biggest fears right now is is participant experience and us as a whole just, I guess, quality controlling the good and well-meaning participants who really want to just, you know, share their opinions out of the equation. Um, it's it's a balancing act that I think we're we're kind of struggling with as an industry. >> Yeah. I I with you I I've said in the past, Bill, but I don't like giving the whole class attention, right? Just because there's a bad actor, I don't want to punish the whole class. So, I tend to advocate for more, you know, and this is why we invest in it because we believe in it. But more passive checks where it makes sense, right? So, some of the in-your-face in survey checks which I have been recommending for 15 years, I am now rethinking for that very reason. I think we have to view the real people taking surveys like you do Bill as like so valuable an asset. They are the absolute diamonds of the whole ecosystem. We have to now start crafting I think you guys do a great job with this bill but crafting an experience that works for them right and yeah not give up fraud mitigation but do that a little in a little more advanced way so you can protect that real respondent experience too. Yeah, I think I don't know years ago uh when online sample was just you know in its heyday and we we grew accustomed to the ease of you know finding people to take part in our research and that's you know that's certainly changed now and we have to be be aware of that um in addition to all the other vulnerabilities and threats that are out there. Yeah. And and to me the process if I raise my hand say sure I'll take a survey now before they even get to the survey like what's embedded in the automation technology that's a gauntlet right so they're spending 15 20 minutes answering redundant questions about themselves before they even get to the project right um that's not a small part of this problem and and it's it's a sticky one for sure but again kudos to you guys for having figured that out too and It certainly shows up in the quantification of your quality. >> Yeah. I don't know. Figuring out is it's a it's a you know an elusive target. That's for sure. Um so the concept of what was it white hat the the white hat idea that you mentioned. Can you can you speak a little more on that? >> Yeah. I mean, think about all these fraudsters, oneoff people in some corner of the world that are making really good money. And in some cases, like we've interviewed for ASMR and for other things that we've done at a fraudster in Bangladesh or fraudster in Venezuela, they're making real money doing this like inc like supporting their entire family off of this. And at some point, they have a realization that whole, you know, I am committing international fraud here, right? This is a crime. if I get caught, I'm in a jackpot. And they that they get scared, right? So they wind up actually flipping and trying to monetize their knowledge of this for the companies like us that are trying to mitigate it all. So we love finding those people at that moment of conversion or maybe a little later and then bringing them in and then they teach us. They teach us what we don't know. We're not doing this full-time. We have a system to scrape and go explore what's going on. But we need people really in it in the dark boards on their own devices, not on company network that are going in and studying all this and then working with our internal engineering team to see well this is what everyone's talking about. Can that beat an opinion round survey? And that's where um the red teaming comes in where the white hats educate us on what's working and what's not working, where the vulnerabilities are, and then we go see if we're vulnerable. We we'd rather find out on our own valition, right, than like seeing it on some board somewhere um after it's already done damage. So that's that's kind of how we do it. The white hat is it's common. It's not something I invented for sure, but it's common in the cyber security tech landscape as a best practice that we're happy to bring the market research industry. >> Awesome. >> All right. Any final questions for Terrence before we let him go back to investigating and and solving the uh the world's fraud issues? >> That's all we got. Terrence apparently. >> Yeah, absolutely. Uh thank you so much. Um everybody, we're going to go dark for a few minutes uh while we hang out for the next presentation which will begin at the top of the hour. So check your email, grab a bite or a cup of coffee, and we will circle back in about 15 minutes. Thanks. >> Thank you all.