Battling Bots and Other Online Predators in Market Research

all right uh welcome everybody to the December accelerant research virtual insights mini conference uh my name is Bill McDow I am president at accelerant research I'm going to get us started for today's festivities um if you have joined our virtual conferences in the past uh welcome back if you're joining for the first time welcome um if you have been here before you kind of know the drill uh what these are all about but I'm going to just give a quick overview and then we're going to jump into some cont ENT uh so excelerant research my firm we host uh and have for the past several years ever since CO as a matter of fact uh hosted these virtual insights conferences um our tendency has been to do these couple of times maybe three times a year basically set up a dayong set of back-to-back webinars um but we are listening to what the people want and also trying to uh offer up this content on a bit more frequent a bit more bite-sized basis so um starting today we're going to introduce um try to do these on a monthly basis but uh these more mini bite-sized virtual insights webinar type conferences um but format the same excellent researchers will host we bring in good speakers uh talk about some interesting content uh and you know current research and insights events and hopefully give each other the opportunity to network together get to know one another uh maybe learn something from other research Geeks um first I'm going to go through some quick ground rules um these are the rules that we tend to operate by uh we are and continue to remind each other that we are at the mercy of tech excellent research is a research agency hosting a webinar so we're not an event planning or or or conference organization so you know if something were to crap out tech-wise if uh you know internet glitch hits us uh please be patient um it's rare that these things happen we're pretty good with uh reliability of the zooms of the world but you never know um so also reminder just because this is a virtual conference uh virtual event um just you know don't be too emboldened by the anonymity of the internet uh try to to be be respectful of one another of our presenters um that doesn't mean that you shouldn't engage with one another Network to the extent possible yes we definitely want that that's the whole point of uh getting together sharing knowledge um but do so respect respectfully excuse me um like I said conference do try to network with one another we're not in person but we'll try to you know simulate that to the extent possible so uh if you're in Zoom if you're on YouTube take advantage of the chat functions that that are ahead of you uh if you're in Zoom Q&A um ask questions to presenter definitely you know keep us posted engage engag with one another right um if you're on the socials grab hash arvic Twitter LinkedIn and you know network with one another to the extent possible um like I said these are going to be happening on a more frequent basis uh we've got our next few uh events stacked up which I will quickly go through and then we'll jump into some content uh next month January we've got an interesting uh webinar coming your way on the new business of ux research uh we've got a March full day conference already scheduled for the 27th probably sprinkle in a February event as well um so you know if you haveen already please do go to excelent research.com resources uh conferences register be on the lookout for additional information but today's event um is going to be a webinar by on its own Julie pavit uh she's going to be talking to us about battling Bots and other online Predators um a subject that if you're in research insights and you're not paying attention to you definitely need to be because it's out there it's a problem it's something that we as an industry need to be thinking about conscious of and uh kind of taking a village to to battle this thing so I'm gonna get out of the way I'm gonna stop sharing I'm gonna let Julie do her thing and take it away alrighty hi everybody thank you so much bill for the intro I'm excited to get started here I guess I just need to share my presentation and make sure that it is in slideshow mode you guys see that good all right all right well thanks so much for coming today uh I have a lot of content to cover I hope that um this finds uh you find this very interesting um it was great putting it together I learned a lot and uh definitely want to uh impart what I've heard and also what I've lived so I've been in market research for really long long time I've worked at neelen for over a decade I've been in businessto business uh research with Johnson Controls I had my own Agency for a while so basically um I've run the gamut and uh so hopefully like I said I can impart some of my knowledge but I also relied on some of the folks that I know to interview to talk about uh some of the um things that are going on in the industry that I think um you know are very important we need to shine some light on what's going on here so um first uh let me just give you the agenda so I'm going to talk really briefly just about you know the industry in general what's going on with the panel companies because A lot's changed especially you know in the last 20 years obviously um there are a lot of threats there are human threats and there are non-human threats and we're going to talk about all of those uh the vast majority of what I'm going to be talking about today is going to be in this third section in defenses so I've got all kinds of uh tips and tricks tricks and techniques uh that we're going to cover and you can try them all um in order to try and you know make sure that we keep the Integrity um of our data in our research intact and then finally we're going to talk about some inventive incentives because as we know the whole reason there's all this problem is because of the money right so follow the money uh let's make it not so easy for people to just take that Amazon gift card let's um you know figure out other ways that we can incent um participants people that are willing to give their opinion to us uh without making it so um you know attractive to the bad guys so let's get started all right so what's going on with market research data why is it so important it's it's really super important to the whole industry and also for anybody that relies on Market Research right to make good decisions um we want to talk to human people right exactly uh we do not want to talk to bots um I think it's you know um absurd that we're having to have to deal with this but it's the fact of reality and and so you know we're going to deal with it and we're going to talk about that today and how how to get away from those non-human people um we want to Ure that when we're providing data to people they're making good decisions not bad I mean I've read you know that there's just 30% maybe of the research data that comes back is from Bots I mean that's just totally unacceptable and uh we don't want our clients and we don't want your internal clients or you know people trying to create new products or you know deal with customers or you know increase satisfaction or or run an ad we don't want those decisions to be based on fraudulent data so it's really important the data is you know um Quality um also you know we're wasting a lot of money uh on giving incentives to these bad guys so let's stop doing that I mean if you think about a survey a th000 people you know and maybe if 30% of them are bad that's 300 if you're giving them each you know 25 bucks I mean you know do the math that's wasted money down the drain and not only are you getting bad data but you're just throwing money away and then finally you know it's the reputation of our industry I've been doing market research for a long time and I really really love to help clients and help people make great decisions based on Research I know it works so you know anytime there's something that's good to affect that you know I take that kind of personally and um all of us should you know want to keep the Integrity of our reputation intact so you know that's that's sort of my take on why I think the quality is so important um and what's going on with the industry in general as I mentioned I had been at uh neelen for over a decade I worked in syndicated uh research side and we did literally 160,000 survey is a year for some of our syndicated research projects um in order to create derivative products but we also sold the market research data um back then that was when people were just changing from caddy or phone surveys over to online surveys Bots weren't such a problem at the time um but there were some panel companies that were up and coming we used SSI um they were a great partner for us um and but over the years you could see that there are a lot of panel companies that have come and gone there's been a massive amount of mergers and Acquisitions I mean companies that existed 5 years ago don't exist anymore and when I reached out to my um colleagues and talked to people that have also been in the industry for 20 30 years you know they're just saying wow it's like night and day from you know how it used to be um there's a lot fewer proprietary panels I mean um we have one at accelerent research and it takes a lot of care and feeding in order to keep the quality up but uh a lot of people have decided you know it's just not worth it to do it so they're throwing in with some of these big guys um that you know tends to be a problem because then the quality is diminished um there's a lot of uh of these big panel companies also that provide sort of hey click this button and tell us do you want men you know age this to this what was the income uh we'll send you your data you don't even talk to like a human being anymore when you're ordering your panel data uh so that's got to affect quality as well because you don't really have a partner helping you sort of make good decisions um there's also different kind of sample sources that are out there now these River samples which um you know are basically you're giving your survey to somebody who just happens to be surfing on the web and there isn't a lot of control over who that person is um there are synthetic panels now that are based wholly on you know predictive um outcomes and and obviously we have this new active threat of survey by and you know other cheaters that are just trying to take advantage of us so uh let's just you know keep in mind that the industry is changed and we need to know what's going on in order for us to be able to um really defend against some of these Bad actors um so who are the threats um I kind of put these three up together because they're more like the systematic um or systemic um problems that we have survey Bots huge we're going to keep talking about that um click Farms let's talk about those um click Farms are can generally usually are in foreign countries but there are some in the US um it's a bunch of people in front of their computer sometimes they're all in one room sometimes they're working remotely and this is a problem I mean I if you start looking and doing some research you'll be able to see articles and I found a great one on greenbook where you can go and and look at it's from 2019 where um these really uh inventive guys went out and figured F out that there was some bad data going on so they investigated further and they found out there was a click Farm in China and in that click Farm um this guy was actually charging his students students 150 bucks a pop in order to learn how to take surveys to make money so um you know that's that's a huge problem so you've got humans taking the surveys but they're trying to do it for the money only and obviously they're not qualified response right um another thing that's come up you know fairly recently are these things uh called ghost completes or redirect fraud and we're going to talk about a little bit more about those in a second but these are what I consider to be kind of the overall sort of systemic issues that that are real threats um to our surveys and to our data um there are also human threats right so we've got these guys uh that are professional survey taker so every panel that's out there they sign up sign up sign up they want to make sure they're taking as many surveys as possible to give as many incentives as possible now they might qualify uh for some of the surveys and they might be legit in some regard but um my take is that you know if they're professionally uh taking these surveys they're just trying to figure out you know how to get the money um same with cheaters and what what we used to call gamers where they trying to game the system right um these are people who will get an invitation to a survey and then try to figure out like what are they trying to get so they're trying to guess what the right answers are um so they're you know just as bad right because they don't really qualify for the survey so we don't want to talk to them um and then finally you've got people that are legitimately um trying to take the survey but maybe they're distracted U maybe they're just lazy they want to get through the survey fast so you know they're they're as much of a problem so these are what I consider to be kind of some of the human threats so what are we going to do about all this um as I mentioned most of the time I'm going to be talking about these defense strategies and that's what we're going to talk about the rest of uh our time together here um and basically I broke it down into three sections so you've got this pre-s survey in the survey and then post survey before during and after right um but I like to consider them to be sort of like deny trap and then toss them so first we're going to deny them even getting into the panel or getting into the survey so we're going to put up defenses and we're going to try and keep them out totally right right that would solve the problem but people get through so then we're going to create some traps and we're going to try and see any of those Bad actors try to get in or any of those Bots we're GNA um put some traps in our survey in order to catch them and then if they still make it through um then we're GNA find them at the end and we're gonna toss them out so deny trap and toss and that's how we're going to structure the rest of this presentation all right so let's talk about denying um as I mentioned there are lots of survey panels out there um and the best thing to do is try and not even have Bad actors or survey Bots get in in the first place um there are automatic ways to do that there are some manual processes and then as I mentioned I'm going to talk about these ghost completes so let's talk about automatic ways now there are several um uh software products out there that are bot detectors and these aren't just for research I mean there's bot detection for you know other problems that Bots are are causing in you know advertising and other things as well but for the purposes of our discussion today we're going to talk just about the survey bot detection here are three on the left that came up when I was doing some research um and they do they do a lot of things right so they have like behavioral things they look at how you're using your computer how fast you're typing um things like that um there is also what we call like uh fingerprinting so they're trying to ID where the technical stuff is coming from and so that can be very helpful um there are also challenges and we're going to talk about some of those uh techniques and they're using them as well to try to keep people out of their panels um there's pattern recognition so you can sort of see when people are doing certain things you know that that's probably not a legitimate respondent and then finally one of the things they can do is they can look at your open ends and tell you if that um uh content in that open it actually had been translated from a foreign language so that can come in handy too so that's some of the ways that the bot software can help but there's also a fair amount I feel of manual intervention because with the software I've heard people say they've had to use like two software programs in order to clean up their data I've heard people say they've used the software and still they find um duplicates when they do the manual um intervention so so you know none of the techniques I'm going to tell you are sort of a silver bullet um it's all multiple techniques that are going to need to happen um but just know that if you're using one of those software um products or if you are hiring a panel that uses one um you know you'll want to also find out if they're doing any kind of manual intervention afterward of to ensure that they're really scrubbing the data um but let's talk about some of the manual processes now so um obviously one of the things you can do is look for duplicate IP addresses um you can filter out any bad email addresses names uh phone numbers like phone numbers with three digits instead of 10 um you can look at geography there's oftentimes in the surway software um you can see the longitude and latitude and you can sort of match that up with where people say they're from so that is helpful um on our panel uh we do encourage people to refer other people and so uh you know that's a great way for us to get more panelists that are qualified but you know if we find that there's one person that's referring like 50 people that might be a flag that we want to check on um and then of course we've got our capture uh so let's talk about capture all right so capture everybody's seen him right everybody is um looking for headlight or excuse me for stop lights bicycles motorcycles right because really what we're doing is we're helping all the uh the driverless cars out uh as part of that process um I did find these couple of other captas that I thought were super cute and I wanted to pull one into the presentation um but I didn't know which one because they're both cute right so I I sent out both of these in a poll to decide which one uh would be better and actually within our company all the salespeople picked the muffin all of the uh all of the operations folks pick the bread so I don't know I just used them both I thought they were cute uh but let's talk about capture right so what's the the pros to using it is that it does capture like simple Bots um but it's definitely not bulletproof especially after all this time I think they're on like capture four or five now they continue to uh make improvements because the Bots are learning um some of the cons to capture are that um you know some humans are failing it and maybe they get frustrated um the other thing is that um the more sophisticated Bots can definitely pass these and then I learned about something else that I sort of couldn't believe but there are things called capture farms and there here's two here's some logos for two depths by capture and U two capture you can go on these websites and they literally say all right if you need one you can set up an API interface with us um tell us when you need to solve one and you know for like two bucks a day we'll solve you know 10 of your captas so um you know go figure people just find the odest ways to make money um but apparently they're doing this so it's something you should probably know about all right what about these ghost completes so this is also I think kind of a fairly new phenomenon and basically what it is is that you fin you get a completed survey but there's no data in it you're like wow what's going on how did this happen um basically what happens is a bad actor finds your redirect link um so when you create a survey say you're in a uh a company and you need to hire a panel to um to host your survey so you go into qual tricks and you do all your programming you say hey Mr pan company um here's my here's my redirect link go ahead and launch this on your site well what happens is somebody some bad actor gets a hold of that link and they're able to change the um the coding on it so it looks like it's completed but there's no there's no data in it so all they're trying to do is get the complete and then get the incentive and so you know that's bad obviously and so one way you can protect yourself is just have the U the panel company actually um program your survey for you that way there's no link at all to be stolen um and if you don't want to do that and you want to continue to program your own survey what you can do is encrypt your links or you know have a direct um you know link to the panel company so there's no link to Steel so all right so that's how we kind of keep them out of the pool to begin with but if that doesn't work and you know history is showing us right now it's not working um as I said there's like 30% people are telling me in the industry of bad survey responses when they launch a survey through a panel I've heard as high as 50% that's often an answer I get and I've even heard 70% which is absolutely unacceptable um so obviously um Bad actors are getting the panel so we have to do something else so the Second Step remember I told you we're going to deny then we're going to trap so let's talk about like 10 ways we can trap these guys um these are I'm not going to read these because we're going to go through them um all right so the first thing you can do is you can create whoops sorry I missed one the first thing you can do is you can hide the topic of your survey so um when I worked at JNS controls I did a lot of B2B research and we were always you know crafting these really great intro emails to people saying oh you know Mr construction guy or Mr architect you're so awesome um you're so great we really need you to help us with this new product that we're going to launch and and it's this widget and and so basically what you're telling them while you're trying to you know entice them to take your survey is that you're telling them the topic of the survey and you're telling them who who we're trying to find right um by doing that though um we're ALS o telling the Bots what the topic of the survey is and who we're trying to find so it's much smarter nowadays to actually just try to hide all that so bury the lead try to get them engaged and try to get them into the screener um before you sort of disclose too much so that's the hidden topic uh technique oops all right the second technique is red hairing questions now red hairing questions are basically kind of um distraction questions or misleading questions they're they're off topic from your survey right uh here's an example so what's your favorite Super Bowl snack buffalo wings nachos Ford Mustang or potato skins um you can actually get a few Bots this way believe it or not um these are getting easier and easier though for the Bots to solve so you can still try putting them in um you know and but the other thing that I read which kind of astounded me is that sometimes survey takers get offended by these questions which really surprised me um so I guess you know don't use too many um hopefully people will understand that the reason we're using these techniques is because the Bots are a problem but just know you know there are some people that have complain about them um let's go to the next one all right trap questions so I love these um basically what you're trying to do is you're trying to create a logic um thing to solve and obviously humans can because remember even though they call it AI or artificial intelligence they're not really intelligent right um they're machine Learners um so what we're going to do and we're smarter than them we're going to actually create these logic checks that they're going to fail probably um and so here's some examples how much is 2 plus 4 you know put the put the answer in as a word in the Box um we can do this we can say hey what are your initials capital letters put those in the Box a lot of bots will fail these and then the other thing to do um we've done this at actually at John's controls too is you put in sort of some fake brands um here's an example what's your favorite streaming service Netflix Amazon Prime Blockbuster Hulu obviously someone says Blockbuster um you know they're not uh probably uh human or they weren't paying attention all right let's move on attention checks um these are to make sure that you can see that people are paying attention they're not uh you know wandering um or they haven't been distracted by another family member what have you um these are actual questions and you could make these questions that are relevant to your survey um but then what you do is you put in some notes um in order to uh have the person know what to do so in this example what's your favorite Christmas song these are legitimate answers but we're telling people select response B to this question um so you're telling them you know don't tell us your answer just pick B and that's one of the checks U the other thing you can do is just tell people don't answer the question at all um and I have personally used this and have caught about five to 10% of bots this way so these are really great ways um and also you know it if someone's at their computer late at night and they're not really paying attention it will also you know flag those people all right what's the next one uh reverse and randomize so this is a technique I don't use very much but as I always doing my research I found some other people find it helpful um you can just reverse the actual categories and sometimes that trips people U trips the Bots up but you know what it also does it trips up people so um you know use these with caution you know try them out test them um the other thing you can do and I'm sure everybody does this on those Matrix questions where you have a whole bunch of statements um you could randomize it so each person gets a different um order of the questions and the attitudes that they're actually rating and that's good for a couple reasons right you don't want to have bias of people being really um paying attention the first few attitudes and then kind of going all right I'm not going to read the rest but also having them randomized also will um potentially help you from you know having problems with some of the bots so uh what's next we've got Honeypot questions okay when I brought these up to a few people nobody had ever heard of them uh so I wanted to just tell you what they are and you can decide if you want to use them or not basically they're a hidden question so what you do is you create in your survey um a question with white text on a white background and so what happens is a human can't read that right but um an actual bot will read that question and try to answer it so then you'll know that's actually not a human being that answered that question um so you can do this through like this white text deal but also I read on you know some of the the survey um software uh blogs you have there's um also information on how to actually code it like in JavaScript so you don't have to do the white text U the pros to this are that it's pretty simple and it and it does work um and it doesn't distract people so if you can't read it obviously you're not distracted by it the cons are that the sophisticated Bots are starting to pick up on this and then also I read like if people are impaired and they have a screen reader it might also pick them up so that's probably low percentage of your respon but I just wanted to mention that as well all right what else mystery terminates all right here's another way that we're kind of helping the Bots when we put in like one of those red hairing questions or a logic check or one of those uh where we're telling people that ski the question and then we terminate them immediately after that they learn that they did something that they shouldn't have done and they got terminated so we don't want to keep teaching these Bots how to be better right but we want do is we want to not let them know what question it was that got them booted out um so what you can do is you can you know have all 10 screener questions for example flag the ones that they missed and then just go ahead and boot them at the end so they don't know or you can just wait until the total end of the survey and say you know sorry you didn't make it um so you know let's make it hard for them let's not make it easy and then finally we've got um open-ended questions these are great uh highly recommend even if you've got a short survey put in one open-ended question um you have some examples here I real like what do you like about Brand X what's your go-to brand or what are your top three brands why do you prefer this logo um you know what are you dissatisfied with and uh in the post survey thing I'm going to show you about how to analyze these to help you but um you can get a lot of you can get a lot of information about how people are answering these so I always recommend putting some open ends even if you're not going to code it um up for your report or anything I really recommend that you put one in if you can um and then finally finally finally test test test right so um as I mentioned before there's no Silver Bullet um so what I would recommend you do if you're writing your own survey um try these things out if they work great if you're um you know making your survey respondence to satisfy take them out um but you know try these things and see if you have success keep track of um you know where you're getting a lot of people that are um falling down on these questions and that'll help you understand the best ways to kind of keep you know curating your um survey so they get better and better and better um so you're going to try the methods out keep you know keep the stuff that works get rid of stuff that doesn't refine it and then launch your survey to the rest of your response all right so that was all the trap part uh now we're going to talk about actually tossing people out um as I mentioned so we're going to deny and then if they get through we're going to trap them and we're g to flag them and then now we've got um some work to do in terms of trying to clean up the data at the end and I don't know if you're a data geek kind of like me um it's this is kind of the fun part um but I've got some techniques that we can use and things you can look at in order to look at the data at this point to see um what things you can try and figure out that um might indicate that you have a bad guy all right so let's go to the first one Speeders um so most of the sorry most of the um survey software that's out there allows you to not only um time the survey the whole survey but also an individual question so if you have a big Matrix question with a bunch of attitudinal statements you know how long does the take like a normal person to actually um read through all that and answer that um create averages for how long it takes to do not only the whole survey but big questions like that and then you can find the outliers and go well it's impossible for someone to have you know filled out this whole 15 minute survey in three minutes so obviously that's probably a bot so flag that person um the other thing to note is that there are different averages depending on whether or not someone um it out on a PC or on a Mac so or excuse me on a mobile device so um you know keep that in mind too you might want to have two different averages so that you can um you know make sure that you're uh comparing Apples to Apples okay the second thing um so straight liners and pattern makers so straight liners are where someone just goes ahead and puts the same answer for uh in a matrix question for all of the um attributes now you have to be careful about this because straight liners sometimes are really accurate I mean I can tell you I've had like a customer service problem in the past and then I call in the customer tech support they solved my problem they were really nice they did it fast um and then I get the inevitable how did we do survey and I have actually put excellent excellent excellent you know you John was excellent um so that could be a legitimate response so what I would say about that is when you see straight liners flag them and then look at some of the other um things on the survey how they answered them and that might help you um to make a decision about whether that person was actually um well intented or not um the other thing you can do is look at patterns um now patterns you know might look like this U you remember your SATs right something like that um now the only thing about the p patterns is that I don't know how many people are really doing this anymore and it's a little bit difficult to look at a DAT data file like an Excel spreadsheet of your data and figure out if someone's done patterns but I wanted to mention it um you know as another thing that you can look out for um another Matrix issue I'm sorry I don't know why this keeps like skipping Pages sorry about that all right another thing you can do when you're looking at a matrix question or just having a couple different questions in your survey is to look for inconsistency so one of the things that you might have in your list of attitudes that you want people to agree or not agree with is something like this you know price is the most important thing to me and then later on you might say well quality is more important than price um you can see they said strongly agree to both of those those seem to be contradicting each other um again you know look for other evidence if you're not sure maybe someone just didn't read the question or maybe they um they think both things are true I don't know um but you know don't just automatically throw someone out based on something like this um all right let's go manual detection um again so you've got your data you got your big spreadsheet and you're starting to clean up the data starting to look at it and you're going to again look at a lot of the techniques that they use to keep people out of the panels you can use in order to clean up your own data and you should um so things like looking for duplicate I uh IP addresses um people that are um finishing the survey like 3: in the morning you know I guess somebody could but that's a flag um again you might have a spike from a particular geography um one of my colleagues told me that he was doing a survey in a state and there was like this little Rinky Dink town that didn't have that high of a population all of a sudden he got 10 surveys from that area so that seemed suspect right um well it was probably not you know accurate um the other thing that you can do is look at ological demographics we saw this a lot at neelen where you've got someone who says they're like 25 years old but they have an income of $250,000 so that seems like not really plausible I suppose it's possible but just something else to look at and flag and then I was also doing some research about like what are Bots most likely to use in terms of actually filling out the survey a PC or a mobile device and it turns out they're much more prevalent in the PC now I I went on and looked and saw there were like three different sources where it looks like about half of all surveys are being taken on a mobile device so that's kind of good and the mobile devices have more like authenticators and and more uh privacy kind of things so they're a little bit harder for Bots to get in um so another thing to look at is you know concentrate if somebody took the survey by the PC um that means there may be a little more likely to be a bot so just something else to think about um and then finally with open-end analyses um so this can this can be timec consuming but it it really is important that you look at your um open ends in detail um and I've had a lot of people that I've had to throw out because of this so you might have a series of open-ended questions and then they say the same thing I love their customer service exactly the same with the same punctuation they just you know the same exact response that's suspect to me um another thing they do is and this is real indicative of AI is if you ask a question like why do you like Brand X so much they'll say the reason I like grandex so much is so they're answering the question with part of the question phraseology and that to me is really suspect um another thing is you'll just see a bunch of gabble um and nonsense in there uh you might see people key smash or just you know again a lot of um random letters or words those are obvious um some people might you know curse that's you know throw those guys out obviously um you might have cut and pasted responses or you might have like AI another experience that I had is that um we were doing some research in construction and we were actually able to see verbatim phraseology uh from the competitor's website cut and pasted right into our openend about like benefits of that particular brand um so you know unfortunately it happens but something to look at as well uh what else do you got here oh this there's also these um software programs out there called is it Ai and um I wanted to test this because I hadn't really used these very much so what I did is I took this text over here is actually the little um write up the description I had for this particular um presentation so I I took three free of these is it AI detectors this one's called scriber this one's called gpd0 and this one's called just done um these two said 0% AI generated or probability that's AI generated is 2% now I wrote this myself with my own keyboard so I know that it's real but this one thought it was 95% AI text generated so um I guess my advice to you is if you're going to use these free services um you know double check um I just thought that was a little bit funny so wanted to just caution you about some of these things um and it really didn't make sense either because look it says that it's also using um these double checking by these other software programs so anyway something to think about um but anyway so we've gone through U before I get to this conclusion slide uh we've gone through trying to get them out of the panel to begin with so they're not a problem second if they slip through what we're going to do is we're going to trap them flag them and then finally we're going to do a bunch of data scrubbing at the end and look over everything really carefully and we're going to toss them out because they shouldn't be in there right before we start paying out but I wanted to just sort of give you a conclusion to all of this which is again there's not a silver bullet to any of this stuff you really have to use your judgment you don't want to just throw everybody out because they made one little mistake um there are humans right that's the the beauty of it um you want to try to avoid you know false positives on you know flagging somebody um don't throw the baby out with the bath water and then you know if you really don't know for sure look for multiple sort of things put them maybe quarantine them look at them again afterwards um you know just be cautious don't throw everything out because then you're really going to be uh spending a lot more money you know and you kind of got to give your humans a break once in a while right all right so uh one last thing wanted to talk about and that is um trying to come up with incentives that aren't just cash or like Amazon gift cards right so um brainstorming with some other folks we kind of came up with some ideas obviously donations now I've heard s of positives and negatives I've heard one one guy tell me he like nobody ever does the donation um they always take the money um other people said oh yeah they always take it I do know that there's kind of a generational thing to with this like the Gen Z folks they really they love to volunteer and give to charity and they're really um love to do this kind of thing so maybe you know there's different audiences that are more likely to want to do the donation but um by doing that you're really taking the uh the um sort of U honey away from the survey bot because they're not going to get the money right um the other thing you could give is promo codes um you can do things like swag one of the folks I talked to said you know we have this customer attention program and we do these surveys these in-depth surveys and we just give them so give them a coffee cup you know a ball cap a t-shirt whatever um the only problem with that is that um you know you can't do that with a blind survey right um or you could provide swag that doesn't have anything to do with the brand obviously um maybe you know buy somebody a coffee um you know our our survey bot's really going to want cofy coffee coupons the other thing you could do is kind of do sort of a combination of like a coffee and a referral program uh you know those are some ideas U feel free in the chat right now to add in your own ideas um one thing that I thought was really interesting some guy wrote um that we should have people you know paying to take surveys which I thought was a very unique idea I don't know how that would work but if someone you know has some ideas about how that might work you know please put them in the chat as well um but the main thing is don't just Auto incentive Vis people um make sure that uh you're checking and cleaning all the data really well make sure you really think they're legitimate you're going to use that data in your reporting um once you do that then release your incentives but don't do that without check checking because that just you know feeds the problem right all right so I am at the end of our little discussion here I want to leave some time for Q&A got a couple of things though one um please connect with me on LinkedIn um number two if you would like to have a copy of this presentation I've got my email address there Julie P excellant research.com just go ahead and send me an email I'll be happy to give you a pdf version of this if you have questions obviously you ask them now or you can reach out to any of us at accelerant um and then and then I guess the third thing this isn't a salesy type presentation but I did want to mention we do have a a proprietary um custom panel called agura uh that we're very proud of we um spend a lot of time uh caring and feeding this panel it's you know quality is super important to us so I just wanted to mention that too um so with that being said I'm going to toss it back over to Bill and we're g to answer some of your questions all right uh questions for Julie drop those in the chat the Q&A I see that we have one from Deborah um when you have a large number of survey responses excuse me what is the balance to avoid manually reviewing so many but still maintaining quality well I guess that's sort of up to your own judgment um you could certainly do a random sample but I think that there are techniques that you can use when you're in Excel that you can highlight certain questions that you think are most likely to be able to give you a flag uh I mean that's what we've done in the p past um our regular practice is as the data comes in on a daily basis I look at it and then I code it either as you know yes it's good it's suspect and I want to look further or it's obviously bad and I'm going to throw it out so I just code them red yellow green and you know it makes the rest of the um makes the rest of the h scrutinizing a little bit less timec consuming um again the other thing you can do is you can just pull the open-ended questions you can just do a look at the time um you know look at some of the techniques that we use and uh really decide which ones you think would be the best judge of what's good and what's bad uh but that's how I've personally done it in the past and you know I guess you know it's really up to you yeah and other questions feel free to drop them even your own experiences um pop them in the Q&A chat um you know I think Ang Julie you captured this in the presentation but the spirit of this is like we're all kind of in this together if you haven't experienced um any of these issues thus far uh news flash they're there uh they just haven't made their way to you yet um yeah I mean it's it's kind of terrifying out there no matter where you sit in research and insights you're you're exposed to this um So You're vulnerable whether you're you know a provider A supplier academic uh you know a a corporate researcher or even end ususer of of research um you need to be aware of you know the environment and what we're up against here um and it is you know we've gotten past the point where there's anything that's that's Silver Bullet this it takes a multi method sort of multi-dimensional approach to to to battle this and it is very much an goinging um and unfortunately highly Innovative space um cheating on survey research um is just a constant sort of cat and mouse like one you know you you shut down one Avenue and another one is popping up uh it's it's scary out there so you know yeah and another thing I would add to that sorry Bill the other thing I would add to that is now that you know that this stuff is going on you'll know what questions to ask your panel company what are are you doing are you doing this are you doing that if your boss comes to you say your CEO comes to you and says hey I'm giving you uh you know $50,000 to do this market research project uh but I've been reading about this stuff in the New York Times about these survey Bots how do I know that there's survey Bots well you have the ammunition now to tell that person hey these are the things that we're doing we know how to detect these Bots we're we're taking these measures so you're giving Integrity to your own projects and your own you know um uh work so that's really important too is not just that you're doing the work but also that you're armed with all the knowledge of hey you know what's going on now you know what to look for you know how to detect them you can answer those questions I think that's really empowering right I mean just being the research grownup in the room right you've got to also you know work on the balance of and you mentioned it several times Julie but talking about you know just participant experience you know it's traps and your filters and automated techniques are emerging and becoming more and more they are not perfect and if we're not careful and if we're not checking the checkers we run the risk potentially of you know alienating good respondents who are well-intended who just you know may fat finger something or may like you know not answer in the way that the AI that we're using to try to to beat the AI is is checking for right so I mean just a lot of just you know just being after it and and being on top of this it it's so much fun we it's like you just want to do research but you know hey this is what we have to deal with um looks like a few more questions are coming through uh We've Got Jake asking have you ever weeded someone out of a survey uh refused to compensate them and then had and then had them come back to you if so how do you handle um I personally haven't but Bill you probably have had that happen to you maybe yeah absolutely um and it's you know in many cases the cheaters can be very um bold in terms of even you know lodging complaints when they know that they are in the wrong so you know to whatever extent you can and I guess that's the beauty of of survey research is that you've got digital receipts of everything so if you have a strong case for why you are dismissing a given participant then you can use that um but again that's you know Julie's point about having multiple checkpoints and just because someone fails one doesn't necessarily mean that they should be booted or not entitled to to their you know compensation and it's this it depends sort of balancing act right yeah see we have uh from ronado uh thoughts on protocols between B2B versus consumer surveys um with regard to like trying to determine if they're Bots or not I mean I I think a lot of the stuff applies the the thing that you're going to have different is that you're not going to use the same checks right so with the consumer research you can look at how they answer the demographic questions and and where they live and that kind of stuff the thing with B2B research that I found really works well are those H those um those trap question the logic checks um and also putting in the fake brands like to give you an example at Johnson Controls we did a lot of research on our big giant chillers right the these are hundred thousand you know hundreds of thousands of dollars these big giant chillers and so one of the first questions you know that we ask people is what brands have they heard of of chillers and we put like Sears in there and somebody you know answered Sears I mean obviously you know they don't know what they're talking about we should you know get rid of them so using the brand names worked and that actually worked in another one I did for construction as well uh we were asking about um you know commercial overhead doors and we were asking people their top uh three brands of where they've purchased them in the past um so we were talking to Architects right and they said um you know Google excuse me they said Amazon or you know uh question you know answered Brands like that which the um Outlets would be absolutely not available to these people so using brand to try to detect is a really great way and also the open-ends is a great way on B2B to check people when they're um when you're trying to get people to elicit why they like a certain brand or why they don't generally looking at the open ends will give you a really big clue that they're um faking it or it's like AI text or they're stealing it from another site I'll pck you back on on one thing with regard to rado's question on on B2B also I think Jim mentions a similar with regard to just you know super low incidence populations um these are areas that tend to be more attractive to the the cheaters the fakers because B2B super low incidents those are areas that tend to come with the highest incentives right so in my experience we tend to see a larger proportion of the bad traffic in these basically higher incentive types of of populations so you know the due diligence is even more important with with these more difficult populations yeah yeah but you know the other thing is there's a lot of technical stuff I mean that the average bot or person won't be able to answer because you need to be in the industry in order to answer the question so that's another great indicator too it just make the you know make the questions very specific to that industry and then it's a lot harder for people to fake uh B recommendation I can give is open dialogue um talking with your you know your clients talking with your subcontractors you know to the extent that we can all be communicating with one another and trying to go after this um you know we're at a point in this industry where there should never be any ruffled feathers over someone challenging uh data that are being produced so you know to the extent that you have to answer for anything that you are delivering that's that's kind of part for the course uh just checking to see if we have any other questions coming through think we are winding down um I know there was a question or two about availability of this presentation for download uh yes you will have access to both you know the PDF slide presentation as well as archive videos um or video rather uh give us a day or two and we'll have the uh those posted for you yeah on YouTube too on our YouTube channel yep indeed yeah uh Julie it's your Forum so any parting shots that you you'd like to give uh oh we have another question has Exel tried any of the software for bot detection I'll let you feel that one Bill yes uh yeah multiple um I think it's you know there are there are good ones and there are more emerging and I think it's important to just assess as many as many as possible um but again silver bullets aren't happening right you know thus far there is not anything that is a you know one shot you know this is going to cure all of it right um it's important to be hitting this from multiple angles yes the detection you know yes the trick questions and yes you know rolling up your sleeves and getting your hands dirty with regard to like getting into the data making sure that it's it's consistent that it's making sense um but again that Balancing Act without like you know over engineering your research to the point where you're you know water watering down your your your survey product right you still do need to you know produce good questionnaires to answer strategic questions and it's you know it's it's challenging yeah as long as we keep offering money it's going to be a problem you know indeed but you have to right I mean we have to you know compensate folks for sharing their time and opinions but if there's money to be had and certainly if there's not you know policing happening the more and more difficult we make it for these Bad actors the less attractive it becomes right you know why is this happening because we as an industry have allowed it to happen honestly right all right we wish you great research projects in the future and a merry Christmas and happy new year thanks everybody great job everybody