Engaging Survey Respondents via Email: Strategies for Higher Response Rates

looks like we are right at 4 o'clock so we can go ahead and get started with our next presentation hi everyone thank you so much for joining in on our second Quarantine virtual insights conference my name is Erica Clancy I'm a project manager at silent research our next project presenter is at John's mic he's going to be talking to us about engaging respondents via email he's the senior customer insights manager at Amazon so John were really excited to hear your presentation today before you get started I just want to go over our ground rules really quick in case you haven't heard them yet please just remember that we're at the mercy of our technology so there may be some technical difficulties so just bear with us but we are recording all these sessions and we'll be posting them and sharing them afterwards with everyone so you won't miss out on anything I just remember to keep that comment section professional and respect each other and remember that this is a conference so take advantage of the networking opportunity here pose about what you're hearing on Twitter and LinkedIn with the hashtag QV I see 20/20 and feel free to connect with each other and just shout people out if you're hearing something that you like definitely feel free to post comments in that section or questions in that section so we can hit them in the question and answer after the presentation and with that John you can go ahead I'll stop sharing my screen and you can share yours and take it away thanks I'm not super zoomed friendly so I'll do my best here no worry but that's so first of all thanks to Eric for the introduction thanks to Bill for the invitation to participate in today's conference I'm gonna talk today about these things around my face here cool I'm gonna talk today about strategies for how to engage respondents for surveys via email but a lot of this is based on standard marketing philosophy so it could have applicability to folks who maybe aren't doing email surveys and hopefully you find some some use in what today's talking covers I looked at the the chat room and I didn't recognize too many names so in case you guys aren't familiar with me my name is Jonathan in the industry for about 15 years Thank You Erica for pronouncing my name Smythe often mispronounced as Smith but that's okay Erica got it totally right I work at Amazon today I've been there a couple of years on part of the Amazon insights team and what we do is we offer full service basically customer insights studies to vendors in our in our program and the reason I'm talking today about email engagement is that I'm an email outbound bar raiser at Amazon and what that means is that I've been I've been sort of selected for my ability to have better than average performance on my email campaigns you know I millions of people a year I get hundreds of thousands of people a year responding to the surveys on our program and our campaigns typically do well better than 50 percent better than average for campaigns other than ours and we compete with marketing campaigns for instance where coupons are given out and things like that so you know to have a survey program that that can outperform a marketing campaign where you're getting buy one get one type offers is it's considered to be pretty unusual in Amazon we also have a subscription panel where our rates of performance are even even higher notably higher than our marketing campaigns so I'm gonna share there some of the the ways that I think about engaging respondents making sure that we keep them enrolled in in our program and then we don't have any problems for for the Amazon customer in general so the first thing we're gonna talk about is is it appropriate to engage by email will have basically four things to think about each of these these modules next thing is sort of like a how-to of how to engage and that's the sort of name of this this talk then we'll talk about how to survey and what to know about that and then there's a special part of our program that's micro targeting that may or may not apply to everyone out there hang on how close you can get to your customer behaviors but will touch it at the end and that's that's kind of a secret sauce as well so to start there's really four things you want to think about if you want to engage a potential respondent by email for a survey or a study first wanna consider the context then you want to consider the content then you want to consider who your target is and then you want to really think about what you're going to get out of it right and if we start by thinking of context the first thing you need to ask yourself is is it really is it really a great time to survey somebody and so that's particularly relevant with with the sort of quote-unquote uncertain times that we're facing now but it's really not just about are we under sort of like an isolation or quarantine it's really more about is the respondent in in the in the right state of mind to respond to your question so if you're doing like a Net Promoter Score survey for instance you want the response to have had some interaction with your with your product you wouldn't want to ask them that the day they bought it if they would recommend it to someone else it's just the wrong timing likewise if you're looking for people who had just gotten married if their if their wedding registry was appropriate and they're on their honeymoon probably not the right time to engage if you're looking at people who maybe saw a political debate and you're asking them two months later how how they felt about the candidates performance in the debate that they saw possibly two months ago probably not the right time to engage and so before you even go into email with with panelists potential panelists you have to ask yourself is it the right time next you have to consider what it is that's happening in their life or in the email channels potentially that you're competing with right and so in our case right we compete with marketing emails directly and customers are getting a ton of ton of marketing emails from Amazon and so we have to have a way to rise above the fray catch their attention and really hook them in and not just consider a spam but it's not just what you're competing with an email it's what you're competing with in their life right and so if it's if it's like you're trying to get maybe take the example of new moms again right you're competing with the fact that they've got this new addition to their household they've got this baby that they're working trying to trying to like care for and raise and it's a new experience for them and so it might be difficult for you to grab the attention of somebody who has multiple competing interests at mind and you really need to think about how easy or difficult will it be to get that attention touched briefly on the idea of being in the right state of mind but this is again one of those things where you want to make sure if people are I'm overly excited about something or overly disappointed about something is that really just a recency effect or is it something that you believe will will be their long-term response to whatever it is that you're trying to ask about and even in that same context you have to really really ask yourself you know is this thing that I want to dive into is this something that it would be a reasonable mechanism to get the feedback through email right you think a lot of times people want know about customer journey or they want to know about like an ideal product or a product profile and these are kind of difficult for people to respond to through a through a a survey that they might receive an invite through email on and so before you decide hey I want to I want to get all this like qualitative research just just make sure that you believe that email is the right channel to do this otherwise you might find yourself disappointed with your response rate the next thing that's super important is the content that you're trying to discover first and foremost email is a channel where anybody could be looking over their shoulder and see what they're looking at and so you want to make sure that you're going to them with something that they won't mind if somebody caught a glimpse of their email in the invite rate and so you think okay maybe somebody maybe they have unusual political views or maybe they're using some sort of sensitive personal care product right it's it's okay to engage people on those kinds of topics but you have to be considerate on how you engage them so that they don't run the risk of potentially being embarrassed by something that they wouldn't want people to know about a classic example might be like a smoking cessation product or something like that where people might be trying to quit they don't want people to know that they smoke and and if somebody saw hey you're taking smoking cessation you know that might be inappropriate so always think about if it's if it's the right thing to put into email and think about how you would do it the next thing that's super super critical is that the person has to the the potential respondent has to really quickly understand what it is you're looking for from a very short amount of information and assess if if they want to give you feedback right if if you bury it into a lot of terms you have a lot of legalese in there if you make it sort of unclear what your intentions are the response just gonna look at that they're not gonna read it they're gonna just move on you've got maybe maybe ten seconds to catch their attention and really get them to do a survey for you what that requires is that you have to have a singular focus on your research objective right you don't want to spend a lot of time talking about things that aren't really going to absolutely pair in on what it is you're trying to achieve in your research that day and to be successful at that it's actually extremely important that you're talking to people at the right level you know when I talk to people people often say I talk in a professorial way it's it's almost I mean it's not quite kind of sending but it's kind of like you know I'm like I'm like Oh hiding a lot you know and to do that in an email would probably not be effective right because most people don't want to talk to a professor right they want to talk to somebody at the bar or at the park or at church or wherever it is and so you always want to make sure that the language that you're engaging potential respondents which is is something that they can relate to that resonates with them and it feels good to them it makes them want to engage with you and and talking about engaging with you the the single most important thing I believe in how you engage with your respondents is is how you screen them in right a lot of classic market research will be based on demographics and for good reason because you're looking for these large swaths of populations that you believe have similar behavioral patterns but we're getting into a point in with technology and and the ability to sort of database actual behaviors that there are some instances where a behavior based recruiting model could be more effective it's give you an example right it's it's possible that I like using I mean I don't put will just say that I like using moisturizer and I like using a brand that's for a female right typically for a female if that was true do you really care that I'm not your target demographic if I'm actually buying your product now of course you don't want to overweight your sample but you know this is something you would have to think of as researcher is it really just about demographics anymore or is it about behavior and if it is about behavior organ or demographics how can you prove to yourself that you don't just have a professional panelist joining your survey for some kind of incentive right if you have behavior based modeling and you have a database of how people have have performed a particular behavior you're pretty safe to assume that the behavior has been then executed as you want but even then you still probably want to nest a couple questions into your surveys to ensure another thing to remember about email sort of invites is that the typical screening mechanism of calling somebody in a in like a mall intercept or calling them on a phone line to see if they might qualify every study people don't get super frustrated if they don't get taken into the study because they'll be like oh I'm sorry we're looking for different people and they're like oh that's too bad they walk on if you take somebody to a survey tool and you ask them like eighteen screen or questions and then you kick them out I will tell you from experience they are not happy right and so if you've managed to get somebody into your survey you got to do everything you can to get viable information of them something you can use but you do have to validate that you know they are who they say they are and that they can give you the information you're looking for also when you're targeting you'll you have to think about your research objectives are you looking for statistical significance which could be achieved at a very low number of people if the difference between your variables is very large or are you looking for four or take a huge number of people if the difference is very slight or are you looking for data stability which in most cases on a five-point scale if you get to about a hundred two hundred people if you double the population you're not going to see that big a difference a lot of folks get tied up on statistical stability for good reason they're going for claim support but in a lot of like marketing context people are really looking for like plateaus of opportunity and all this stuff is sort of an extrapolation to a larger population and in that sense statistics are very very powerful but they can they can over complicate your your sampling requirements to the point where you may be unable to get enough people engaged via email to meet your statistical requirements so you really have to think about what is the actual number of people you're looking for and then within that context email us as we know from polling for instance political polling they often talk about how like cellphones over-index on the younger generations and landlines over-index on the older generations and so they create these models to wait you know the different populations that they're able to sample in their surveys email it's gonna have the same issues right it's not everybody uses email at the same rates everybody responds to email at the same rates there are definitely differences they're gonna depend upon the industry or in the topic you're going after the behavior or demographic that you're going after but you need to really kind of track that initially until you have a good understanding of your population at which point you can you can sort of assess if you need to balance or relate your data and talking about rewriting your data you also want to consider if if you're looking at a product good or service that is a use or an occasion based product you have to ask yourself should I be over weighting those heavy users to to be a bigger part of my sample because they're gonna be driving more of my business you know I used to work at the different company where we we knew who some of our top sellers of our products were and so we would actually go to them at a higher frequency because they were responsible for the majority of our business and they were our most important customers that's not true for every every situation most times actually it's not true but there are cases where you would want it overweight the the heavy users of your product Twitter service and yet you have to consider that and can you access them easily but via email or are you gonna be getting the people who really aren't representative of those people if that's the case if you can't get the people who are like the high frequency users maybe email is not the right channel the other thing that you really want to think about when you're thing about email is you know we talked about how many responses you need but what are your response rates right how like if I'm sending this to a thousand people how many people are gonna respond reasonably and on what timescale are they gonna respond right because if you're saying look I need 10,000 people let's just say like a cold call for a sales pitch right you're calling somebody and you're saying hey I want to try to sell something let's say you got a 1% you know hit rate where somebody will talk to you right well if you're saying I need 10,000 people for a study right well you're gonna need to contact what is a you know a million people to get 10,000 responses are you really able to get 10,000 people to respond to your survey and especially if you think about I need the data by Monday right well a lot of people are just gonna open up their email and they're not they're not gonna they're gonna look at it and maybe they'll take the survey later maybe they're busy right now maybe they'll do it next week at work or whatever if you need responses to come in quickly you need to remember that you're going to have to enroll at an even higher rate to meet your timeline so what normally will happen is you'll enroll all these people and you'll find hey I was too aggressive on my timing or I didn't put enough people in the sample and you realize I need to go back and there's two ways you can do that one is to run additional email campaigns where you would go and you would find new populations of people to contact and generally that's advisable cuz it doesn't feel spammy but the problem is going to be that you might run out your your your well may have run dry and so you want to make sure you understand if you can you can run additional email campaigns to contact additional populations but a lot of people like to send reminders and reminders are okay in theory but in practice what's gonna happen is people get a reminder and they're like look I didn't want to respond why are you asking me again and then they'll click the opt-out button right and every every business is different but when somebody opts out of a program and they can never be heard from you again they block your email or whatever it is you want to make sure that that doesn't have a negative impact on other financials that matter to your business so if you're looking at like a subscription community of people and and all you're running is is like a it's like an insights program okay it's gonna be the the health of your panel but if you're in a company like Amazon where if somebody opts out of my survey it's gonna impact their likelihood diversity's marketing emails from Amazon in general you need to know what that costs the company and you need to over index on ensuring that you don't create a negative customer experience but a negative business experience and a lot of folks don't really think about that too much next what we'll talk about is a few things really the title of this talk is inspired by these these three terms that I've always told people to have effective marketing campaigns is just inspire inform and motivate and you know I throw an extra one on here the the old kiss right but what do I mean by these things right so to inspire someone obviously is to get them really really excited about an offer or an opportunity and it all starts in email with the subject line it's got to be short it's got to be catchy you've got maybe oh I don't know 5 to 20 words depending off their mobile device you know some folks will look at and they'll just know it's spam right off the bat you don't want to feel like spam or if you are going to have a spam format it's got to be a compelling offer for them to be interested but a lot of people stop at the subject line and they don't realize there's something that's equally as important as the subject line and that's what's called the super subject and if you've never heard of the super subject the super subject is the text that comes under your your subject line sometimes it's below it and outlook sometimes and Google or Yahoo News it's gonna be you know to the right of it and this is a huge opportunity that most people miss to really drive this inspirational connection with the potential respondent and so here in this example you can see the super subjects it's hello all that's typical that's what you expect from somebody who's got a personal relationship with you this one you see this is like a really great super subject line it's it talks about like this guy's emotional connection to this thing how this is worth more than it ever has been how emotionally connected to you I mean I don't normally read these things if they have this kind of header but I saw this and I was like I gotta read this email this sounds like it's gonna be great and of course I read it and then you have the glass example which is the one that more often than not you'll see where there's nothing right and it's not that that's bad to have nothing but it's a missed opportunity right it's better to have nothing than a bad super subject line for sure right but but if you can think about keeping your super subject line compelling and and and sort of like engaging you're gonna have more success with people opening the email and when they open the email that's how you get to you get them to continue in the process right but for them to continue you need to make sure that they don't open up the email and see this like huge block of text so reading like a book right anytime you get these these emails where there's a ton of block text in there it's really difficult so wanna continue repeating because you're like this doesn't look like it's for me and so it's extremely important in these emails that you have this sort of laser-like focus on your goal and and why why the potential respondents should be interested in what's happening and in doing that it's it's it's often a good idea to sort of engage them in some way that makes them feel like it's it's super relevant to them right so if I was doing a survey on let's say I'm doing a survey on pet food right I could go in there and I could say hey you know we're looking for folks who love their pets right and would be willing to give us feedback on a potential pet food that they might consider buying even more let's say we're looking folks who love their dogs right that's like an inspirational thing I've built this connection right but if I if I had a database and I don't but if I had a database that said that you owned a dog named Baxter right I wouldn't want to go in there and be like tell us what you think Baxter would like right I mean that's just like that's maybe too personal right like yeah I don't know you how do you know this much about me it's kind of weird right so keep it personal but generic focus on the things that you think are going to really with people and you're gonna inspire them to to read your email but as soon as they're reading your email the that you need to make sure that they understand why it's relevant to them and if they if they never get the understanding of why it's relevant to them they're not really going to connect with the questions that you have or the interest you have in the responses so it's again super important that you're helping them feel singularly important and unique but at the same time they need to know that you are also singular singularly important and unique and you can help bring their message to the sponsors of the research if you don't do both of those things they're just gonna feel like it's just again kind of like spam so you know you're gonna go in there and you're gonna you're gonna briefly explain what it is you want to achieve but you it's kind of like it's kind of like that first date right you always want to kind of make them really interested in learning more and you don't want tell them your life story and everything you know because that's gonna kind of freak them out and they're gonna be like oh man you know if you want to go for more first dates if you want to go for more more people engaging in your research keep keep it a little bit cold a little bit back right don't just tell them everything tell them enough to get them involved the next thing you want to do is you want to make sure you have them able to act you know there's this this idea that you can get them into the email and you can get excited but they can't find how to give you the feedback that they want so you want to go in there you wanna make it super super simple for them to connect with you to give them the feedback that they know you want that they want to give right um but if you don't go in there and put like a time-bound element on it or you make it feel like it's a limited time offer or it's only for the first however many people who respond um you know they might deprioritize it right so if you go in there and you say hey this is open to the first 50 people who contact us which hopefully you're you're being honest in that regard they're gonna feel more compelled to try to get in there quickly or this is gonna close on Tuesday or we're opening it till the end of the weekend that sense of urgency is also going to be something that really drives people to to action upon your request but beyond that let me see I'm kind of going on here the real ups we talked about make it easy acts when you're when you're building these things to to be simple for the potential respondents who engage you've got to remember that you're gonna have a lot of stuff that your lawyers your legal team your your company whatever are gonna want you to include but none of that matters to the potential respondent none of it right all they care about is why you're important why they're important and why it's important for them to share their information with you so if if the information that you know you have to include gets in the way of that in any way try to structure your emails so that it's it's available if they want to find it but it's not it's not in the beginning of the email where it's gonna make them want to just stop reading and then talk a little bit about kiss we all know the old friend kiss keep it super simple right so anytime you put information into these emails we talked earlier about folks on a singular goal if it's not focused on a singular goal of getting that person into your study take it out write anything even if it's just an extra word look look for like what what we like to call it Amazon weasel words words that they if they weren't in the sentence you would still have the same message and and it didn't really add anything it's just the serve window dressing but also just you know things about you know what your motivation is if it's relevant to them put it in there if your motivation for doing the study is not going to motivate them to take your study eliminate it right and in doing all this stuff right you know I mentioned earlier have been sometimes referred to as profess Oriole you you want to talk to people like they would want to talk to you like I said the church or at the park or whatever we as researchers have a tendency to use industry terms we have a tendency to use legal terms we have attendance fee dnews terms that will try to really really make it feel like the research is authentic but in so doing what happens is you create these like form letters that don't feel natural and so what what I would normally suggest is like you take your letter you have a you have a like where it has a as a has a function where you can you can put it in there and it'll read your letter out loud in a computer's voice if you can get a computer's voice to sound like a human and not like a robot you've got yourself a good email if that computer voice sounds like a robot you probably need to rework it a little bit but even better than just having great language is just the visualization of information right people don't like big blocks of text so try to use things that are nice images use buttons whenever possible break up the flow of the layout and just make it really easy for people to engage with your offer which is really your offer to take their feedback give it back to the research sponsor and make their life hopefully better now what does that look like this is kind of a sample of what if you receive an email yeah I said we send you know millions of emails a year for my program you're an Amazon customer you may have seen something that looks like this from us and you'll notice first and foremost what you're gonna get is this opportunity to opt in based on if you were inspired by the subject line alone if you're inspired by the subject line alone go ahead click that button but if you if you aren't you need to read a little bit more right and as you read a little bit more you'll get the message and then there's a bunch of legalese at the bottom but before I get to the legalese right I make sure I give you another chance to opt into the survey and why are there two chances because it's on a cell phone you'd need to scroll back to the top right on desktop maybe it shows but my cell phone nope nope you're just stuck in a bunch of text and so to go back to the top and opt and again or to get past legalese it's gonna be too much friction and so you want to have multiple opportunities for them to engage with your offer making it easy and really when you think about what the text of the email looks like and this is not a real study so it's nothing confidential but again this is an example of what it might look like you know there's a there's a very clear reason they're being contacted there's a very clear statement of what their involvement will be as well as what they will get out of it there is this sort of sense of urgency and it and it's time out in this case and then we just make it very very easy for them to get involved and this is this is a pretty straightforward template that almost anyone could leverage to have an effective email structure now just because you got somebody into your to your research doesn't mean that that you're done right yeah you really have to continue the process of understanding how to engage them with your survey itself and so there are a few things to consider usually you need to know what kind of device they're using you need to know why they agreed to take the survey you need to know how long your survey could take at the worst possible case and you need to understand what the user experience is going to be what they're gonna see and what kind of cognitive burden they're going to have to take right and so to start with the device right a lot of people don't realize it but like without thinking about it but if you think about it you know probably half the people you know are taking surveys on mobile devices but probably half the researchers you know are designing surveys for desktop devices and there's other things too tablets you know are kind of popular lexis been showing up for surveys tools you really got to think about how people are gonna be interacting with your with your survey and you really need to know the constraints under which you're going to operate right what will display on a mobile screen will display very differently on if it's in landscape or if it's in portrait most people hold their cell phones when they're browsing in portrait mode so what does that mean you really have to go in there and take the survey yourself on all the devices that you think people might interact with and so like a good example on mobile of some things you might try to avoid would be things like ranking questions right it's very hard to drag and drop boxes with ranking questions and just cognitively you never want to rank anything more than four items anyway on mobile it's gonna be especially difficult but you think about matrix questions super-efficient on desktop they display very poorly on mobile right and you think about questions with long horizontal scales things like NPS or whatever very often they'll just show the first four buttons of the scale on a ten zero to ten point scale but you're gonna over index on on the stuff that displays and people won't even realize they could have scrolled the screen to the right to get more scale options and having to do so is kind of a pain and it creates friction and it leads people more likely to to basically opt out of your your surveys and I talked just for a second about Alexa Alexa is definitely this emerging technology obviously I work at Amazon so obviously you know gotta have my sort of obligatory Alexa plug I'm just kidding maybe I'm actually not speaking on behalf of Amazon but what what the thing about Alexa is is that also has its boundaries right people you can do some really cool stuff with Alexa that you can't do in any other way you can get people in places that they will not survey Oh tell me about how how your experience is waiting in line at the amusement park okay well I can do that on Alexa but you're only gonna be able get one or two questions in because Alexa times out over time and it only has a certain amount of conditional pathologic at this point and so you do not want the experience to be cumbersome if people are trying to do surveys on Alexa think I'm more like polls right where you can get one or two key insights out of an individual and it has to be kind of in rapid succession otherwise you're gonna run into some limitations of your device the next thing to think about is what motivates these people to actually take the survey it's not always money it's not always a gift card it's not always a coupon or a discount some people are just like they'd love the idea of testing products before they're released to the public and so they just they just like that or they think it's it's really fun to to see how research is done and they'd like to do that other people think that like you know to be part of the the leading edge of innovation is like this this real like badge of pride right like oh man I remember I was in the testing group for you know this prototype before it came out some people are gonna love that and you won't even need to incent them but the key for you is understanding what and sense people for the category or topic thing about um but no matter how you incent them you have to keep it simple you know my wife loves beauty products and she was invited to do some study for for beauty products and she was about 18 minutes in and she's like you know I love this brand but this is just ridiculous and she just dropped out I don't know how much longer she had to go but it was too much and so she couldn't finish it she wasn't given what she was looking for which was this sort of feedback mechanism that her her feedback has been taken in and it will be considered and she will helps you know build the next generation of fraud products it was just work right and you know this stuff's really volunteer yeah they might get incentive but you know we're paid to go to work but we're not paid that well to answer a survey and so it's important when you do these things is that you try to keep it fun and engaging right you make it feel natural right it's got to feel like a natural experience but it's got to feel natural in a way that it just doesn't create this big cognitive burden and the idea is that when they're done with the survey right and they leave they're gonna have to say that was worth my time right because if like I used to do product testing and people might love like like a like a sandwich right but if the if the bread was oxidized or something they wouldn't know it when they're eating but when they're done right and they walk away and they're in their car and they're driving home right there's gonna be this like bitter like sort of oxidized taste in their mouth and is gonna be terrible and that's all they're gonna remember and so they're never gonna buy the same chicken same thing with how you serve a make sure that when you when you go in there the last experience they have they leave on a high note and that's what they remember now we talked about duration a lot too in my work at Amazon and for us the ideal serving length is about five to seven minutes what we found is less than five to seven minutes you're not really going to get that much out of the respondent but more than seven minutes and your people are going to start to the open-ended comments after seven minutes are like this surveys too long it's kind of a pain so so do your best to try to aim for five seven minutes for a survey if you can if you can't you know break it up into smaller surveys try to survey more frequently more often with different populations and two of the biggest like killers in extending past the five to seven minute mark are just the the abundant use of open-ended questions or like you know you know other please specify type questions and also if you have somebody popping up to go to another window to look at something and then you expect them to come back to the survey right because what can happen with pop-ups is I mean obviously with open-ended it's just difficult to type on a cell phone so just use those sparingly but with a pop-up what can happen is unless you have open into a new window it might open in the same window and then they might go back to the survey and then the survey resets and they have to start all over and that's just a terrible experience you'll never get back but even if it opens into a new window especially on mobile it's not always super easy to toggle between these windows and it kind of you kinda have to pull things down and slide things over and it just takes a lot of time and it's kind of a burden so think about what you're doing how it's impacting people for for the devices they're on and and kind of talked about this little bit on ranking right where it's like it's not that hard to rank three or four products but you know we've had we've had folks ask us to rank 15 things like good luck with that right ranking 15 products or services or anything features it's gonna be like next to impossible if you're doing more than ranking of four items like you probably just do a check all that apply for like what's important and then you know take those that they chose and ask what the most important is that's probably the best you're gonna get out of it but another thing I see a lot is that people love to change like the scales are using like oh this one's gonna be a drop-down menu and this is a five-point you know agreement scale and this is a seven-point you know like extremely a dislike extremely and this is like this is like a fill in the blank for a price and this is I mean it's just it's just all over the map and so ask yourself right do I really if I've been using five-point scales throughout this survey or nine-point scales throughout this survey what do I get by changing the scale type rate is it something I really need to do and again from my experience on mobile devices you know I come from product testing sensory I know that donek scales are like the thing to do there I've never been able I've never not been able to get actionable guidance using a five-point scale that I couldn't get with a nine-point scale so since so many people in our world today test things on mobile really ask yourself do I even need to do anything more than a five-point scale sometimes you do but you know just make sure that you really really need to do it and another thing that I see a lot in research is you know people are sort of being asked questions that are obvious right now if you're asking that because you need that to like parse them out into another part of the survey later yeah go for it but if you're doing it because you're like I just want to validate that like our last study said this I just want to validate that eighty-three percent of people do this like it doesn't really matter like if if you actually believe it it's true or is it something that you you thought there was something wrong with the old study so if it's something that you can look at prior data for if it's something you can look at data that you've sort of benchmarked against in the past you use the secondary data sources first whenever you're doing primary research just try to understand everything you can about that topic before you send out a questionnaire and then you'll ask fewer obvious questions in the questionnaire last thing you talk about on surveys is the flow and the what I see all the time is like people will just basically they have this like this huge portfolio of questions to ask and they ask everyone every question in the survey and they get to be like super long but if you look at it it's like you know what you probably didn't need to ask this person everything right so a good example is like carrying forward of choices where if you say in one question like Oh which of the following do you think is a good idea or like which is find things have you heard of right and they pick the top three options and there's ten options right and then in the next question you would say Oh which is fine do you think you'd be interested in like would you recommend to a friend right well if they have never heard of it why would you show them those other seven options you should carry forward only those options that make sense for subsequent questions right a lot of folks don't use that kind of exclusionary logic and you really should because it's going to just shorten the experience another thing that's super important to do and this is something that is maybe more complex is the use of embedded data now embedded data what it does is it takes an answer or a piece of information that that a respondent gives into the survey and locks it into it into a table and then it recalls that data later in the survey to help either make logical decisions on where to go or to actually carry forward that that response into a future question so like if somebody said other please specify for like what's their favorite kind of car and like they put in oh I love outies because that wasn't in the list most people will carry forward the other but you can actually map that Audi into this embedded data field and carry that forward at a later point if you wanted to to use the respondents words back with them and when you use the language that respondent has given you back to them they feel like it's this like that the survey is really paying attention to them but what you can also do with embedded data is what what I call conditional path logic which is you can say all right I'm gonna I'm gonna take people who have answered these things and I'm only gonna show them things related to what they said no matter to them right I'm not gonna ask them questions about things they say don't matter like this is a big run with conjoint right is you're doing conjoint and you you believe everything is important to people but maybe you should just show them in the conjoint things that matter to them and you can use conditional paths to do that another thing though that you would say is you'd say okay I've got these different blocks and and you know the respondent liked a B and C so I'm gonna ask what about a B and C well instead of doing that cuz that's gonna make the survey super long um you know you should use these randomization functions that are common in most surveys packages and you should say alright this person qualified for a B and C I'm going to ask him about one of those randomly and most of them allow you to assign them to that condition with the lowest incidence rate at that point in the search right right and so if you go in there and you use a randomizer to assign them only to one condition for which they would qualify and you assign them always to the one of the lowest incidence rate that is relevant to them you're gonna have a pretty good success rate of filling up your survey with a lower population without making the the surveys themselves extremely long and difficult the last topic I'm going to talk about is micro targeting and this is something that not everybody can do but the idea of micro targeting is that you're going to find people who are a very very specific target and ask them a very very specific topic because they're just going to be more likely to respond and give you sort of deeper insights there's four things to think about micro targeting and so we're kind of like related to archery like you have precision and accuracy but then you also just want to make sure that what you're doing is going to work for you and your business and that you're getting sort of non diagnostic information if you're gonna if you're gonna micro target don't go for Diagnostics go for actual insights and so for precision this is the old idea that like if I were shooting 10 arrows at a tree in front of me and all 10 arrows hit hit like a stump three feet to the left right well they're all in the same stump and so they're very precise and being precise means you're you're finding a very very close-knit group of people right and these people are going to be responsive to a very sort of like targeted set of questions or topics that relate to their interests there's there sometimes the the sort of I don't call it like the the interest in applying micro targeting to like broad things but like everybody uses toilet paper right unless they're using bidets why would you try to micro target people who use soft toilet paper right everybody wants soft toys a paper that's not this like unique target now drinking vinegar it's gonna be very hard to find people who do drinking vinegar as like a as a product that they will consume and so make sure that you're going after the right kinds of people for micro targeting because if you if you if you don't go for the right kind of people you're not gonna be effectively you're gonna be wasting your time you know be putting in too at the same time if you're overly restrictive on your micro-targeting and you're saying oh look I'm looking for you know people who have taken drinking vinegar ten to fifteen times in the past two months right your target may become so small that you'll never hit your response rates so micro-targeting is a balance between finding the right topic finding the right customer or respondent but not parsing it so tightly that that you can't complete your study now actually by contrast if you remember the old archery example I was aiming for that tree and I hit the tree but I hit a branch I hit a root I hit the trunk one of it maybe he is up in the leaves but they all hit the tree right and so what I've done is I've effectively hit the tree well and that's accuracy right you want precision and accuracy but accuracy is about um sampling from from a population in a way that you can extend to model a larger population and so in micro targeting there's often going to be the risk that you are not modeling the larger population as a whole right any kind of sampling is good for interpolation very bad for extrapolation and so you got to be super super careful that you've not overly done your micro targeting and when you've done micro targeting by that same token right you want to be very careful about generalizing outward to groups that you haven't sampled right if you if you did a micro targeting for for small dog owners of puppies right for dog food you don't want to just go out and be like oh well everybody who has small dog puppies I can use this for all puppies right big puppies might have very big puppy owners might have very different interests and what they're looking for than small puppy owners so you just have to be really cognizant and and intention like really really be intentional when you're when you're talking about your results and then we'll talk a little bit about like the impact of what you're doing right when you're trying to get those micro targeted people in there you gotta remember they're going to be even more so than normal motivated by that topic that you're screening them for right and so if you've gone after a micro target it's actually super easy to engage them by just highlighting how important that topic is to your research objective they will feel uniquely qualified to give you that information because they have themselves you know their whole life never realized when no one else realized how important this is and they will they will want to give you those super deep insights but at the same time you know oh I you know we we saw that your dachshund Dexter just turned three you know would you like to tell us about his birthday like okay I'm micro targeting you because I have special information about you but don't tell me how much you have because I'm just gonna be kind of freaked out and be like this is just too creepy and so so try be effective in that another thing to think about is like how you ask the questions to these people and I was I always talk about like a cognitive funnel right so if I was studying a brand I would want my all my questions about the brand to happen first and then I would never ask a question about the brand again then I would focus only on questions about the category right that I care about and then I would ask specifically maybe questions about a particular SKU or a particular product lie right in that situation and so by starting broad and kind of bringing it down into a more precise target there's less cognitive burden people kind of feel like the flow is natural and we always always stress have a natural flow in your surveys start broad go down to specifics do not start with specifics and then go back to broad it's gonna just feel weird do not kind of toggle between the areas also just kind of weird and then finally the difference between sort of insights and diagnostics is that you know Diagnostics help you understand you know sort of like the landscape of what's happened but you know that's an experience and it's important you know like what got people to said like how do people find your product like that it's good good diagnostic but an insight is for like really creating this actionable thing to do next something that you want to create build sell grow advertise when you go to these micro targets they're gonna really help you understand what will motivate them to respond to your future offers and so don't waste too much time on Diagnostics focus your time focus your energy on those things that will drive an action and you're gonna you're gonna be better off because it's not easy so micro target and you don't have a lot of chances to do it and when you run out of when you run out of that population if it's micro-targeted you've probably got one shot and if you blow it there's no going back so that's it for me I don't know if there are any questions I'm not really good at at these kinds of things so I'm gonna stop sharing my screen and then I didn't check anything so I don't know from Charles he says for the instances when you may need to go out to multiple samples to reach the objectives of the research do you use fusion techniques or other things to combine the data yeah I mean I think that's kind of we tend to report everything out magnetically as independent baselines and then we would compare the baseline samples if maybe I'm misunderstanding his question but you like it let's say you've got multiple targets right and you're saying okay I want to look at this population this population this population this population and you say oh I'm gonna weigh them in certain ways like you totally do that we tend to because our lawyers actually don't let us drive statistics because if you use Amazon data to make a claim that would be a violation of our Terms so we tend to use distributions of our different populations so we wouldn't typically fuse our samples together as you might in the classic market research everyone's different right and so if you if you are and uh if you're looking for claim support and things like that you have to do a fusion first so I'll just make sure like you you can actually understand why the data is being fused and then also if the if the if the fused data set will be actionable if not a question from Doug he says great presentation what kinds of questions do you ask that will generate insights versus diagnostics and can you provide an example yeah absolutely I mean insights and diagnostics this is like the crux of any kind of questionnaire right a diagnostic is like where did you see an ad for our product right oh I saw it on a billboard I saw it on television I saw it on radio but once you have that data you've just got this distribution or like how many times have you bought a product right and that's it's good to know but it doesn't actually tell you what to do next now in an instant a type question would be like which of these online bloggers would you be most likely to listen to if they were to ask you to purchase or try product right and they'd say Oh YouTube right or whatever right then you'd be like okay well now maybe in my marketing mix I might want to over index on that particular blogger right so the diagnostic versus the insight is it does it give me useful information or does it tell me what to do and you want to always focus on what to do whenever you can another question from Chris given all you've shared about email and messaging in general is there any room in the remote future for video interviews and focus groups for qualitative insights I mean that there's not room in the future there's room in the present that stuff definitely can happen right now a lot of that stuff's gonna be contingent upon your legal teams and what they're gonna allow you to do there are there are a lot of constraints in our organization about sharing personal identifying information and so we are not allowed for instance to send you a like to ask you to give us your phone number or to ask us to give you your name right because that's just that's just how Amazon operates but we are allowed to invite you to a zoom call and have an online discussion and you can send that invitation via email the trick is is that we're often playing for scale so if you're doing like you know a hundred thousand people a year two hundred thousand people a year those are gonna be more difficult but if you're doing deep dives on individuals for a very specific topic email can actually be a super effective invitation mechanism you just need to be super super clear with your legal department that you're allowed to engage in that channel and in what way and again the big tip for us is like I can't send you Chris I can't send you like a note saying hey what's your phone number I'll call you like absolutely not could never do that but I could say hey Chris if you're interested in giving you your feedback on the video chat why don't you call this chime or this this our systems call shine what do you call this zoom on Thursday at 2:00 totally acceptable totally available and and super super important not seeing any other questions coming in but lots of great feedback cool comments and I know I definitely enjoyed that presentation super clickable to our what accelerants doing and even just like our email invites for our surveys I think we'll be going back to this one quite a bit so if no other questions we can let you go thank you so much John that was a really a great presentation thanks so much for your time and for joining us thanks Erica and to everyone who listened to thank you so much and have a great