Alex Strauss, Consumer Insights Specialist at Roku, walks through how Roku built a program that connects behavioral log data to email survey research. The talk covers why integrating behavioral data improves survey efficiency, how to stand up the program across internal teams, and what to watch out for during sampling, fieldwork, and analysis. The approach is most relevant to brands with large first-party databases looking to reduce survey length, improve targeting, and eliminate reliance on claimed behavioral responses.
Key Takeaways
- Appending behavioral data on the front end of a survey lets you cut screener length, free up space for attitudinal questions, and avoid relying on claimed responses that respondents often get wrong.
- Expect a one to two percent response rate on unincentivized email surveys. At Roku, reaching 1,000 completes typically requires sending to 100,000 users. Skew toward weekday sends, early morning or the five to seven PM window, and avoid holidays.
- Legal review is not optional. Get standardized question language, email copy, data usage rules, age restrictions, sweepstakes terms, NDAs for concept testing, and international compliance (GDPR) approved before you launch, not during.
- Use behavioral variables from your database as analysis filters, not findings. Your analytics team already has full-universe behavioral counts. In the survey, use those variables to cut attitudinal data, for example comparing heavy vs. light users on satisfaction scores.
- Never make explicit inferences about individual behavior in survey emails or questions. Telling a respondent you know they watched a specific show on a specific night reads as intrusive. Ask validation questions instead and let respondents confirm the behavior themselves.
- Maintain a contact exclusion list across all researchers pulling from the same database. At Roku, with roughly 40 to 50 researchers sending surveys, a three-month exclusion window per user protects the brand experience and prevents unsubscribe spikes.
Questions & Answers
- Have you ever run into problems working with other teams to use internal and external data?
- The main issue is compliance. The legal team has occasionally flagged specific question wording or survey language that needed to be changed before sending. On the analytics side, having a dedicated relationship with someone who understands the data makes a significant difference. At ESPN that relationship was less defined, making it harder to pull data files on an ongoing basis. At Roku, having a more dedicated analytics partner has made the process much more efficient.
- What response rates do you generally expect from email invite surveys, and is there a better time or day to send?
- Without incentives, expect one to two percent. It can go as high as two and a half percent or as low as half a percent depending on the audience. A Hispanic user survey, for example, came in around half a percent, requiring a sample boost. For timing, weekday sends outperform weekends. Sending at 6 or 7 AM so the email is at the top of the inbox at wake-up, or sending during the 5 to 7 PM window, tends to produce the best results. Holidays see significantly lower response rates.
Session Notes
Why Connect Behavioral Data to Surveys
Brands now hold more data on their customers than ever, yet most survey programs treat that data as a separate silo. At Roku, the insight team built a direct connection between log data (what users watch, which features they use, how long they stream) and the email survey program. The goal was to solve three persistent problems in survey research.
- Low incidence and high cost. When you know exactly which behavior you want to study, you can target only users who have done that behavior, eliminating wasted sample and inflated cost per interview.
- Survey length pressure. Behavioral screener questions consume survey space. Replacing them with appended data frees room for the attitudinal and motivational questions that generate the real insight.
- Unreliable claimed response. Asking someone how many hours they streamed last month produces guesses. Appending the actual count from the database removes that uncertainty.
What This Approach Is and Is Not Good For
Strong use cases
- Retention and engagement research among existing users
- Brand tracking and satisfaction measurement (Roku runs its Roku Channel tracker this way)
- On-platform marketing effectiveness, such as measuring the impact of banner ad exposure on brand consideration
- New product and feature testing with a core user base
- Behavioral and attitudinal segmentation combined
Poor fit
- Acquisition marketing, where you are reaching people who have not yet created an account and are not in your database
- Competitive intelligence, such as understanding why someone chose a rival platform or device
Building the Program: Teams and Infrastructure
Stood up from scratch, this program requires buy-in and active collaboration from several teams. Rushing any of these relationships creates risk later.
Data analytics and data engineering
The most critical internal partner. At Roku, sample files are pulled either via SQL queries run with analytics team support, or through a self-serve data management platform (DMP) used for ad campaigns. Having a dedicated analytics contact who understands the data well makes the process dramatically more efficient. A generic account-level ID (not PII) should be the primary key linking behavioral records to survey responses.
CRM and marketing
Two important questions to resolve here: Does every survey email need to go through the CRM team's send schedule, or can the research team send independently? And how frequently is the team allowed to contact users? At ESPN, the research team had to slot into a fixed send calendar. At Roku, research emails are sent independently but a three-month exclusion window is maintained per user.
Legal
Legal review touches nearly every part of this program. Items to clear before launch include:
- Standardized survey questions and email copy
- Rules on which data can be used for targeting or analysis
- Age restrictions (Roku does not contact users under 18)
- Sweepstakes terms and conditions, prize procurement, delivery, and winner selection
- NDAs for concept testing of unreleased products
- Data privacy agreements with all research vendors
- International compliance, particularly GDPR, which places stricter limits on sweepstakes prize values in Europe
Research vendors and survey platforms
Whether using a self-serve platform like Qualtrics or a full-service agency, a secure file transfer pathway must be established so that email addresses never touch a local device. At Roku, files move from the database directly to an encrypted cloud platform (Microsoft SharePoint) and from there into Qualtrics. Legal must review vendor data privacy agreements before any file transfer begins.
The Survey Life Cycle: Four Stages
1. Identifying and pulling respondents
Define audience parameters with stakeholders at the same time the survey is being written, not after. Think through both who you want to contact and what behavioral data you will want appended for analysis. Key principles:
- Keep targeting criteria broad enough to produce a viable sample pool. Highly granular behavioral filters (for example, users who watched a specific title on a specific day during a specific time window) may yield too few people to reach response targets.
- Pull all behavioral variables you will need for analysis upfront. Going back to the analytics team for additional appends later is inefficient. For example, raw streaming seconds should be bucketed into low, medium, and high tiers before the file is finalized.
- Include only PII that is strictly necessary. Email address is required for contact. Beyond that, stick to behavioral and demographic fields rather than names, addresses, or other identifying details.
- Maintain and update a contact exclusion list every time a pull is made, so no user is surveyed more than once within the exclusion window.
2. Survey setup and data connection
Before fieldwork begins, the behavioral data in the sample file must be linked to survey responses. In Qualtrics this is done by building embedded data fields so that each respondent's behavioral record travels alongside their answers. Without this connection, the back-end analysis that makes the program valuable is not possible.
Even with behavioral data appended, do not eliminate validation questions entirely. At Roku, where one account covers an entire household, the account holder may not be the person who performed the behavior being studied. A brief screener confirms the right respondent is answering, and also separates explicit recall (the user remembers the experience) from implicit behavior (the action occurred but was not salient enough to be remembered).
Avoid referencing specific behavioral data in the email or survey in ways that feel surveillance-like. Do not tell respondents you observed them watching a specific title at a specific time. Instead, use generic invitation language and ask a validation question that lets respondents confirm the behavior themselves.
3. Sending the email
Roku uses a standardized email template developed through iterative A/B testing of subject lines, sender addresses, and body copy. The consistent format produces response rates in the one to two percent range. Key operational notes:
- To reach approximately 1,000 completes, plan to send to roughly 100,000 users.
- Weekday sends outperform weekends. Sending at 6 or 7 AM local time (so the email is at the top of the inbox at wake-up) or during the 5 to 7 PM window tends to maximize response.
- Holiday sends see significantly lower response rates.
- Incentives and sweepstakes can improve both response rate and age composition. Roku is currently running tests on prize formats and values and their effect on demographic balance.
4. Analysis
Once responses are collected, remove all PII from files before downloading to any local device. Use account IDs, not emails, if files need to be sent back to analytics for additional data appends.
Apply weighting to ensure survey respondents represent the broader user population. The most rigorous method is to get the behavioral composition of the full eligible population from analytics and weight to match it. A practical shortcut is to use the composition of the 100,000-person send list, which is large enough to be representative, as the weighting target.
You should be using your behavioral variables as filters and not findings.
Because the analytics team already has full-universe counts for behavioral metrics, re-reporting those numbers from a 1,000-person survey adds little value. Instead, use behavioral variables to cut attitudinal data. For example: among users who engaged with a specific content zone, are satisfaction scores higher or lower than among the general user base?
Always analyze in aggregate. Do not draw conclusions at the individual respondent level. The exception is verbatim quotes from open-ended questions, which can add useful context to presentations without identifying the respondent.
Key Watchouts
Response bias by age
Email survey respondents skew older. Younger users (18 to 24) are significantly less likely to complete a survey received via email. Quotas, weighting, and incentives all help correct for this, but it requires active management.
Account-level vs. user-level data
For most streaming platforms, one account covers multiple household members. Behavioral data attached to an account may reflect the activity of someone other than the email recipient. Validation questions are the primary safeguard.
Users outside your database
Not every user creates an account. Roku channel is accessible on the web without a sign-in, meaning some web users are not reachable through account-based email sampling. If those users behave differently from account holders, the survey results need to be caveated accordingly.
Varying response rates by audience segment
Some audiences (for example, lapsed users or specific demographic groups) respond at lower rates. Plan for this by pulling larger sample pools when targeting harder-to-reach segments.
The program requires ongoing refinement
Dynamics differ across markets. What works for a US sample may not translate to Germany or other international markets. Build best practices, but treat them as a starting point and update them as you learn.
Transcript
Read the full transcript
all right so our ground rules for Arabic I'm sure you've heard this a few times but just for the new people that may be joining us um we are at the mercy of the tech as we've all learned through the last two and a half years it can be a little finicky sometimes um so far so good today so I have High Hopes uh second always be respectful not just of the person sharing or me but of everybody um and please take this time to network engage interact chat with each other post on LinkedIn and Twitter and wherever you want using our hashtag arvic um but be respectful that's all I ask so we are going to hear today or right now from Alex Strauss um with Roku he's the consumer Insight specialist and I specifically chose this one because I write a lot of surveys Alex so awesome really learn about leveraging user data for smarter surveys and you can go ahead and I'll stop sharing Alex and you can do your thing great thank you so much I'm really excited to take everyone through this today um let's just make sure uh am I sharing the user screen or am I sharing the full screen I see your screen okay oh no we see the there you go there's full screen okay great um so today we're going to talk through you know obviously sending email surveys is nothing new to uh the world of research but um you know a step we've taken at Roku is is really building a bridge between our um behavioral data our our log data of what Roku users are doing and and helping that to make our survey smarter more effective and more efficient um and I'm going to go through how we built that program um and kind of some ground rules and and watchouts as you as you go along that process of integrating your data um into surveys so just starting off with a little bit of background around me I've been in insights for 10 years after college I really didn't have a direction I didn't know where I wanted to end up but um a colleague or a friend of a friend recommended me to Holland Partners um so I started on the agency side and I really found uh inside such a great space to be in such a great combination of leveraging kind of creativity with uh data and I feel like it's I've kind of run with this since as it's like a perfect marriage of my skill set um and so after being on the inside for about five years I switched over to client side with a real focus on like understanding uh streaming and digital media so I was initially at a company called Whistle sports that uh really focused on making social media content around sports for a gen Z audience um and then from there I went to ESPN focusing on the ESPN plus product and helping to do content development and product development for that product um and now I'm at Roku for the last year and a half focusing on the Roku Channel um as well as some uh burgeoning Sports initiatives we're working on at Roku as well but uh for those that don't know the rookie Channel rookie channels roku's free ad supported streaming service that's available on Roku devices um but also in some other locations like Amazon Fire devices and Samsungs and things like that um but since I've gone to the client side um I've really had an amazing opportunity to build out the survey research program and and and help to build that bridge of connecting data that we have both on these digital properties like ESPN plus as well as the Roku channel to our survey data um so that's what we're going to go through today um so we'll start off by just talking about like why what's the value why do we add our behavioral data into surveys and how does it benefit us and then I'll kind of go stepwise through that process of just like kind of building that program out from scratch how do you even set it up and then once you've built that framework you know what what is that life cycle how do you go through uh you know integrating the data as well as sending out the survey um and then we'll we'll wrap up with some kind of key watch out to things I've learned through this process that you know as you may be going through this process yourself it's good to keep in mind um so the first first thing to cover off is in today's world so many brands have so much more data than they had on their customers than ever before um and it's important to not let that data go by the wayside and so a lot of companies are building out really robust analytics and data engineering teams to help mine this data and come up with insights and so obviously Roku as a digital first platform we have a tremendous amount of data on what people are watching where they're going the features they're using as well as just when they build an account you just have some basic identifiable information like their name their email their age things like that um but it's it's this grow like hugely valuable uh repository of data that you want to really take advantage of and while Roku is inherently in the digital space you know I think across the industry now we do see like Delta Airlines for example as a brand I worked with in the past you know they have data on their flyers and even like you know uh uh fast food brands like Jersey Mike's you know they know like if you're buying online they know what sandwich you ordered last week and how often you're coming in um So within the space now there's so much data and it's important that we as survey researchers are not just reliant on our emails and our and our surveys to collect data but to tap into this huge wealth of resources and and so you tapping into this resource can help to solve for a lot of the common issues that we see with online surveys um so just to kind of cover off I would probably know this is new to anybody but it's good to kind of reinforce some of these common issues that we see in survey research um one is that you know if we're trying to go after a small audience or a targeted audience um you can hit very low instance rates and you're if you're using a sample provider or things like that you know your CPI can shoot up and it can become a very expensive study or it can just be really hard to tap into that audience um and then also within the surveys you know we always are trying to make our surveys as short and efficient as possible to maintain uh users attention spans and and make sure they're highly engaged throughout the survey uh and and when we're writing these surveys oftentimes you know you have to have a long screener or you have to ask a number of Behavioral questions to validate that you're talking to the right person and and understand what they're doing so it limits the amount of uh perceptual questions that you can ask to your respondents um and then lastly you know when you're asking these behavioral questions in a survey you're relying on claimed response and so you're rely on someone to ask like how many uh like shows have you watched in the last month or how many flights have you taken last year and and most of us can't remember what we had for lunch yesterday so how you know whenever we're relying on these behavioral data points it's hard for the the respondents and us to rely on the the data that they're giving us in those claimed questions so by integrating user data we're able to solve for a lot of these key pain points that we see in in user research so you know going after these uh small targeted audiences by knowing exactly the behavior that we want to go after we can Target that we can Target users who have engaged with that behavior in our database and send this email directly to them to make sure that we're not wasting sampler that we're not sending emails to people that we think may have done the action we're sending people emails people that we know have taken that action um so it makes uh you know surveying and and targeting a lot more efficient um also by uh appending behavioral data on the back end of our surveys it really helps to reduce the number of questions you that you need to ask of the respondent in the survey in your screener that are behavioral questions um and in doing so that really helps you to free up more space to ask those deeper sexual questions that are really you know I think the value of survey research is to get at that why why are you doing the behavior you're doing and by not having an asset behavioral questions now you have the space in your survey to ask those deeper prodding questions and get at that why which is really really valuable um and then the last one which I think is frankly maybe the biggest value of doing this is is now there's no longer that Reliance unclaimed response because we know exactly how many streaming hours that you spent on the platform in the last month or we know exactly which shows you watched on the Roku Channel and we don't have to uh kind of speculate of like oh they said they wanted this did they in fact do that and so I see this uh this email survey program is a part of your toolkit um but that doesn't mean that it can like you know replace your entire uh survey uh uh program at any brand um there's things that it's really good for and there's things that it's not as good for so I just wanted to kind of highlight a few of the like kind of the the great use cases we have at Roku as well as some of the things that we we don't rely on this uh type of research for um so uh I think one of the big distinctions is like from a retention standpoint versus acquisition um so like if you're looking at how to maximize engagement among your users and like understand which behaviors you know drive them to kind of stay on your platform more often um it's really great to contact them for those reasons um we actually conduct our tracker we try to conduct our brand tracker for the Roku channel uh by cap by sending out email surveys and capturing user satisfaction um and then for like on platform marketing Effectiveness um you know it's like driving people in to watch different shows um we can Target people that have been exposed to uh for those who are familiar with Roku there's always that big band to add on the side of your screen and so we can look at people who are exposed to those types of creatives and understand uh what kind of impact that had on their consideration of the brand or the service um it's also a really great testing ground for new products and features um you know you have this core user base that's at your disposal and you can understand like what types of features are they interested like what what is their desired experience to help inform features to design in the future so it really is like a great test and learn space to to help develop and improve your product and then the last uh really great opportunity within this is you know our behavioral and analytics teams are doing a lot of work in in segmenting users based on behaviors but you know obviously that attitudinal piece or that perceptual piece is a big part of that as well so leveraging both the behavioral and attitudinal segmentation it's a really great opportunity to tap into different types of users across your database and understand the mindset that they have in our case about streaming but you know if for whatever industry you're in it can also be valuable to tap into the mindsets within that industry straight and kind of where it doesn't make sense is acquisition marketing so if you're doing marketing off platform uh you know if you're doing marketing uh on TV or different uh touch points that aren't within your product stat um it can be obviously it's not going to be able you're not going to be able to contact the people that you touched with all those different uh campaigns um and then the other one is just understanding your competitors right it's it's not great for us to understand Fire TV users or Apple TV users uh within this methodology if we want to understand why someone's buying a competitive device at Roku you know we have to employ different methodologies to get at that competitive uh mindset and where we might be winning or losing versus the key competitors so first let's start from scratch how do you build out this program and how do you uh kind of get to this place where you have a nice quick bundle of targeting and accessing these users and and appending the survey data um and so as with any you know new initiative and as Amanda referenced on hers it's a very collaborative process that like you really need to tap into a lot of different teams uh to to kind of make this come to life um so we'll go through each one but you know for us at Roku we've had to work across a number of different teams to employ this program so our data analytics team is key about linking the data um you know also it's our data engineering team data privacy teams are also involved as far as like building out these mechanisms and making sure data stays secure so there can be a lot of people on on that side of the coin that are required and I'd say those are probably your most key partner um in in making this program come to life um the next team that you need to work with is your CRM and marketing team as far as sending out these communications making sure they stay on brand things of that nature maybe the not the most exciting part but also equally important is you definitely need to be in touch with your legal teams to make sure you're compliant with you know again privacy concerns as well as you know making sure that every like verbiage and things like that is all compliant and uh with uh kind of the requirements that are set within the company and then last of all is is your research vendors whether you're working with self-serve platforms like a qualtrics or a full service vendor um you know setting up a really strong relationship and again that data privacy is a really big component of building a successful relationship there so when it comes to data analytics as I mentioned like that is the Key Bridge to making these programs come to life um and and they're going to be your key partner both at setting up but as well like throughout the entire process and it's important to have a really collaborative uh relationship with these Partners to make sure uh that everything's working and and efficient and the right order so uh the first thing you need to do is to build a connection to the data you need some way to access the data um and so you know at Roku we have two different ways that we're accessing the data uh we can tap into the database using uh SQL code uh or we have a data management platform otherwise known as a DMP um that's more of a self-serve tool um that uh is also used for a lot of our ad campaigns but we each happen to that Resource as um as a way to pull sample as well um but but getting to that place is is is an important step to get there so you know either building a DMP which obviously can take a lot on the data engineering side um or just having a way to tap into that database data is is it really like the the first step to get there um the next is just understanding what's available to you what do we know about our consumers and and what data can we leverage to uh to talk to our consumers and Target them as well as the format that is in uh you know sometimes like some of this data is not the prettiest and in order to make it usable for you as a researcher and make it translatable for your stakeholders in the end it's important that we understand how the format how the data is being collected and and how it's made available to you another like Baseline thing that I again your analytics team is the one that's going to be responsible for this but it's really a key important part about maintaining user privacy is having a generic ID Associated uh with each individual user of your brand so that you're not relying on pii to contact uh or like kind of to pull that individual user so at Roku we called it we call it account ID very simple but at ESPN there was another name for it that was across the Disney org um but is it just a unique identifier to identify your uh users and then you know once you've built that connection of the data you just have to figure out what's the best way to pull these lists of users from the database to use for email surveys so as I mentioned we have the two Pathways but you know the first one is using SQL code to query our database and I don't know how many researchers out there no SQL code I'm not one of those um so I need to build a partnership with my Analytics team and the contact there to help me to pull the to query the database and use that SQL code to pull these lists I am getting better at SQL but still still not proficient enough to do it all on my own and then also just figuring out how to to leverage that self-service to one of the things that you would need in a DMP to be able to pull those lists in our instance you know it's a little bit more high level so when we don't have quite as Target of surveys or need quite as much data about our respondents it's a really great tool to just quickly pull sample and not burden our analytics team so that there's kind of these two Pathways and I don't think most companies would need both but you do need one of these like as a viable tool that you can rely on and that's not going to be a heavy lift for all parties involved every single time that you want to pull a survey so then with the CRM marketing team there's a lot of considerations here um and and it does vary like company uh you know my experience at ESPN versus my experience at Brook could have been very different in this regard and I'll call kind of call the key differences there but um you need to really understand the rules for sending out an email you know this is consumer facing communication um and it's important that it matches the brand of you know your brand that it matches the tone and the language that and kind of the the sentiment that all Communications that come from the company have um so so that everything you know syncs up with users expectation of what you're gonna get from an ESPN or a Roku or things like that um and then the last really key thing you need to understand from CRM marketing is is how often are you allowed to send the surveys so that's actually one of the big distinctions about my experience at um ESPN versus a Roku at ESPN we had to slot into their uh email program so you know they had set Communications of like you know we send sports updates game updates on this day we send uh you know broader information about the product or deals on this day and you know you can have this these slots to send out your email survey so I was very limited at uh at ESPN as far as the frequency with which I could send surveys at Roku our email program operates a little bit more independently um of our CRM team they've helped us to build up the pathway um but we send out emails separate from our CRM team at this point [Music] um and because of that uh you know we just maintain a three-month exclusion so that no user of Roku is getting a survey more than once a quarter uh so it's just a matter of maintaining a good user experience in that regard by just making sure that no one's getting burdened by getting a survey in their inbox from Roku every single week um so that is kind of I think as you're finalizing this program is the main thing you need to to figure out and kind of get sign off on is is uh you know can you send the survey yourself you know like a self-service platform like like we use qualtrics here or does that does every single email need to be sent out by your CRM team and confirming which which pathway you're going to go down is probably going to be um based on you know the comfort comfortableness of your CRM and marketing teams then as I alluded to maybe not the most exciting or fun part of the process but legal is a really key component in setting up uh this this type of program to make sure that you know you're not overstepping any rules and like not using any language that could be uh harmful to the company um so you know Soup To Nuts almost when you're setting this up almost every single thing needs to be uh approved by legal in most instances you want to set it up so that not everything needs to be cleared in the future so getting standardized questions that are approved getting standardized email language that are approved understanding what data you can and can cannot use in the surveys or to contact people um are important understanding parameters on who you can contact so at Roku we don't talk to anyone under the age of 18 it's a very strict policy that we can't um contact anyone who's under the age of 18. and then also if you're sending any surveys internationally you have to understand that there's varying data policies and the ways you can contact people internationally so making sure that you're compliant with International policies like gdpr are also really essential as well and then there's other Dynamics based on specific types of surveys you might be sending if you're including a sweepstakes and your emails to everybody you need to work with legal to set up terms and conditions if you're you know doing new product research or concept testing of things that have not launched yet you need to set up an NDA with your legal team to make sure that those users that you're sending these surveys to are not going to be disclosing any sensitive information and then as well like as far as setting up relationships with your research vendors you need legal to be involved to make sure that you know privacy uh terms are set up for them and so I I do reference that again here you know when you're setting up with your research vendors um if you're working with a survey platform like Paul tricks or something like that you need to build a secure link to be able to upload your sample files and make sure that you know emails and other personal information is not being compromised and then once the data is on those platforms you know most of them do have really strong uh encryption and data policies but it's important that you know those are compliant with your own internal policies as well um and then if you're working with a research Agency on these types of setups you really want to make sure that uh you're setting up some type of NDA or data privacy agreement so that if you are sending emails to a person on the other side of the agency that they're uh in agreement and compliant with the policies of protecting that data and you know there's been no shortage of data breaches and leaks in the in the world in the last few years and as a researcher I don't think we ever want to be the ones that are responsible for for that data getting out there so um ensuring that those policies are set up and that you know users data is not getting leaked is obviously the the most Paramount importance so before I dig into kind of like the survey life cycle I was I wanted to open up see if there's any initial questions and if not then we can kind of proceed uh with uh the the actual employment of the the emails if you have any questions you can type them up in the Q a or in the chat and I will forward them onto outfix so far there's no questions I have a question for you of course in working with the the other aspects of uh or the other teams that you have to work on in order to use internal and external data um have you ever run into any problems um being able to do that I mean really the only like problem I'd say which isn't really a problem is just making sure you're compliant it's like I have had our legal team say like no you can't ask that or no you need to change the wording on this um before you send it out so really the only problem is make sure that you're compliant with all the the rules and regulations and the the legalese if you will that like most of us aren't completely privy to so we can in our heads think oh yeah I think that's fine but then there are some times where you're like oh maybe I towed too close to that line and kind of get maybe a slight hand slap to say maybe change the wording on that one so I think that's really the only issue I've run into the other one I guess is um working with your analytics team I've had a much more collaborative experience at Roku actually I have someone on my team that helps me to pull a lot of these sample files and it is very well versed in the data at ESPN um I didn't have necessarily like more of like a dedicated relationship with someone and so pulling the data files and knowing was available was uh like a little bit more challenging or it was a little bit harder to do that on an ongoing basis um so just like making sure that you've like built that really strong relationship with your analytics team and working hand in hand with them is something that like can potentially be an issue but like if you build a good relationship can pay huge dividends as far as like creating a more efficient uh process excellent thanks Alex um still no questions so if you want to ask all right we'll we'll keep buzzer along then so now we've set up our process we're good to go we're ready to start sending some surveys um but obviously there's a lot of steps to make that happen and and I've kind of broken it down into four components here and we'll go through each one um so the first step is is identifying the people that you want to talk to and pulling them from your database um and and this is where a lot of your data work is going to be done in a lot of your work with the analytics team is going to be done it's actually really great to be more proactive in this step and really identify all of the data you're gonna needs you don't have to kind of keep going back to that well throughout the process um and then once you start programming a study just like building that connection so that your survey responses are tied to your behavioral data as well as you're also just protecting all of the protect the uh personally ident personal identification information in there um is also really important part of setting up this process and then you send out your emails you you have to make sure that again you're complying with the right terminology and and and contact new people in the right way and then on the back end is I always think the most fun part is doing that analysis is that like taking your survey results combining them with that behavioral data to really build out a holistic picture and tell a really compelling story of you know the what and the why pulled together and I think that you know that's kind of the big like selling part of this proposition is to be able to tell that deeper story um by not just showing like this is what people say but also like this is what people do and how those relate um is really where kind of uh you provide value for your stakeholders so upfront when you're identifying respondents to call you want to be doing this in like the very beginning of your survey research process so as you're developing your survey and writing the survey you know that's when you really want to start to to pull together what's the spec going to be of like who's your targetable audience and and how are you going to contact them so you want to work with your stakeholders to Define audience parameters that are going to meet the the business needs and the business questions for that specific product um so you know a good example is uh you know we did a survey we recently on the Roku Channel launched in espacio Latino a Zone dedicated to Spanish language content um so we wanted to contact people that were both already using Spanish language content on the Roku Channel as well as people more broadly across the rugby platform that we had seen had a propensity to use uh or watch Spanish content um so in that instance we were considering the users that were doing that behavior as well as users that we want to do that behavior on the Roku channel in the future um and so understanding like that full gamut of your audience is really going to be important at this stage in the process the other thing is that you know while you want to create a really targeted criteria of who you're contacting you want to make sure that it's broad enough so that you can pull in enough people to contact we'll get into the details of it but you do need to contact a lot of people you need to send a lot of emails to get enough responses to your survey so you know I don't want to look at people who watch John Wick uh on Thursday between the hours of six and eight pm because I probably don't have enough people but I can look at since we launched John Wick a month ago um of all the people that have watched John Wick this month um you know can I ask them a survey about you know why they were watching that specific program on the Roku channel so I think it's it's making sure you have like realistic parameters and you're not really digging into this specific tactile example that may or may not be feasible to contact and then when you're pulling the audience uh it's important to to be proactive and think about all the data that you're going to want on the back end as well so you know there's kind of like two components here it's like who do you want to talk to and then what do you want to know about them um and and so in doing that you know I think the best way to think about that is like what are the behavioral questions that you would ask in this survey like you know how frequently are you using the Roku Channel or um you know how did you how do you enter the Roku Channel um you know what types of ads are you exposed to on record channel in the last few weeks um and and then figure out like those are the types of questions I'm going to ask could that data be available to me is that things that I could append to my sample files that I don't have to ask in my survey um so once you've kind of gone to that well then you figure out like what do I have and what is available and how is it formatted so if I wanna one of the things they asked me like how much did you use the Roku Channel last month you know that data might be available as a raw count in seconds or minutes or hours so you want to figure out uh you know how is that data formatted and how do you need it to be formatted for you to be valuable in your survey so you know in the instance of like if I want to know how many hours someone spent on the Roku Channel last month I I generally like to have that bucket into three tiers of like a low medium high user and and so instead of having this raw count of streaming hours I'll actually bucket it into uh three different buckets of streaming hours to kind of get that low medium high user and that's a lot more a lot easier for me to use in my analysis um and it's also going to be a lot easier to do up front so really thinking proactively here and appending your data on the front end is going to be super valuable to set up your program um and also to make sure you have everything available but also your analytics team is definitely going to thank you in this instance because you know they're gonna only have to do one data pump for you instead of having to kind of go back and append data on the back end of your survey so it's really good to kind of think through that full approach as you're setting up the survey um and then also just the last note here is as you're pulling in this data really only include pii that's absolutely required you don't want to be pulling in you know name and address and and you know maybe date of birth you might want if if age is important for your survey but really like if emails that you need to contact the people that's going to be your essential beyond that you probably just want to focus on like more of this like um you know obfuscated or like more just behavioral data that's that's not like personally identifiable just to make sure that you're not um not interfering with any data policies and so once you've pulled your sample uh now it's a matter of getting the list to your platform or to your your uh your your survey vendor uh to to set it up so One path there is to just make sure that that data is protected along the way and the main thing is that you know again we don't want like any data leaks happening and we don't want that to get ending up out there in the either so really like email should never touch your local device so you have to figure out a secure pathway whether that's um using you know you're like uh we use Microsoft SharePoint here but that's a cloud protected uh platform and it's encrypted so you know we we download our files directly to there and they don't actually end up touching our machine and then from there they go directly from our encrypted Cloud platform straight into the survey platform uh through connection with qualtrex so the emails never touch our device and that's super important because you don't want you know God forbid your computer gets stolen while you're in the process of this and now there's like users data on your computer so really making sure that process of getting the list from from your data provider your database to your sample platform where you're going to send the emails from uh is a real it's really important that you know you're protecting your data throughout there and then when it comes to setting up you know I think the the biggest this only works and it's only valuable if you're setting up a connection between your survey data and the behavioral data that you captured so you want to make sure whether you're using something like qualtrics or using a sample vendor that there they see the behavioral data in your file and how it's connected to the respondent and that when uh your your program in a study that that data connection would be built so that your survey responses are in line with someone's behavioral data so in qualtrics it's called building embedded data fields um and other platforms you may be able to call something else but you want to make sure that those are properly set up so that uh you know there's this even connection between your data and and your survey response and then as I alluded to as you're pulling this list you also want to make sure that you're maintaining a really strong exclusion list especially if you're not the only person at your company at Roku we have um you know almost like 40 or 50 researchers that are all pulling uh list for the surveys um and so it's very possible that users could end up getting surveys sent to them on a very frequent basis and so you want to make sure every time you're pulling this list that you have a good process of updating your database with some type of flag to make sure that it's known exactly when this person was last contacted for a survey so that nobody's getting bombarded and creating a negative brand experience because the fast track to upsetting your CRM or marketing team if they're finding that like all these people are unsubscribing from Roku emails because they got 10 surveys in the last three weeks uh so in maintaining that exclusion list is is also really important part to maintain a great user experience for it for all the the users across the platform so we're working our way through it and so we pulled our list we've set up the uh the the survey program and we're ready to send out the the email and and so to do that um setting up the email there's a few steps that we take here so first of all we just have a standard email template um that we use that maximizes survey response so we've done this by doing an iterative approach where we've tested and learned on kind of like almost a b tested our own like research on Research where we ask uh we tested different subject lines we tested different from emails we tested different survey bodies to get to an optimal format where we see the highest response rate um and one of the key things that we found in doing this testing is that consistency is king um regardless of who you're sending this survey to which type of targeted audience I've even found even when targeting a Spanish-speaking audience that just having and I actually saw a higher response rate for a just English survey than when I did an English and Spanish survey um uh email I'm sorry so you really want to make sure that like consistency is K and that you've established an ideal email format um that's really going to maximize your response um and and that's really important because typically our response rates between one and two percent so as I referenced before like you don't want to pull a very small survey population because your your targeted audience because you're gonna need a lot of people to get the amount of response you want so in our typical survey where we want to get around a thousand responses we're pulling a hundred thousand people uh for a list to make sure that we get proper response um and and so I understand like Roku at Roku we obviously have the great opportunity where there's 60 million Roku accounts so there's plenty of people to pull from but your user base may not always be so large and so another way to maximize your response rate and make the most of your sample and potentially lower the amount of sample you need to pull um is incentives um and so if you're going to include some type of incentive or sweepstakes program uh you know in setting up the email that's something you'll want to clear with legal and there's there's multiple things there's just setting up a standard terms and conditions but other things you need to consider in setting up a sweepstakes program is how do you buy the prizes how do you deliver the prizes how do you select who wins the prizes all of those things need to be regimented processes so there's like there's no impropriety or that there's no chance that you know consumers can feed back and say this was unfair or this you know went in the wrong way um so it's important to kind of maintain each of those process processes when you're building it up and then for international studies and and in particular if you're studying up a sweepstakes you need to understand if different policies vary so in in Europe there's much more stringent policies on the amounts that that you can give out in a sweepstakes where it becomes then I forget exactly they're trying to use I think it's like a lottery or something like that but you can't uh you can have to limit the amount that the prizes are and they have to be delivered in a certain way so you also want to make sure that those are factored in for International Studies and then the last part here is you know when you're contacting these users I've always obviously spoke all the the virtues of appending this behavioral data but at the end of the day you can't be a thousand percent reliant on the behavioral data to to supplement uh your survey questions that you do need to do some validation to make sure that you're talking to the right person uh so when we're you know especially I think the the way to make this kind of translatable for Roku is you know we have account IDs if you have a Roku account you have to associate to your TV or your device it's in your household there's one account per household but that doesn't mean there's one a person in your household that's watching Roku so you know if we see that that account has watched John Wick in the last month and we want to talk to people who watch John Wick we need to make sure that the person that's receiving the survey is the person in the household that watched that show because it's possible that you know the the mother is the one that owns the account in the household but the teenager was actually the one that watched John Wick and in which case the Insight that we're trying to gather is not valuable by sending it to that email person so it is still important to validate for your core behaviors that you're targeting the survey on that you still ask that question so we we slim down the amount of Behavioral questions but we don't eliminate them completely uh so you still want to ask about those core questions uh that are important for confirming that the type of people you're talking about and there's also things like if you're trying to confirm that they use another part of the app so like in the instance of uh you know when we've launched this bastio Latino we target Hispanic users but if we also wanted to look at the people as a subset in the survey of those that engage with Esposito we also want to make sure they recall that behavior because even if they did this Behavior perhaps they don't recall it and if they don't recall it's not very valuable to uh capture that feedback because they're they're just going to be speculating it also is another good valuable check to understand which experiences that people are having are explicit and they recall doing that versus implicit like you were just clicking around and doing things but you didn't actually internalize that behavior and it wasn't an impactful experience so it kind of gives you two levels of insight there which can also be really valuable and then the last thing that can happen is like you do still want to ask a handful of Behavioral questions because if you don't making that inference of like oh we know that you watched John Wick on Tuesday at 8pm uh in your living room you know people are gonna think that's a little bit creepier they might think it's kind of big brother so like just making those inferences in your survey or even in your emails um you know saying like we saw you watch John Wick take this survey about John Wick like those types of inferences can come off as intrusive to respondents and you don't want to uh you know kind of create negative experiences in that way it's okay that you know there's a generic uh uh path to insert the email um and then there's also you know that confirmation when it like which of these movies have you watched on the Roku Channel and you know we'll see that you know 75 or 80 of those people who we know did watch John Wick will say they watched John Wick and then we can go from there and look at the people that actually recall doing that behavior but we haven't like made that inference for them so I think that's a it's an important distinction to make and it's important to kind of step in making sure you're creating a really great consumer experience and then not to not to beat a dead horse on protecting data but once you get to the analysis there's that's another step where you really want to make sure that you're not uh compromising the data you have because now you've added even more data on these users and you don't want to that kind of show even more like make even more data about your users available so once you've collected your survey responses um and you now need to take them down to excel to do analysis so now you need to upload them to you know a recording software you want to make sure that again the emails aren't hitting your computer so when your vendor is sending you the files back or when you're downloading things from qualtrics that you're removing like now you don't need any emails anymore that shouldn't matter at all so uh when you're downloading these files make sure that you have now removed all that personally identifiable information and that that's again not touching a computer so it's kind of on the front end and the back end you just want to make sure that you're you're protecting user data because that's you know the most important part of this process um and then you know I I said we want to be proactive on the front end but sometimes we don't always have all the anticipation of every single data point that we're going to need so in the instance where now you need to go back to your analytics team to append additional data points uh that you hadn't anticipated on the front end if you're sending that file back to your analytics team you want to make sure that they're going to be marrying that data based off that generic account ID that I referenced as opposed to marrying off email because that again is just a more secure way to to build that bridge it also I think usually for our analytics teams it's easier to connect um the data based on the account ID than it is on email anyway so it's just it's uh just a better way to do the process so we've made it through the whole process we've kept our data secure and now we have this amazing data set of survey data as well as behavioral data that we can dig into and and create some really great analysis on uh and so the first thing we want to do is applying waiting and waiting is really important when you're doing these types of studies because you want to make sure that your rep that you're presenting data that's representative of your user population and so it's important to identify kind of key Benchmark behaviors um our standard one that we use at Roku is just platform streaming hours um how many hours they've spent on the Roku platform in the past month um but using like kind of these Benchmark behaviors to make sure and the age is another one that you can really use to make sure it's representative of your like demo composition but you want to make sure that it's representative of your entire user base and uh sometimes when filling out survey responses you know it the respondents can skew away from what's a representative mix so there's two ways that you can go about you know main making sure that the composition is right the first one is working with your analytics team to understand of this full population of views of all the John Wick viewers what was the how often were they streaming on the platform in the past month and pull every single genomic viewer and uh and an understand of the say I don't know making up a number of the same million users of those million people that watched John Wick uh how many stream this much how many streamed that much have I streamed that much um that's going to be your most like Ironclad method to come up composition but the easier kind of cheap way to do it is if you pulled a list of 100 000 people you can go and a thousand people filled out your survey you can go out you can go back to a hundred thousand people and just look at the behavioral data there and say of that 100 000 you know streaming hours throughout this way that way in that way and and by just looking at that group a hundred thousand is still going to be very representative of your platform that's a huge sample size so you can just leverage that and and build your weights off of the composition of that hundred thousand to make sure that you're a thousand people that answered is representative of the hundred thousand you sent the email to um and so once you've done that now you can really feel confident that your survey responses represent um that entire streaming or product population um another really important thing when you're conducting this analysis and kind of in this phase is you should be using your behavior behavioral variables that you've appended as filters and not findings and and the reason that's important is because ideally your analytics team is doing all the analysis on behavioral data they're going to know at scale you know like this many people use despacio Latino this month and they use this many streaming hours so there's no sense in reporting out that data in your survey because you're only going off the thousand people that answered your survey where they have the full Universe of data but it is valuable to say of the people that use dysplasio Latino they're more likely to feel that way compared to the general population so like leveraging those behavioral variables you have as a way to cut your data and understand those people that have this Behavior we're more likely to say this is is where you can really Drive meaningful analysis um and and help to add context where just looking at the data you may not understand how people feel and then just the last call out again on the Privacy space is they never really look at it at a respondent level you shouldn't be kind of looking at Cross lines you should be looking like down the column in the sense that you should be looking in aggregate of like of all the people that said this um and have that behavior this is how they feel not like this person who streamed 10 hours last month he also said that he really liked the experience um so it's important to kind of look in Aggregate and not at a user level of course if you ask an open-end question and you wanted to grab a really great great quote from that open-end question to add some fun context on your slide then yeah of course I think that's totally fine um you also probably shouldn't be saying that person is this person that that uh like you know George from St Louis said this uh but at least having that quote in there can obviously provide more context to your uh to your presentation so that's the whole process of setting up this program um I wanted to share a few learnings that I've had in doing this at multiple companies as far as kind of things to keep an eye out for um and things to make sure that you're controlling for when you're doing this um so the first one is response bias uh when you're sending out these surveys you can't because especially if you're not incentivizing uh the people that answer your survey from an email May skew away from your general population so the main issue that we see at Roku and that we we solve for you know through managing quotas and weights and offering incentives is that the people that answer email surveys skew older if you're 18 to 24 and you get an email from request asking to answer a survey you're a lot less likely to answer that survey and someone who's 65 and so you just want to make sure that again representativeness is key and that you're taking measures to balance that and be representative of your general populations as much as possible so incentives helps we're actually running a number of incentive tests right now um to understand which different formats of sweepstakes or value of prizes drives both better response rates as well as improved composition across ages so if you want to follow up with me on LinkedIn in a few weeks I can probably share some some results or guidelines on how that worked out but um it's important to kind of take those measures to make sure that you're getting a representative sample um and then another really important thing in response bias is to figure out uh you know that those who you have available who have accounts um may not be all of your users so in the instance of Roku you know you can access the Roku channel on the web at Roku channel.com uh but you don't have to sign in to watch content on that website so of our web users only a portion of them have accounts so if we're reporting web usage data and and web uh user perceptions based on people we contact with the account we have to factor in that some of the people that are watching on web were out of our usage of of our sampling set and and they may have varying perceptions especially if you know creating an account is a sign of higher engagement um it's important that you're reflecting on that and making sure uh that you know you're caveating that the users that you have in your survey may be higher engaging users like if you're thinking outside of a group realm like if Jersey Mike's like if you have an online account and you order sandwiches online half the time then the other half the time you go into the store and you buy different sandwiches uh then you know you may not like understand all the sandwiches that that person's buying uh or things like that so it's it's important to kind of factor that in to know who exactly you're talking to and that there may be a portion of users who you are missing uh in doing these surveys and then I alluded to this one earlier but important to kind of reference again is that you know you have to understand if your user base is at an account level or at a user a one-to-one user level and and making sure that you know and you validate that the person you're talking to is the one that conducted the behaviors uh that you have appended to that that user is really important if some companies you're going to have that 101 relationship but in the instance of Roku or friendly any other streaming company uh it's hard to understand uh or it's there's that level of differences that could be happening between an account level uh and a user level so it's important to factor that in and then the last one there is to just uh you know you you may have varying response rates based on who you're talking to for that survey so uh you know if we want to talk to people that didn't use Roku channel in the last 90 days or we want to talk to a Hispanic population who we've observed has a much lower response rate to email you know you have to make sure you're factoring those in uh when you're pulling your sample populations and and knowing that you're going to be talking to a harder to reach audience you may need to pull a larger list to accommodate for that lower response rate so those are all different factors that like are important to consider when you're building out these programs and of course as you go through this process you know we're constantly learning still and constantly like refining our policies and refining our processes to improve and get better because we send out a new survey internationally in in Germany we see a different Dynamic than we saw in the U.S um and and so we have to change things accordingly so it's really important that you know it's not a rigid process that you set up best practices but that you know as you go through this process you continue to learn and continue to get better so let's open it up for some questions all right we have a question from Heidi um she asks what response rates do you generally expect from email invite surveys and is there a better time or day to send an invite for your users yes that's a great question so generally speaking it's we do see that between one to two percent response rate without any incentive um and it's pretty standard we I have had it go as high as two and a half I've Had It Go as low as a half percent um as I alluded to actually that Hispanic survey that I sent of Hispanic users um that was actually like a half percent response rate so I had to ended up boosting that sample so it can vary based on who you're talking to but one to two is what you generally expect um as far as when to send uh I find sending during weekdays is better um and then I also find as far as time of day either having it be the first thing in your inbox so sending it out at you know 7 A.M or 6 a.m East Coast time often because I'm on the west coast here I'll I'll queue it up to be sent out for the next day uh but sending it out either first thing so it's at the top of their inbox when they wake up in the morning or sending it out around you know in the kind of that prime time of the five to seven PM uh is where I tend to see maximal response rates sending over weekends sending over how holidays holidays in particular like are definitely uh where you see much much lower response rates questions and type them in the Q a we'll give them just a minute to see if anybody else has any questions of course but we are button up on time so let's work yeah good timing on that Alex a practice foreign I don't see anyone typing anything so um we'll go ahead and wrap it up Alex thank you very much uh that was a great step-by-step on um that whole process of integrating the two from internal data and and primary research thank you so much I am a pleasure
