Two MMR candidates from the University of Georgia, supervised by Dr. Marcus Kunia, presented findings from an A/B test examining whether the number of days a survey is in field affects response quality and results. Using a soft drink evaluation survey fielded through AYTM's panel, they found no statistically significant overall differences between single-day and seven-day fielding, but identified gender-based subgroup differences that suggest fielding duration may matter in gender-sensitive research contexts.
Key Takeaways
- No statistically significant differences were found between single-day and seven-day fielding for any of the seven survey questions overall, using chi-squared tests and ANOVA.
- Gender subgroup analysis revealed meaningful interactions: females rated beverage liking higher in the multi-day condition, while males rated it higher in the single-day condition, with a statistically significant mean difference of 0.44 between genders in the multi-day condition.
- Male respondents reported meaningfully different soft drink consumption frequencies depending on fielding condition, with 61.4% of males in the single-day group reporting 1-5 drinks per week versus 38.6% in the multi-day group.
- A TURF analysis found that focusing on taste, thirst-quenching ability, and convenience reaches 96.6% of soft drink consumers.
- Results are specific to soft drinks surveyed in January and cannot be generalized to other product categories, seasons, or survey lengths.
- The team emphasized that a much larger, multi-category study across different times of year and survey lengths is needed before drawing any broad conclusion about fielding duration and data quality.
Questions & Answers
- Were reminders sent to respondents during the seven-day survey?
- No reminders were sent. The survey was deployed through AYTM, which made the survey available each day and capped responses at 15 slots per day.
- What days of the week did the survey run, and when did it start?
- The survey started on a Wednesday. On day one, 100 responses were collected for the single-day condition and 15 for the multi-day condition. The following six days collected 15 responses each for the multi-day group. The survey was made available at 9:00 AM each day.
- Were both A/B cells balanced to the U.S. Census?
- No, they were not balanced to the U.S. Census. The team acknowledged this as a limitation, noting that without census balancing it is not possible to fully randomly assign respondents to experimental conditions.
- Did you look at individual days from the multi-day survey and compare them to the single-day results?
- The team considered this but noted that with only 15 respondents per day, statistical power would be too low to draw reliable conclusions. They suggested that multiplying the sample size by roughly ten (to approximately 150 per day) would make day-level comparisons feasible. Looking at trends across days for key measures was noted as a viable next step.
- Could time of day have been held constant as a control variable?
- The point was acknowledged as valid. Different segments of the population have different availability across day parts, which could introduce sampling bias. In this study, the survey was opened at 9:00 AM each day. The team agreed this is an important variable to test and control for in future research.
Session Notes
Research Background and Motivation
The research grew out of a recurring theme heard from industry professionals visiting the University of Georgia's MMR program: project timelines are often extremely compressed, with clients wanting data "yesterday." This raised a practical question for the students: does rushing a survey into a single day of fielding change what you find, and does it risk missing parts of the target population?
Research Question
Does the number of days in field impact the results of the survey?
Study Design and Methodology
- Design: A/B test comparing a one-day fielding condition against a seven-day fielding condition
- Sample: 205 total respondents. 100 in the single-day condition; 105 in the multi-day condition (15 respondents collected per day across seven days, including day one)
- Survey content: Seven questions. Questions 1-3 covered soft drink consumption habits and most recent brand consumed. Questions 4-7 used scales from the Marketing Skills Handbook to evaluate beverage pleasantness, color attractiveness, taste, and overall liking
- Panel provider: AYTM
- Fielding window: January 2022, starting on a Wednesday, with the survey made available at 9:00 AM each day
- Statistical tests: Chi-squared for categorical questions (Q1-Q3); ANOVA for scale-based questions (Q6-Q9)
Overall Results: No Significant Differences by Fielding Duration
Across all seven questions, there were no statistically significant differences between the single-day and multi-day fielding conditions.
- Chi-squared tests on consumption and brand questions (Q1-Q3) showed no significant differences between conditions
- ANOVA comparisons on the rating scales showed minimal mean differences: Q7 differed by 0.03, Q8 by 0.04, and Q9 by 0.02, none of which were statistically significant
Subgroup Analysis: Gender Differences Emerge
When the data were sliced by gender, some notable patterns appeared, even with the small sample sizes.
Beverage Liking Ratings
- Females had a higher mean beverage liking score in the multi-day condition; males had a higher mean in the single-day condition
- Within the multi-day condition, the difference between male and female means was 0.44, which was statistically significant
- Within the single-day condition, the male-female difference was 0.10, which was not significant
Weekly Soft Drink Consumption (Q1) by Gender
- Among males, 61.4% in the single-day condition reported drinking 1-5 soft drinks per week, compared to 38.6% in the multi-day condition
- 44% of males in the multi-day condition reported drinking fewer than one soft drink per week, while 55.6% in that same condition reported one per week
Second-Choice Purchase Attribute by Gender
- No gender difference was found for first-choice attribute across conditions
- For second-choice attribute: in the multi-day condition, 45% of females rated taste as second most important; in the single-day condition, 55% of females did so
TURF Analysis: Reaching Soft Drink Consumers
A TURF analysis was conducted to identify the most efficient combination of attributes for reaching soft drink consumers.
- Focusing on taste, thirst-quenching ability, and convenience reaches 96.6% of soft drink consumers
- These three attributes are the recommended focus for advertising and marketing communications targeting this category
Practical Implications
- If gender differences are not a meaningful driver in the category or research objective, fielding duration is unlikely to affect your results based on this study
- If gender-based purchasing or consumption differences are relevant to the category, fielding duration may introduce variation that affects your findings
- Researchers should ask: "Do gender differences matter in this industry as a function of fielding?" before treating speed-to-field as a neutral decision
Limitations
- Results are specific to soft drinks and cannot be extrapolated to other product categories
- Fielding in January may have influenced responses: respondents may have been on New Year's resolutions to reduce soda consumption, or winter temperatures may have suppressed soft drink preference
- The survey was only seven questions; a longer survey could produce different or more reliable results
- Respondents were not balanced to U.S. Census benchmarks, and true random assignment to conditions was not fully achievable in this design
- Per-day sample sizes (15 respondents) were too small to support reliable day-of-week comparisons
Broader Context: Critical Thinking in Methods
Dr. Kunia emphasized that this project is part of a broader pedagogical goal in the MMR program. The aim is not just to train students to apply standard tools, but to question whether those tools are appropriate for the specific research problem. He noted that a much larger study, spanning multiple product categories, seasons, survey lengths, and sample sizes, would be needed before drawing any general conclusions about fielding duration and data quality.
Transcript
Read the full transcript
hi everybody it's the top of the hour i want to welcome you uh to our uh arvik um our virtual insights conference um just a few ground rules before we uh let our presenters start first and always as you well know over the last two years we've definitely been at the mercy of technology um so hopefully nothing will go wrong but you never know with zoom and fun stuff like that uh second ground rule uh for this conference and generally for life is to be respectful of our presenters and of each other and then please take the opportunity to network engage and interact with one another as well as our presenters if you have questions you can send them through the chat or the q a feature on here and we will field some questions at the end if there are any um with that i would like to introduce dr marcus kunya he's a professor of marketing and the director of the masters of marketing research program at the university of georgia and julie and katie to mmr candidates at university of georgia and i will hand over to you guys oh thank you courtney and let me go ahead and share our screen here so we have a few uh interesting results of our research and research project that we conducted here within the mmr program uh first i'd like to thank bill and accelerant research for inviting us i've been doing this for three years now i think i was in the very first one i presented by myself dan in the past couple of years we've been presented with our students so it's it's a great opportunity for us to for the students get experience as presenters and also for us to showcase the kind of talent that we are uh training here in the mmr program to offer to the industry in terms of professionals of marketing research and consumer insights so um bill always allows me to plug the mmr program so if you are not aware of our program our program is was founded in 1979 the first class got in 1981 so we are not foreign but we are not in a mid-life crisis uh we are uh were recognized internationally by producing top talent and being the gold standard in terms of producing professionals of research and and insight so whether you are interesting interested in hiring research talent or you know somebody who wants to start this career dmr program will be a you know a great place for you to contact us so another plug that i have for the mmr program is that our we have a research summit called the future of the insights uh summit okay so and this summit we run every two years and uh bill and his team have been uh to the summit in the past and i talked to him the other day he's he's coming back and bringing some uh some of the members of his team and potentially also sponsoring right though so uh so if you want to learn more so it's in august 10 and 11 on the 10th we're going to have workshops where we're going to do workshops in uh on max diff and coin joint analysis both for uh suppliers and clients from you know using these techniques from a supplier standpoint and from a client a client standpoint and also on marketing metrics and then on the 11th we will have our presentations we have a call for uh speakers out that is due on may 6th so if you have something like to present uh go ahead and submit it if you can if you go to the website uh to our website uh mmrfois.com you have the opportunity to learn more about the program and our sponsors and registration information and the call for for speakers so now that i plug the mmr program and our summit i would like to hand the presentation over uh to julia and katie uh two of our uh very top students here in the mmr program so uh you guys take over thank you marcus thanks marcus all right so we just wanted to introduce ourselves real quick before we get into things um like dr kunia said where um katie and julia um went to the university of georgia for undergrad got a degree in marketing and now we're here for the mmr program to add about yourself to stay one more year we couldn't leave athens and you know this program was definitely worth it so very excited to be here and be presenting today that's right okay so to kind of get into our research we conducted an a b test to determine if survey results are impacted by the number of days of survey fielding in the mmr program we're very fortunate to be able to talk to many industry professionals just coming and sharing about their experiences within marketing research and so we noticed a common thread of saying how often times their projects have very short timelines you know we want the we want the data yesterday um so this made us wonder is this an issue within the industry is there a trade-off between the speed of fielding and data quality um so we also are wondering is this causing researchers to potentially miss part of the target population um so that led us to our ultimate research question does the number of days in field impact the results of the survey great so in order to answer this research question we decided to run an a b test where half of the respondents were collected over a one day period and half were collected over a seven day period and then overall we had 205 responses where in the one day fielding we had 100 participants and in the multi-day fielding we had 105 where we collected 15 responses each of the seven days which included that one day fielding so we only collected 15 for that group on that one day and the survey everyone saw was consistent they all saw the same exact survey the only conditions were that it was between a one day and a seven day so they were grouped into those respective fielding conditions so yes our methodology was a survey and we used aytm's panel of respondents to field the data and as you can see it was a brief survey just seven questions so the first three questions asked respondents about their consumption habits and asked about their most recent um brand that they consumed of soft drink and then for the bottom four questions we referenced the marketing skills handbook and found a previous study that was conducted that recommended certain questions and scales to use for beverage product evaluation so those asked about how pleasant it was while drinking the beverage how attractive the color of the beverage was how tasty and how much they liked it overall all right so jumping into the results of the data when comparing the two data collection periods we found no statistically significant differences between the results of the single day and the multi-day fielding for all seven of the questions so for questions one through three which julia just said were consumption and the brands they consumed we ran a chi-squared test and we found that there were no statistically significant differences across those conditions and then looking at the means of the scales that we had for question six which was how tasty was it for you question seven how much did you like the beverage and things like that so we went through those and compared the means through anova and as you can see question six you know there was almost no difference at all for q7 the multi-day was a tiny bit larger by .03 for q8 um the single day was a little bit larger by 0.04 and for q9 the multi-day was just tiny bit less by 0.02 so no real significant differences there so because overall we did not find significant statistical significant differences across the number of days of fielding we wanted to dive deeper into the data to see if there were significant differences between subgroups and so when slicing the data by gender we were able to observe a few differences for a few of the questions even with a small sample so first we found that there is a difference between the gender and average ratings for how much they like the beverage across the fielding conditions females seem to have a higher mean average in the multi-day condition whereas the males seem to have a higher mean average for the single day condition um within the multi-day we did find that there is a significant difference between the means of males and females so that's a 0.44 difference and then the single day it's not significant it's just a 0.1 difference but overall females have a higher mean average in the multi-day and males have a higher mean average in the single day great and so for looking at q1 which was the average weekly consumption of soft drinks when slicing between male and female we found some significances between males and so 44 of males said they drank less than one soft drink a week during the multi-day period and 55.6 percent said they drink one soft drink a week in the single or yeah in the multi-day period so the multi-day was slightly larger in that preference and then a larger proportion of males in the single day condition said that they drink one to five soft drinks per week than in the multi-day condition so for the single day it was 61.4 percent of males said they select one to five drinks per week and then in the multi-day 38.6 percent said that they select or they selected one to five soft drinks per week so that was a fun fact we ran a turf analysis to see how you can best reach soft drink consumers and we found that you can reach 96.6 of soft drink consumers by focusing on taste of the beverage how its ability to quench thirst and how convenient it is um when advertising and in your marketing communications um so then we did find that there is no difference between genders and in the different conditions for the first choice attribute however we did find a difference for the second choice attribute in the multi-day condition females um 45 rated it um taste as second important but then in the single day um 55 rated taste as um second important attribute so we did find that that difference there all right and for some implications as we said there were some instances where there was an interaction between the gender and the number of fielding days so with this being said if gender differences do not impact your industry with what you are conducting research building days may not affect your results based off of our research but if you do have do work in an industry where genders do impact the way people purchase or things like that then fielding days may have an effect on your research so make sure to ask yourself the gender differences matter in this industry as a function of fielding so then to review a few limitations of our of our study study um these results are industry specific as we were asking respondents about their soft drink preferences so it cannot be extrapolated to other industries or product categories um so important to keep that in mind additionally these results were fielded in january of this year so it's a there's a potential that the time of year could have played a role in their responses people could have just made a new year's resolution and been trying to drink less soda um it was winter in the united states so um they could have been swayed by the temperature outside if the survey was sent out in the summer they might have said that drink more soda in the summer so the time of year could have impacted their responses and then finally like i mentioned earlier this was a very brief survey it was a seven question survey so there's a potential that a longer survey could have led to more reliable data or different results so with that that's our um presentation we're happy to answer any questions you may have yeah i also like to reinforce the students point that we're not making a blanket statement that uh you know the way you field your survey uh multi-day versus a single day does not affect the data we would say like we need a much more extensive study you know we need multiple product categories different times of the year longer and shorter surveys and a lot more simple uh in order to be able to uh to provide them a broader scope uh of understanding of uh the impact of a number of filtering days uh on the research also this is part of some of the things we do here we train the students mostly to do industry research but our goal is also to for the students not be like let's just apply what's out there we wanted the students to learn how to think critically it's like are the methods that i'm using appropriate for this kind of research is there or anything i can do differently you know we don't want our students just blindly oh this is what everybody else is doing we should just do that now we wanted them to critically think uh critically think and say for this type of research for this type of industry for this type of research problem what's the best set of tools uh that i can use we don't want the students just say like well i learned these tools now let me go apply them like you know even if if i don't have a use for it i'll build a wall just so i can use my tool right we really want them to think and learn and question the methods and improve the methods uh not past year if some of you were at our presentation we did a project where we tested whether presenting the scales in ascending order labeling ascending order uh for for example dissatisfied to satisfy or whether uh rating uh from uh descending order uh satisfied to dissatisfied or whether the uh the scales represented horizontally or vertically affect uh your results uh for the most part what the finding that we saw is that it only affects for the people who failed the red hair uh question the attention chat for the people who do not fail the attention check it didn't really matter how you present your scale so also if you have any ideas anything that's intriguing you any other questions we're always doing these projects we have the support of aytm to use their panel to uh work on our project so any ideas that you might have would be interesting interesting for us to explore uh i would like to hear uh to hear from you now go ahead no just say if there's any questions that we will be yeah we have one question um from an anonymous attendee and they were wondering if reminders were sent in the seven day survey no there were no reminders was just that uh the way the server was deployed through a ytm so every day they said there's uh you know the the service available then they capped at 15 slots for each day okay and then i mean i just had a question i was wondering what days of the week so the one day survey when did it start and so what days of the week did it start on and and i believe we started on a wednesday yeah yeah the the day one uh started on wednesday so that day we collect 100 responses for the one day condition and 15 responses for the multi-day conditions and then for the following six days we collected 15 uh data points for each of those days i was just curious because i know we see differences on days of the week and things like that um megan a question from megan's want to know if both ab sells balance to the us census uh they were not and i know one potential concern of not doing that is that because it's multi-day per single day is that we cannot fully randomly assign people to experimental conditions so that's one limitation that we have to to live with like all surveys we have limitations yes i was just wondering if there's any more questions to go along with my day of the week one um were they all sent at the same time of day say it again were that were the surveys sent at the same time of day they were made available for people to complete at the same time of day every day okay i think it was like yeah at 9 00 am yeah i was open at 9 00 am every day uh john asks did you ever look at any of the single days from the multi-day survey and compare that to just the single day yeah so uh that's a very interesting question something we thought about it the issue is um statistical power right because we only have 15 uh you know data points on each day of the week would be very hard to show any results because of a sample size so if we if you could do like this sample if you could multiply this sample by ten fold we'll be able to do because then we have at least 150 respondents for each day and then we could compare day of the week but one thing we could do we could at least look at trends over the days or some measures you know see if they they the trend vary across across these okay are there any other questions from anyone else we'll give you a few minutes see if there's any more questions that come in i do want to say thank you very much that was very good i always enjoy the student presentations i asked for this one always one of my favorite sections okay um have a question could you have kept the day part kept constant as well yeah keep one for morning versus afternoon versus evening yeah i think uh that's also uh you know a very good point because we know different segments of the population have availability at different uh day parts uh the issue here was the limited uh sample size that we had for this uh for this project but that's something that you should definitely uh you should definitely take into consideration so i guess i guess our insight is more like you will not see differences if the survey is about soft drinks and it's a morning people that for people that take the service in the morning but good point but that's some something that if you're worried about this this kind of potential biases in our sampling that those are good questions for you to be asking and to be testing in your data yeah and especially for us practitioners to remember we do ours all right again i'll give it a couple of minutes and see if anyone else has anything to ask doesn't look like it i don't see anyone coming in with anything so again i want to thank you ladies very nice job i enjoyed that appreciate it uh dr kunya thank you again for doing that um we do have a comment from him that he will be sending uh his owner to your school hopefully so some time to look forward there too excellent so thank you again i hope everyone has a great afternoon and arvik will be back again at the top of the hour so thank you thank you
