Does Scale Direction and Orientation Affect Survey Ratings? A Research-on-Research Experiment

all right um it's 12 o'clock so i think we're going to get this uh this conference started um welcome everyone my name is bill mcdowell i'm ceo at accelerated research we are happy to be hosting a set of pretty fantastic webinar presentations today this is our accelerant research virtual insights conference the april installment if you've joined us for conferences in the past welcome back if this is your first time joining us thanks welcome glad to have you i'm gonna go through a really really quick spiel and then hand off to uh marcus and team from from uga um i am gonna be essentially playing the role of the dude who stands at the podium during the in-person conference while everybody's filing in with you know danishes and coffee um given some quick housekeeping and overview information um so i won't bore you for too long but just a couple of quickies i'm going to give you our ground rules that we have been operating by if you've joined us for any of these in the past these are not going to be foreign to you at all i think we've all been doing our fair share of zoom meeting and webinars virtually so um you know you know the drill at this point i'm sure uh but we'd like to remind folks you know here we are we're at the mercy of tech if you know audio goes out if we have any issues or snafus we've still got presenters who are primarily working remotely so you know bandwidth can sometimes get a little funky for folks at home so you know be patient be forgiving um i will mention that accelerant is a research company not an event planning or conference organization so you know the the conference that that you're attending will have a little bit you know a little bit less polished than maybe some others um what we do bring to the table is you know some really good content some great speakers um and you know the opportunity to sit back and hear some of some research experts talk a little shop which is great um so you know i have the second point here which is be respectful it's never something that we have an issue with but just a reminder that yeah even though this is remote um let's avoid the the trolling let's you know make sure that we are respectful of our presenters um you know that doesn't mean that you can't you know challenge and ask questions um everybody's a researcher we're certainly used to having to defend the results that that we produce so you know compelling conversation compelling questions are great we certainly encourage that um and to the extent possible i know this is a virtual conference um we're not in person you can't raise your hand and ask questions well let's try our best to to network to engage with one another uh virtually so we've got a lot of different uh spokes in this this webinar we've got you know folks who are accessing on zoom we've got linkedin youtube twitter um we're gonna do our best to monitor chatter on each um if you have questions certainly pose them comments um you know whatever either for the presenters themselves or other attendees to the extent possible try to treat it you know like an in-person conference shake hands introduce yourselves um say hi to one another um apologies for what i think was a scheduling snafu i think some of you may have gotten an invitation for this uh starting at 11 a.m eastern actually start time was obviously here we are at noon um so sorry about that um i will mention i know we've got a lot of cx folks that are in attendance so um i'll mention that you know just to keep everyone satisfied we are offering you know full money back on your registration fee which was nothing because it's a free conference a little joke there um and that's it i'm gonna give you very quick just rundown of our first speaker our speakers are going to be from university of georgia uh we've got marcus cunha who's going to be sort of leading the show but very excited about the fact that we also have um you know some of his students uh presenting as well which is which is really cool i think i'm just gonna give you a real quick rundown of the remaining schedule for the day and then we'll loop back hand off to marcus and team and and let this bad boy get going um so at 1 pm we're going to have molly malson from schwab who's going to be talking about preference testing in user experience research to do or not to do um we've got our 2pm which is going to be mandy drew from capital one um and then batting cleanup is going to be josh freitz from walgreens uh talking about design research and the covid 19 vaccine rollout so a lot of good content today very excited about it like i said first off is going to be marcus and again from uga marcus is going to be joined by four of his students we've got adam clements we've got luke miltner luke is also in addition to mmr candidate he's also an incoming research associate at ksnr reed riley who is also an inbound senior research analyst at ipsos as well as zoe zirlin who is uh gonna be incoming strat quantitative strategist at egg strategy um so you know this is great i love the fact that we've got some some students presenting um you know i think based on these these credentials we've got you know our industry is in good hands so very excited about this um i think that's it for me i'm gonna shut up i'm gonna come back at the end of the presentation uh handle some q a but marcus and team can certainly feel free to to field any questions as as they come up uh but yeah marcus i'm gonna let you take it away i'm gonna stop sharing and let you roll with it thank you bill can you guys hear me okay hi sir uh thanks to uh you for inviting me uh back i actually i was in one of the first installments of this at the beginning of the pandemic when i presented on behavioral economics and consumers insight and and research so i'm glad that we are also headed towards the end of the pandemic and that bill continues to do this a great service to the to the industry so last time i did all the heavy lifting this time i have the students to do the heavy lifting so i'm just going to stand here for a few minutes and look pretty and tell you a little bit about our program some of you might not be as familiar so uh the master of marketing research program at uga at the terry college of business it is the first and most recognized program in the research and insights industry was created to suit because of a need of the industry the need to support data-driven decisions and analyze growing amounts of data available from retail scanners according to the 2017 grid report our mmr program is recalled three times as often as all the other program mmr programs combined it was founded in 1979 so we didn't have midlife a crisis uh hopefully not but in the first class graduating 1981 so which means this graduating class that's going to be graduating in a couple weeks is going to be our 41st class uh when they graduate we're going to have 730 uh alums of the program who are thought leaders in the industry so our mmr students are training both methodological and consulting skills so they learn conjoint analysis they learn regression models they learn multivariate um skills but they also learn how to present data data visualization uh presentation skills storytelling and so uh and so on so they have access to um state of the art software and hardware that the program provides we're actually presenting from our dedicated state of art lab our students are highly thought after by companies recruiting sites and research talent so we have a hundred standing 100 percent placement rate so whether you're looking to recruit top talent or further your research education or if you know somebody who wants to enter the industry please contact us by contact me directly using the email or contacting our program and with that i would like to introduce our our team um we're all back so yeah um so i'm going to start us off my name's reid reilly i'm from charleston south carolina and i did my undergrad at clemson university while at clemson i was able to partake in a research study under two professors with some other students and it was for siemens corporation out of their atlanta office it was year long and we did qualitative um research looking at generational differences with their um mass retirement that they're finding that they're having and so that really gave me a love for research i wanted to pursue it as a career which led me to the mmr program this past year has been awesome i really feel like we've really honed in on skills and it's really prepared us to have this career as researchers in the industry and i have to plug it again the professors have been awesome they've made it a very memorable year with the unknown and challenges of kovid and so it's been great awesome i'm zoe zerlin i'm originally from chicago i did my undergraduate degree at florida state in advertising and statistics florida state university has an incredible undergraduate research program i was able to conduct an honors thesis it was a quantitative content analysis of nazi propaganda from 1933 to 1938. so i'm really thankful that i've been able to kind of flourish in my research career here because professors have been incredible and yeah very thankful awesome uh i'm luke miltner i've since gotten a large haircut since that picture on the slide i'm what they fondly refer to as a double dog i went to uga for undergrad studied digital marketing got a certificate in entrepreneurship and i kind of stumbled my way into the research scene pretty late in college i thought i wanted to do non-profit work for a long time i have a huge heart for that kind of smaller business and i love that about entrepreneurship as well so very i guess goal oriented person i like serving and giving back and i feel like i can i have an impact to do that in the research industry and i really love the program here at uga and finally uh hi i'm adam clements i'm a native of cincinnati ohio and i graduated from northern kentucky university i really found my love for research in my final years at northern kentucky i did a majorly secondary research project but we did some cool stuff on the back end too and i actually got to fly out to denver and present to the whole marketing squad um and that really signaled to me that this was what i wanted to do with my life going forward um and i ended up finding the uga program on a lark and have loved it the entire way through it's been one of the best decisions in my life awesome yeah so yeah adam and i are gonna step out for the first part and i'm five nine so they're just very very tall so we're really thankful to be here and that you're taking the time out of your days to listen to our bindings um so we're going to jump straight into our presentation so to start off we had a research question it was inspired by our mmr alum who was wondering whether direction orientation of radiant skills influenced respondents and so as a team we created this overarching research question which you can see in the red is which scale points being presented and ascending versus descending order or vertically versus horizontally affects respondents ratings and so as a team we design an experiment to answer this question it did include a mark ii type attention check as the mmr as we hypothesized that attention paid to the experiment might influence ratings as a function of how scales were presented so a little bit about our experimental design the respondents were presented with an experiment which asked them to rate items on well-established psychological scales they were measuring impulsiveness and dietary control we pulled these scales from the marketing handbook of scales so they have already been tested on validity and reliability which allowed us to take average score on each scale and use that as our dependent variable in the analysis regarding how we were going to measure this we had we were looking to see whether differences in the structure of a scale affect the ratings so we manipulated the two independent variables which one is the direction of the scale labels that ascending versus descending and then to the scale orientation which was the vertical versus horizontal so this resulted in us having four experimental conditions that is sending vertical ascending horizontal descending vertical and descending horizontal so we designed the survey in qualtrics and we utilized protege mmr panel for fielding the research the respondents were randomly assigned to one of the four conditions and then were presented with both the impulsive and dietary control scales in random order so note there was two randomizations occurring and this resulted in 762 respondents that participated so a little bit of what respondents were seeing these are the skills that we pulled from that handbook so on the left you can see let's see if it's controlled right oh no is it controlled no sorry we were trying to do a pointer yeah so on the left you can see them pulse buying tendency scale these statements we're asking how often they spend more than they can afford do they like to indulge self-control um do they seldom plan and it's a seven point scale we strongly disagree all the way to strongly agree and then on the right you see our dietary control scale and this was nine statements talking about do they do small helpings to control weight do they skip meals calorie count eat diet foods and this was a five point scale from never to always and so this is what our respondents looked at in the four experimental conditions as you can see we have on the left number one descending vertical with always being at the bottom and then we had the descending vertical with always at the top and then on the bottom was our horizontal ascending descending with ascending horizontal always being on the right and then descending horizontal always being on the left so now i'm going to pass it off to zoe awesome so before we conducted this in this research we wanted to look at a bit of a literature review kind of see where heads were at in the industry if someone had you know already done this research before we replicated it so we found quite a bit of associated research by and i apologize in advance for my pronunciations of these names ting yan and florian koisch who are are at the university of maryland um and liru hey um so they did two different pieces of research the first of which um was the impact of question and scale characteristics on scale direction effects so they found that when scales are presented in the forward order an average of 46 percent of respondents chose from the first half of the scales however only about 43 and a half percent of respondents chose from um say sorry from that side when the scales were presented in the reversed order um so that was they did you know find a significant um difference between the two uh those halves do look a bit equal you can see the degree of freedom is 1254 so it might have been more doable to approach significance because of the large sample size we had a sample size of about half of that so we wanted to see if we could replicate this research with a smaller sample size and find the same results they also published a piece in the public opinion quarterly where they noted ratings for those countries so just the variable they were testing are higher when the scale starts with a high number than when the scale begins with a low number this is more of a question about anchoring an anchoring is not something that we approached in this research but it's important all the same to note when you're writing your questionnaire you're writing your survey where where you anchor your questions will have a bias or will have an impact on the responses that you get so for our experimental design each uh sorry each respondent sees both dietary control and impulse buying statements um so they see both of those sets um they are randomly assigned to one of the four versions of the scales and then the question was when you know ratings between treatment groups are compared do respondents ratings vary as a function of the structure of the rating scales so the order the order of the items within the survey were randomized they could have either seen dietary or compulsive first so they were completely independent of each other so the first test we did before we actually began the statistical significance tests where was a factor analysis so this is a confirmatory or an exploratory factor analysis we wanted to ensure that the different scale items were loading onto different factors because we did take the average of these two scales so if you can see to the right of your screen we have the diet control questions and the impulsive questions and there um there's no cross-loading between those two factors so we were able to confidently take the averages of those two scales in order to ascertain whether the scales had an impact on like you know how people responded to the questions we also did a cronbach's alpha test so um we found chromeback alpha integers of 0.8 and 0.92 for each of the scales this is great this means that they're highly intercorrelated within the within the scale so really this is telling us that we are measuring the same things when we're measuring impulse buying and when we're measuring dietary control so this factor analysis and the reliability um the reliability statistics were really important to kind of ascertain before we got into further research to ensure that the statistical power of our of our research and our objectives you know kind of held up so our top line findings our executive summary from this experiment attention checks and screening for straight lining can mitigate the impact that the structure of a rating scale may have on participants ratings so in other words the pattern of the data varied meaningfully across those who failed versus those who passed the attention check so if you take away one thing from this entire presentation know that you should be doing attention checks and you should be screening for straight lining in your questionnaires and your surveys and if you're not doing that then the scale of your questions might have an impact and might have a bias on the responses you're getting so it might lessen the statistical power and the insights that you're getting from the survey so now i'm going to pass it off to luke and adam and they're going to go into more findings that support this executive summary good height all right thank you guys um hi i'm luke again i'm gonna be taking us through um some of the findings um gets a little tongue twisty in there but um bear with me uh so before i get into the specific findings i wanna talk a little bit about how we approach the analysis our first step as reid mentioned earlier was looking at the attention mark we had a classic mark ii attention check where we told respondents to answer the second option and we wanted to start by looking at those who failed the attention check to see if there was any sort of systematic bias within those respondents so that left us with a sample size of about 80. and when we were looking at that group we found that directionality which is ascending versus descending but not the scale orientation significantly influenced the ratings for both dietary and impulsiveness scales now that is significant at a level of .001 which is pretty robust statistical significance so when you plot the data out this is what it looks like it's kind of helpful to visualize now because we're looking at only directionality as having statistical significance we're going to be comparing the ascending bars to the descending bars from dietary and impulsive independently so as you can see on the left with dietary control those ascending bars are systematically lower than the descending and then the same uh kind of pattern carries over with impulsive where those ascending ratings um average out to a mean of about 3.5 or 3.4 where the descending is much higher for impulse around 4.6 and 4.8 we also plotted the baseline rating from attentive respondents just to kind of even emphasize that difference in ratings now looking at finding number two we decided to move on from those who failed the attention check and just started looking at those who passed the attention check which was a sample of about 682 respondents and what we found here was that neither the directionality nor the scale orientation significantly affected ratings for either the dietary or the impulsiveness scales so when you put the data out this is what it looks like um which is exactly what we want to be seeing right so this was more of a sanity check than anything because we want the scales no matter how we're presenting them to be producing consistent um and accurate ratings so a good level of stability across the bottom there getting into finding three this was perhaps our most interesting one we started we continued looking at respondents who had passed the attention check but we decided to dig in a little bit more to straight line so for the purpose of this experiment we defined straight lining as respondents who had a variance a total variance of zero in their answers so if somebody had responded with say answer number four on every single question and then selected five at the end for the last question they probably weren't paying very good attention um but they were not categorized as a straight liner in this example so we wanted to keep that definition nice and strict to improve our interpretability so when we're analyzing those who passed the attention check and then straight lined we found the same kind of pattern that we found in finding one with those who had failed the attention check so the directionality but not the orientation significantly influenced the ratings for both dietary and impulsiveness scales so you can see across the bottom again comparing ascending to descending you have dietary control approaching significance and then impulse spying uh as very significant where the descending values are consistently higher um one more thing to to know before i pass it off to adam um is that this effect seems to be a bit more pronounced with the vertical categories as you can tell with dietary specifically the only real difference between ascending and descending is found in that vertical category and then with impulse buying you see a very large difference between an average rating of about two for ascending vertical and then an average rating of about 5.1 for vertical descending awesome so to wrap this up in a nice tiny little bow we went ahead and prepared some key takeaways for you guys so just to reiterate a lot of what luke was saying so direction of rating scale labels and that's ascending versus descending but not scale orientation vertical versus horizontal uh influence participate participants ratings when both they failed attention checks as well as straight lines so because the effects of direction are so so tangible it's really key to ensure that you're utilizing data cleaning on the back end of your survey as much as you possibly can and i think we have another finding here that really really echoes that um specifically we see that in descending um the the ratings within those that straight line and fail the attention check are very very highly inflated when compared to basically all the other values across the board so in addition to that we did run a bunch of uni anova tests on a bunch of demographics that we had from the survey and realistically we didn't see a ton of massive effects most of the effects were negligible across the board we had some kind of fun ones i think specifically i was laughing at the fact that those with professional degrees whether that be juris doctorates or medical doctorates had massively inflated ratings within the impulsiveness scale um i have no idea why that is perhaps it's they have more money so they can be more impulsive or perhaps they're just too busy to read the survey um regardless we did see effects like that as well as in diet control we saw that asian respondents on the descending saw inflated ratings and on the impulsiveness scale black respondents saw lower ratings um again on that descending scale and with that here's a list of a few of our references and we'd like to open up the floor for q a do you want to take the survey so uh this is a project was a project that we did in the course i teach in the mmr program during the fall uh semester and um you know the idea is for this course in addition to do a several different types of analysis including turf analysis maxdiff crosstabs chi-square analysis survey design we also work on this idea of research on research you know they do several readings on how to uh you know more correctly write survey uh questions how to better analyze data uh what's the impact of the collection method on on the data and then this was a project of what we call the research on research project so we we like to do this kind of things because uh you know we we help to promote knowledge across the industry and to help to answer questions that people in the industry might be asking themselves but they might not have the resources and the time to um to answer those questions so through our partnerships we have a partnerships with two panel companies with aytm and prodigy mr and those partnerships have been instrumental for us to do this kind of project because because the forge's uh partnerships uh you know was mostly like students begging for their friends on social media to answer to their surveys and through this partnership they learn a lot in terms of uh you know the trade-offs that you are facing when you are designing a survey or running an experiment i want to ask this question this way well that's going to take more time it's going to become more expensive or i want to run a max diff or i want to run a coin joint and then because those tasks are very taxing on the respondent and then also the price goes up so we have some budgets that are allocated for those kind of panels and then students tend to learn a lot about these uh trade-offs uh i'm gonna check uh the chat if there's uh questions was one of the scales was one of your scaffolding did you allow smartphones survey takers and if so how do you do so we we didn't screen for the type of um device that they were taking and i don't know should they have do you have the data available that i booked a really good platform but i don't think we have a way to test whether there was a difference in uh the the way people took the survey in terms of the device that they used to take uh to take the survey but that's that's uh a uh a good point so jenner generally is speaking as you saw in the results what we see let me see here if you go back to the main uh finding here when we the the check two uh kind of question uh it was something like this we gave them a list of animals and say of this list of animals uh which one is your favorite right so we had like pets kind of animals they were like dogs uh you know birds and uh cats and we also had like tiger lion whale and then we said okay with this list of the animals which one is your favorite and then say and then we we added if you are truly paying attention when taking this survey please select option two tiger right so then uh 80 people were not able to you know did not pass that attention check and then whether the labeling of the scale was ascending on descending you saw we saw that there was a an effect here you know where people probably assumed that the scale was going uh in an ascending uh direction did you see it missouri took quality exclusive scale vertically when people are using this mobile device should i be looking at skateboarding so let me see [Music] my survey tool how to expose these skills vertically when people are using this on well should i be looking to scale order by checking the people the answer on different advice yeah uh you can do a quick uh checks on that you can uh it's a you know depending on how many uh categorical variables you have you have like if you wanna test just smartphones versus known smartphones you can do a quick t-test on the scales and see if there's differences uh or if you have more than two options if you have a smartphone desktop ipad or whatever you want to test then you do an anova and you can quickly tell whether there's the difference what our research is suggesting is that the extent to each people pay attention to the questions and to the scales and the order of the scale there shouldn't be any issues with uh with your data did you see tess you test any differences by number of items of scale so the pattern pretty much replicated in both dietary and impulsive buying uh one way of quickly doing this if you if you guys uh plan to do that and you have some of your skills are five point and the other one seven points uh you can do a repeated measures uh anova where the scales are levels of a factor and then you can see whether this uh within subject factor or repeated measures factor interacts with the other factors which would be the order of the scale or the kind of device that you are taking the uh the kind of device you in which you are taking this the survey or the experiment so it's not that really hard uh to test for that but it's very very easy right so because in this case we had a repeated measures people responded to both the dietary control and impulsive buying we randomized the order that they receive each of kind of scale and also the items within each kind of scales to try to prevent carryover effects from one scale to the other because to some extent they're measuring things that people might think they're similar but our factor analysis the fact that the scale items uh match it exactly on in a single factor for each of the scale tells us that they're actually measuring two different uh constructs so um but all these can be easily done with a statistical statistical tests in general the direction shouldn't matter in fact so this one yeah so on this slide here these are the people who paid attention right so when people paid attention you can see there is no difference whether the is this case horizontal or vertical where on ascending or descending those differences were not statistically significant none of these differences were statistically significant and the same pattern replicated for the impulsive uh by so when people pass the attention check we didn't see any of this um impact of the orientation of the scale or orientation of the labeling of the the scale one thing that we have here we have a baseline you know uh the baseline that uh that we're reporting here is the overall mean or we call a recalling statistic the grand mean of all these four cells so the grammy for all these four sales of the people who paid attention and did not have any impact in terms of the orientation of the scale degree and mean was 2.6 overall so this is what the rating overall rating of somebody who was paying attention should have been okay the fact that the descending scale is going above that grand mean means that descending labeling of the scale inflates the rating uh the overall rating relative to what we would have seen if people paid attention to uh the the question and the orientation of the scale the same result is replicated here right where because the baseline and ascending the baseline uh matches the ratings uh in those conditions but when we go to descending the ratings go well above the the baseline so if you have any more questions you see the scales okay you i don't have any questions on other platforms yeah it looks like we had a couple um one was an apologies about this i missed you already responding to this but could you describe straight mining again okay so um straight lining is when somebody if you have like a series of items right so here we had given example based on the scale here we uh the scale items for the dietary control condition were uh set nine scale items people had to rate themselves using this five point scale for each of these nine questions they receive one of those questions one at a time now if somebody you know click on sometimes for all nine uh different scale items or on all nine of the questions we defined that as straight line right you could add manually counting uh but that's a lot of work so what i thought the students should do is just just measure the variance of these nines uh scale items right so if the variance in that scale items on the nine scale ions it's equal zero means there was no variance means that every answer was considered rated the same it could be the case there is a chance that somebody really felt like that so perhaps you can use some other measures so for example how long it took them to respond those questions to to figure out whether this is true straight lining or not uh you know i would expect that people who were not paying attention straight line to get through with the survey and get paid they didn't spend a lot of time rating those scales but if they were thoughtful they might have spent some time you know a little more time and say like yeah my my questions my answer for all these questions is sometimes so there is a possibility that there is somebody who was paying attention and straight line but did not straight line as a way to get through the survey without paying attention and it looks like we had another question from rebecca um asking was there any qualitative component to this study or was it just quant so it was just a quantitative they do take a qualitative course at the same time they take this course my course is more uh quantitative and the reason for selecting jesus well-established scales is because these researchers already did the qualitative portion of the study to id they've identified i don't know 25 or 30 items for each of those scales and then they went through statistical procedures of validation of um scale measurement and they came up with designing that best explain the variance uh in the data right so we could skip that that step because our focus was mainly to quantitatively measure if there are differences in the ratings depending on how you present the scale does that is that clear yes so the recommendation is to do the attention checks and screen for straight line and removing the response of those participants that fail both of those yeah i'll say both can impact the data one thing that we saw is that when you just by removing the attention check the mark 2 kind of attention check we eliminate this boost in the rating uh so the trade-off that you're facing here it is and if you can do both it looks like it's great uh what happens here is that you're facing now a trade-off of sample size right so if a large number of people fail the attention check and the straight line uh what happened like your the power of your design goes down the statistical power of your sample goes down and you are less likely to find differences that might exist in reality in the population but your sample does not show because of lack a lack of power so say if you have to remove more than uh 10 to 15 percent of your sample depending on the size of your sample you might be hurting your ability to identify statistical uh significant tasks but if you're seeing that you know large percentage of people are missing uh your attention check and are straightening like straight lining and they're rushing through your study it's probably time to talk to your panel and talk about the quality of the data that they're getting i think we're wrapping up on questions i don't see any new ones coming in i would just echo at last conversation um you know that's where the art and science comes in in play right you've got these things at your disposal from a quality standpoint you know the straight line checking timers you know asking sort of trick questions or or making sure that folks are paying attention but it is something that you have to take kind of conditionally um you know if you're for example terminating respondents based on you know not passing attention checks um you know you it's one thing if you're purchasing from from a panel from a from a sample provider it's another thing entirely if for example you're doing like a customer sat tracking study you're you're measuring the the opinions of your customers and you there's a that you know sort of cx issue potentially at play as well yeah there are other things you can do that we have seen in our in my in my course for example you can do like a commitment statement at the beginning of the survey say i am committed to you know thoughtfully answer uh the questions in the survey your first option is that i'm not committed but i'm still gonna do the survey right and just doing that also tends to improve the quality of the data and people are being honest that they are not going to be committed to thoughtfully answering the questions of the survey but overall just having this commitment question in the beginning tends to improve the quality of the the data okay one last check for for any questions i think we have officially wound down um yeah guys this was great um i on behalf of all of the insights industry i am feeling very good about our future when we've got folks like like your presenters coming coming into the workforce you know keeping us in check from a from a quality standpoint um really outstanding job guys i'm you know this was this was great for me um so you know any parting shots marcus or team um i would say like you know every almost everything can be tested right so hey if you guys have an idea of something that you want to do research on research i'm going to be teaching this class again uh in the in the fall i'll be willing to do that i've done some things for some panels that say like panel aggregators they say hey we're getting data from these different 10 different panels uh and we are measuring nps for this company we're imagining over time you know how do i know uh which panels are reliable or not you know and they sent me the data and did all the analysis like here are the panels that are both below and here's who the ones that are trending with the different trends that you're seeing uh over time it's you know with the the right tools statistical tools is very easy for mo most of the time to test for all these kind of questions that you might have about the quality of the data how people are responding to your surveys or your experiments [Music] yeah if you don't have any more questions i would like to thank uh the opportunity that's a great opportunity for our students uh they are you know transitioning into their new careers they were graduating two weeks and probably between 15 and 30 days after that they're gonna start uh their careers and so it was a great opportunity for them to be like a you know new speaker kind of that conference uh and so uh i really thank accelerant and uh bill for giving us this opportunity absolutely and uh you know way to lead us off we've got you know seasoned career researchers who have to follow this and i think they're nervous at this point so um yeah thanks a lot guys uh looks like we've got about 10 or 15 minutes until the top of the hour when the next presentation kicks off so everybody hang tight you know grab a beverage um and we will be back at the top of the hour uh marcus and team thank you guys so much again thank you