Bill McDowell, CEO of Accelerant Research, walks through the company's framework for creative and ad testing, covering KPI scorecards, normative databases, and dial testing. Using publicly available ads tested on Accelerant's own panel, he illustrates what strong and weak performance looks like across the funnel, and explains why context, norms, and ongoing measurement discipline matter more than any single score.
Key Takeaways
- A single ad score means little without normative context. Build a consistent testing process and accumulate a benchmark database before drawing firm conclusions.
- Metrics split into two groups: recall, likability, engagement, and viral potential are more influenced by the creative itself, while consideration, intent, and recommendation are harder for any one ad to move and are often shaped by the category.
- Brand recall is table stakes. Highly engaging creative that fails to build brand association is a problem, regardless of other strong scores.
- Timing affects results for category-sensitive metrics. A fitness brand ad tested at New Year's will show inflated consideration and intent scores compared to the same campaign tested two months later.
- Dial testing works best as a diagnostic, not a go/no-go gate. Use it to identify specific scenes or moments driving positive or negative reactions, and to inform edits when cutting a long-form spot down.
- Start simple, then add complexity. Stand up a standardized evaluation process, begin testing ads, and layer in custom audiences, custom norms, and custom brand questions over time.
Questions & Answers
- How do you account for variances in target audiences when different brands have different norms?
- In the examples shown, the focus is on mass media and general consumers. For individual clients, Accelerant builds a custom audience and a custom set of norms, applying the same core methodology but targeting the specific category or audience. These customizations take shape over time as more ads are tested within that context.
Session Notes
The Creative Testing Funnel
Research needs vary depending on where a piece of creative sits in the development pipeline. The questions worth asking at each stage include:
- What should an ad say? Qualitative and quantitative consideration driver research identifies the inputs before creative development begins.
- How should it say it? Articulation research tests early-stage creative as messaging and executional approaches take shape.
- Where should it run? Channel and media planning decisions benefit from audience and context data.
- How well did it work? Finished execution evaluation, the focus of this talk, assesses polished or near-final creative against a standard scorecard.
The Scorecard: Core KPIs for Ad Evaluation
Accelerant uses a consistent set of KPIs across ad tests, applied to a census-representative sample of 250 American consumers. Using identical audiences and methodology, tests are normalized against a growing database so any individual score can be benchmarked against prior results.
- Brand recall: Does the audience remember which brand the ad is for?
- Likability: Do people like what they are seeing?
- Engagement: Would they pay attention to this ad?
- Viral potential: Would they share it on social media?
- Consideration: Does the ad increase likelihood to consider the brand?
- Intent: Does it drive likelihood to take action?
- Recommendation: Does it make them more likely to recommend the brand?
- Relevance: Does the content feel relevant to them?
- Competitive comparison: How does the brand compare to others in the category?
Two Tiers of Metrics
The top of the scorecard (recall through viral potential) is more directly influenced by the creative execution itself. The bottom (consideration, intent, recommendation, relevance, comparison) is harder for any single ad to move and is heavily shaped by the product category. Financial services, for example, carries a much higher barrier to switching than a quick-service restaurant, so consideration scores will run lower even for effective advertising.
Brand recall I always consider table stakes. If you've got a really engaging, really compelling piece of creative that people aren't remembering who the advertiser is, that's a problem potentially.
Examples: Strong and Weak Performers
Apple Watch
The ad using real 911 calls from Apple Watch emergency situations scored very high on engagement and likability. Brand recall came in lower, raising the question of whether emotionally driven storytelling sometimes overshadows brand attribution.
Liberty Mutual
As a financial services and insurance brand, Liberty Mutual showed lower bottom-funnel scores (consideration, intent) against broad population norms. Filtered to financial services and insurance norms only, performance looked more favorable. Category context matters when interpreting these results.
Burger King
As a low-barrier category, Burger King showed the expected pattern: consideration and intent scores were easier to move relative to financial services benchmarks.
DirecTV: Serena Williams as Wonder Woman
This spot scored well on engagement and was a notable performer on viral potential, a metric that most ads do not move. The follow-up execution placing Serena Williams in a Matrix theme performed considerably worse, demonstrating that a strong first execution in a campaign does not guarantee results for subsequent spots.
Energizer Bunny
Despite first appearing in 1989, the Energizer Bunny campaign still scored favorably, illustrating the sustained equity a strong creative icon can hold over time.
Planet Fitness: The Timing Effect
A Planet Fitness ad tested around New Year's resolution time showed unusually high consideration and intent scores, metrics that typically do not respond strongly to creative alone. The same campaign tested a few months later produced much lower numbers. The takeaway: category timing can inflate or suppress bottom-funnel scores independent of creative quality.
Super Bowl Ads
- Frito-Lay 'Push It' (featuring a sloth and jungle animals): top overall performer in the Super Bowl set tested.
- Booking.com and Sam's Club / Kevin Hart: solid performers.
- Bud Light Seltzer 'Flavor Town': lower ratings overall.
- Uber 'Don't Eats': low overall ratings, but viral potential scored above norm, suggesting even divisive creative can drive attention and sharing.
- Budweiser Clydesdale and Kia Robo Dog: both showed dips in dial test scores during moments where the animal or robot was injured, which reflects intentional emotional storytelling rather than a creative flaw.
Dial Testing as a Diagnostic Tool
A moment-by-moment dial test, shown as a continuous line during video playback, adds diagnostic depth to the survey scorecard. Accelerant recommends treating it as a tool for identifying specific scenes or moments that resonate or turn people off, not as a pass/fail mechanism.
- A steady build toward a peak, as seen with the Google Shopping ad tested, represents a strong dial test pattern.
- Dips during emotionally charged moments (an animal getting hurt, a tense rescue) are often intentional and expected.
- Dial testing is particularly useful when editing a 60-second spot down to a 15 or 30-second version, helping identify which elements are essential versus cuttable.
- It can also surface unintentionally controversial moments before launch, something that historical ad failures demonstrate can have real consequences.
For non-video formats, equivalent diagnostic tools include heat mapping and eye tracking for print, and neurometric measures for other media types.
Practical Guidance for Standing Up a Program
- Start with a standard process and a general audience. Accuracy and consistency of method matter more early on than customization.
- Add custom audiences, custom norms, and brand-specific questions once the baseline is established.
- Test on an ongoing basis. The value of norms compounds over time as your database grows.
- For B2B or niche categories, mass media norms may not be appropriate. Plan for a custom audience and custom benchmark from the start.
- Recognize that consideration and intent scores carry category-level context that a single creative execution can rarely override.
Transcript
Read the full transcript
all right welcome everybody um my name is bill mcdowell i am ceo here at accelerant research and you are here for our 1pm webinar on anatomy of a creative test uh thanks everybody for joining um if you have any trouble seeing or hearing me let me know in the chat or the q a otherwise i'm going to keep drawing on and we'll get rolling so again uh my name is bill mcdowell um i'm ceo at accelerate research um if you have joined us before for some of our our virtual insights conferences that we've been running for the past couple years now um welcome back thank you we're going with a little bit different format today and we're gonna mix things up a little bit over the course of this year with some of our uh virtual networking events um today we're just doing a straight up half hour one-off webinar um takes a little bit easier on you and your schedules um but we're going to be talking about uh creative testing so adjustment right uh before we get rolling i'm just going to run through a couple of ground rules and and you know kickoffs um this is a webinar that's put on by a research company uh we're not a an event planning company so if we have tech issues or trouble forgive us but you know zoom has tended to be our friend when when we've done these types of uh webinars and conferences in the past so you know i expect things to go pretty smoothly i'm going to try to be interactive with this presentation so you know certainly feel free to hit me up in the chat or the q a um ask questions i'll try to keep an eye on that um but i do tend to you know get sidetracked and you know drone on with my commentary so apologies if i miss a question or a comment that's passed but certainly you know here's my contact info on this this slide um feel free to you know reach out to me email linkedin twitter you name it um engage you know let me know what you think if there's any requests for for future talks or subject matter definitely feel free to you know to run that past me as well um before we get into the subject of today's uh webinar i did want to just make a quick you know save the date announcement speaking of uh digital conferences we're going to be hosting at the end of next month uh one of our virtual insights conferences um i said if you've been with us over the past couple years you've seen that we've done what began as we what we called the quarantine virtual insights conference this was you know when none of us could actually get out and copeland was setting in uh we started running these these virtual conferences just a chance for insights folks to get together talk short shop and nerd out do a little networking you know these are going quite well so we we continue to do them if you have suggestions for uh topics or if you'd like to present yourself definitely feel free to you know take a look at accelerant's website um go to the conferences and webinar page submit your topics or feel free to to join in and register for for the next event so that's that uh but what we're going to talk about today uh is going to be ad testing uh creative testing more specifically and what i what i'm gonna plan to do is try to share with you some examples of the types of advertising and creative testing that you know our team here at accelerant runs and likes to you know publish results of hopefully give you a little bit of inspiration for you know how you might apply some of this to your own practices uh your own brands your own clients um you know it meant to be a bit of a best practices using our own techniques as an example um you know if you can rip off some of this and inspire yourselves for your own work um that's pretty cool so you know conceptual framework you know when it comes to advertising testing and creative testing uh there's a lot of different ways to to go about um this is you know the the framework that we like to use when approaching ad testing um you know and this won't be the first funnel that you i'm sorry this won't be the last funnel that you see in this presentation but um you know depending on where you are in the creative pipeline uh your needs for research and insights could be quite difficult um so if you're you're all the way at top of funnel you know just embarking on this getting into you know creative brief and what to even begin including um your research needs and the type of research you might be doing is going to look a lot different than you know if you've got you know real polished ready for or even in you know market advertisements um so at the top of the funnel you know we like to to catch this in terms of you know the questions that you're looking to answer with it within a bit of research right so what should an ad say um how to say it uh where to say it and how well did it work um and that can start all the way at top you know with qualitative and or quantitative consideration driver research really getting at you know what are the elements and inputs before we even get rolling with you know actual creative development what are the ingredients that we should be including right um okay so next step articulation all right we determine the stuff and the the ingredients that we need to input into our advertising um how should we go about actually communicating these things um now we're getting into the point where creative is being developed and we want to to begin testing and that can again be you know in some cases qualitative in some cases quantitative examples that we're going to show today so we're going to focus in on more the bottom of the funnel so the the finished execution um how you might filled out uh beginning to set up an ad testing program and some of the things that are useful to include in such assessments um and just real quick this will be the only sales piece of the day uh but just and this is mentioning so at accelerant we have a a panel uh we call it aggro usa it's you know our database community of research participants who take part in research studies um we like to get these folks engaged in you know not only work where end clients are hiring us to do research but also you know self-funded um just almost secondary type research right and that's what we'll be showing results of today so not ads that have been tested on behalf of clients that are necessarily proprietary these are things that we've tested ourselves using our panel um using our process so we can actually share results which everybody's always excited about you know not being as black box right okay so when it comes to creative ad testing so the examples i'm going to be sharing and showing um are going to be more finished ads like in market tv ads but this process can apply to really any type of creative assessment so it could be you know tv spots digital social prints out of home radio even rough cut storyboard automatic animatic type executions depending on what you have and where you are in the process your types of questions can vary but what what we find is that you know a lot of the core questions that we ask in in a survey instrument are going to be quite similar uh regardless of um the type of creative that we're testing right you know i've got examples over here on the on the right side slide you know kind of a score card that we um tend to assemble this is our our typical recipe of kpis when it comes to ad test right so you're starting again with another funnel but you're starting at the top with you know to what extent are people recalling the brand that we're trying to advertise and communicate in this this bit of um ad advertisement uh you've got likability to what extent you know people actually um like what's being shown to them engagement and again we you know we're kind of funneling down and getting more precise so engagement uh extent which they think they would pay attention to this um this thing that we're showing them uh viral potential likelihood that they would actually you know see an ad like this and share it with others on social media um consideration likelihood to consider the brand that is being advertised intense um likely to actually take action based on the advertisement um and then over on the right side of the page we've got you know to what extent this ad makes you likely to recommend the brand um you know how relevant do you find the content and how does this the brand being advertised compared to others right so some pretty you know standard evaluative measures um but i will say that and i can flip over to the next page um and this is kind of an example of how one of the these ad tests looks um so this is one that we ran doing this zillow ad um and how their their ratings shook out on each of these kpis that we showed in the last slide now the thing to to realize and recognize is that okay if i'm looking at if i do a one-off survey of my ad and i have a likability score of 73 so what um what does that mean um very little unless you have some sort of context right um and that's where you see so we have these ads you know we're showing raw scores but then we also have you know the the thumbs up and down indicators at the bottom of the page now what these do are show how these ads stack up against our normative database so we test ads over and over again using the exact same process using the same audience um not the same people necessarily but identical audiences right so what we do is what's called a nomadic evaluation so you know and think about your you know laboratory setting when you're doing research right test and control you want to have the environment as equal as possible um the only thing you're changing is the stimulus so that's what we do in this case so we do a census rep 250 american consumers um and we test the uh exact same process only changing the the piece of of creative um and breaking down further so this this scorecard that we're showing here right um so we've got you know and before i jump into this i think you know it's worth saying that you know this is kind of our recipe what we find to be effective uh questions to ask um ways to go about evaluating creative um kind of on an overall basis right um again this is a mass audience that we're looking at so if you start to get a bit more niche in nature if you're advertising for example in a b2b audience this type of kind of mass media evaluation may not be as appropriate um the questions that are being asked so remember we talked about those you know topo funnel consideration research right so you've identified what are the things that really turn people on and off that you're looking to to impact those may be very unique to your brand so these are custom questions that you may be wanting to include in your advertising um i see robert was asking a question i think i was kind of getting into that but essentially how do you account for variances and audiences target audiences can be different in terms of normal measures yeah trying to you know in in this case um in the examples that i'm showing we're talking about mass media right general consumers um we tend to take a very targeted and custom approach when it comes to the individual clients um and what i'm not showing you and something that you know we build for for in individual clients and oftentimes is basically a custom audience custom set of norms custom set of questions that we you know apply essentially the same um type of uh approach to uh but it's more targeted to you know the particular client's um audience or or category so breaking the scorecard down a little bit further um so we're starting at you know kind of the the top of the funnel and we're looking at so brand recall likability engagement viral potential these are the things that tend to be more influenceable by a given piece of creative right so the ad itself you know it is able to influence these scores a lot more the stuff well and i will say that all right so recall likability engagement viral potential yes that is the stuff that you know is influenced by the the piece of creative itself uh the brand recall you know i always consider that table statements right because if you've got a really engaging really compelling piece of creative that people aren't remembering who the advertiser is you know that's a problem potentially right um and this is you know something that fantastic creative can sometimes do is lose sight of you know the very point of the research and that's creating that association between brand and you know the creative itself um so that's why i say brand recall table states if that is low um that could you know represent a serious issue the stuff at the bottom here uh the consideration intent uh likely to recommend uh relevance uh comparison to other brands these are much more difficult for a given piece of creative to move the need along um and these often so you know like i said i'm gonna go through some examples show some you know kind of strong performers and weaker performers on a lot of these measures but these are often subject to the the product category or the the subject matter um you know likelihood to consider a given brand even though we catch the question in terms of you know to what extent this advertisement is you know pushing that likelihood people are grounded in terms of you know the category that they're operating so for example financial services um much higher barrier to entry and switch than you know something like a qsr or something like that right you know or you know easy you know packaged type of category so you know these considerations intent when you're looking at you know again mass media across multiple categories and brands some of these you need to take with a grain of salt or an understanding of where you're operating right um and that's why norm's you know while great you you need to have an understanding of what you're looking at what you're up against and you know be able to explain accordingly and these are things that start to take shape over time if you're just standing up uh an ad evaluation program you know i think it's often easier to just just get started begin testing ads you know start to set up this process in this discipline um and then over time start to layer in some of these more you know these additional complexities right so what i'm going to do now is just essentially go through a bunch of examples of you know different types of ads some stronger and weaker performance right so this is an ad for uh apple watch um really engaging piece of creative people are really really compelled by it right uh and this is you know if you haven't seen it before this is essentially just the audio is uh 9-1-1 calls of people in emergency situations you know on their apple watch uh essentially being rescued they maybe didn't have access to the phone but they did have the watch and you know happy ending because of the apple watch um so really strong ratings across the board people were really really compelled by this advertising but you know that brand recall at the top a little bit lower um and i don't know maybe the argument could be made for for an ad like this really pulling the emotional heart strings um maybe that's somewhat okay right um next ad so this is liberty mutual again i mentioned financial services um you see you know these really lower ratings at the bottom of the funnel for things like consideration and intent you know that compared to other financial services so again we're looking at you know the broad norm across general population we dig into to financial services only and insurance specifically well it doesn't look so bad in that case um this example is you know kind of the counter to that and this is a burger king ad again lower barrier um so the consideration and the intent um maybe a little bit you know easier to blow um let's see this was one i wanted to to just take a look at um i don't know if you recall this um i believe this was summer fall of last year but the serena williams uh direct tv wonder woman spot scored really well um it moved the needle on on this viral potential which is actually something that you often do not see most ads don't do very well as you can imagine i mean it's it's a commercial how often are people actually going to be going to the trouble of you know sharing it with others uh but this spot really did do quite well um in that regard as well as engage so you know really you know getting the eyeballs as it were right but what was interesting about this spot and something that i wanted to share here is you know you've got same campaign same thing but a much different result so you know just because it worked you know the initial spot doesn't necessarily mean that it does with some of the the follow-ups in the campaign um so this was the same same basic theme still serena williams um but in this case she's embedded in the matrix as opposed to wonder woman and you know a much less favorable result um and this is just you know a good example of you know i think we talk so we talked about you know campaigns over time right i mean look at the energizer bunny right it's been around forever what was it 1989 i think when this this was first introduced um so you know and still doing pretty favorably right so big swing for for serena energizer bunny still killing it as a as an icon right this was an interesting one that i wanted to share and this was for for planet fitness so these top of funnel metrics so the recall likability engagement these tend not to be uh subject to when an ad is tested um but this is a good counter to that right so you know the really cool thing about an orbit of um and kind of standardized setup as we're describing here is that i can test an ad using this recipe i can test you know i've got another spot that's ready to go you know six months from now and i can test it in the same manner among the same audience and compare and contrast the results right and there's a lot of power to be able to you know start building this you know repository of of ads that have been tested over time um so the previous example i was showing so this was again planet fitness this was right around new year's resolution time you know through the roof scores um especially you know in the consideration and the intent um these are things i said you know an ad is typically not likely to move the needle there and these from a brand standpoint tend to stay pretty pretty um pretty consistent from ad to ad right but then another planet fitness ad same campaign uh which was you know i think a couple of months later after new year's resolution time um much worse result from consideration an intense standpoint because again it's not that that timeliness right so you know back to the overall snapshot and the uh the scorecard that we're looking at um the one thing in the playback that we've been showing in a lot of these examples and you see is this this kind of heartbeat line under the the video playback and that's you know a dial test um so you know you got on the left is more of your evaluative survey-based metrics on the right is more of a diagnostic tool um when you can insert something like this into your ad testing it tends to you know just improve the richness of the the feedback that you're getting um my recommendation tends to be not to use something like a dial test result as a go no go decision uh but more of a kind of nuance right what are the you know specific elements within your ad that are really turning people on or turning people off now a dial test can be really effective for uh that kind of pre-launch gut check if there is maybe there's a controversial scene in your in your ad that you maybe didn't realize internally you know once you take participants through this evaluation maybe there's something that really you know tanks from from a dial test standpoint that's something that you want to know before uh before launch obviously um there have certainly been you know historical examples of of that having not been done and and ads that have really you know gotten some bad pub as a result um and this is something that all right so we're talking about video examples but you know you can work in some sort of uh a diagnostic assessment to you know any other media type as well um you know if it's print uh you can get into you know heat mapping for example um high tracking or or you know some of your neurometrics as well again trying to get at you know what is that that nuance what are the specific elements of your ad that are doing well or poorly and often this is a really good tool to use for so for example you're testing a 60-second spot and you need to determine for the you know the 15 or the 30 second version you know what are the elements that you know must be included what versus what are the things that can be sort of cutting room floor um you know this is a really good way to to go about assessing that and figuring that out um and i've got just you know a few examples of of dial testing here so this was a google um google shopping ad which you know this is the heartbeat line that you you're really going for as an advertiser right it's this sort of constant build up to a crescendo um this is what you would you know consider a really strong performer from this assessment right uh versus got a little caesar ad here where you know you can see ad starts out well and starts doing pretty good but um i know we're getting a little short on time so i'll kind of zoom through some of these but once the robot comes on screen uh people get kind of turned off by this bad boy a couple of super bowl ads which you know had some some valleys to to their donald test uh the clydesdale ad and in these cases you know the valley's not such a bad thing right i mean it makes for a more compelling experience you know when we see the the the down slope is when you know the the animal gets injured or when he is having trouble you know that's just taking folks on an emotional roller coaster i certainly wouldn't argue that that is a a bad result of the ad it's kind of you know what is intended right um we also saw the same thing for another super bowl ad which was uh robo dog by kia um when the you know little robotic puppy who falls off the roof uh and is injured you know people don't like that as much but you know it's part of the story right got several other super bowl spots here um you know i'll run through them pretty quickly because i think we're just about out of time uh if you have any questions definitely feel free to submit those in either the chat or the q a i'll i'll try to get to those but otherwise i'm going to go through a few more examples and kind of let you guys on your way uh this was a strong performer um this was booking uh sam's club and kevin hart was solid as well not so much when it came to the bud light seltzer uh flavor town ad folks didn't didn't tend to be as drawn to that one big winner of the super bowl ads uh was definitely this spot uh for free to lay and this was the push it ad with the sloth and all the various uh animals in the jungle so this was definitely a big winner and the not so big winner was the uber don't eat ad don't eat sad so you had stiffler's mom and was it greg from uh succession uh and various other folks um with this really weird spot uh you know ratings were really low for this but very interesting was the the viral potential score which you know was above near um which is interesting uh you know you could argue that you know some of these lower ratings maybe not as bad a thing because obviously you're getting people's attention and they're going to be talking about and sharing right right so quick webinar um i think we're running out of time or actually out of time so thanks everybody maybe this should have been an hour i think i could go through lots lots more examples um definitely feel free to hit me up if you have questions um you know i certainly encourage you some of what we've talked about here for your own uh ad evaluation uh but definitely you know put your own spin on it right um these are easy to stand up and get rolling i just definitely recommend you know doing some type of assessment on an ongoing basis right all right everybody um really appreciate it i will record uh post a recording of this webinar so if anybody missed any of it definitely you know you can catch the rerun verse all right everybody thanks so much