Liz Castles, head of enterprise consumer insights at Lincoln Financial Group, walks through Lincoln's two-year journey adopting agile research platforms to close the gap between consumer insight and real-time business decision making. She covers the platforms adopted, how her team adapted traditional research processes to fit an experimentation mindset, and the hard-won lessons learned along the way. The talk is grounded in the practical tension between research rigor and speed, and how her team learned to navigate that trade-off inside a highly traditional insurance company.
Key Takeaways
- Define 'good enough' before you start. The shift from statistical significance to directional sufficiency is the core mindset change agile research requires. Teams must agree upfront what level of certainty a decision actually needs.
- Design small and plan to repeat. Agile platforms are built for three-to-five question studies. Trying to squeeze complex, multi-objective surveys into them causes fielding errors and negates the speed advantage.
- Remove budget as a gating conversation. Investing in unlimited-capacity platform licenses lets teams bring research into any decision without triggering a separate budget approval, which is often what keeps insights out of early-stage development.
- Use agile tools to earn a seat early, not to replace robust research. Short iterative studies build business confidence and surface consumer language issues well before a project reaches the stage where a full custom study is warranted.
- Leverage platform access while keeping specialist skills. Lincoln's team used the platforms to recruit and host sessions, then brought in trusted vendor partners to handle moderation and analysis, cutting timelines by nearly 50 percent without sacrificing quality.
- Expect a cultural and process learning curve. Traditional researchers naturally try to apply familiar design patterns to new platforms. A deliberate checkpoint with the team around 12 to 18 months in is useful to identify where old habits are slowing things down.
Questions & Answers
- Of the agile solutions you explored, which did you find most beneficial?
- Castles cited both UserTesting and Feedback Loop as heavily used, for different purposes. She singled out UserTesting's qualitative capabilities as personally most impactful, citing the speed of results and the quality of respondents, who are screened to be articulate and platform-proficient. She noted that even for relatively tough recruiting, such as B2B decision maker profiles, the quality of respondents met expectations consistently.
- Did you experience initial reluctance from your team or stakeholders to embrace agile research, and how did you overcome it?
- Yes, on both sides. Stakeholder skeptics defaulted to familiar objections about sample size and rigor. The response was to frame agile results explicitly around the decisions at hand and to position them as inputs into further research rather than conclusions. For the team itself, the challenge was breaking habits around comprehensive research design. Castles described a checkpoint at around 12 to 18 months where she found the team was still trying to apply traditional approaches to the new platforms, requiring a reset.
- Any failures or broken eggs along the way that you would share as lessons learned?
- Yes. The team pushed platforms beyond their intended complexity, adding display logic and screening that caused fielding errors and slowed everything down. Studies grew too large, essentially becoming mini custom projects that consumed the capacity benefit. The core lesson was that these platforms demand small, simple, repeatable studies, and getting the team to truly internalize that took sustained effort and iteration.
- Any tips for getting stakeholders on board and joining the research conversation earlier?
- Castles recommended securing a recurring presence in agile sprint meetings, even weekly, and committing to deliver iterative feedback on the questions teams were actively working through. Showing up consistently with timely, relevant consumer input, rather than arriving after decisions were made, was what opened doors. She acknowledged it does not always work and requires organizational receptivity, but the ability to deliver consumer voice at the moment of decision making was the most compelling proof point.
Session Notes
The Business Problem That Sparked the Journey
Three years ago, Liz Castles moved from Lincoln's B2B sales and marketing organization to lead its enterprise consumer insights function. Her first step was interviewing internal customers about how they perceived the team's work. The feedback was familiar: research takes too long, costs too much, and does not have enough impact on decisions.
Lincoln is a centuries-old insurance company built around long-term commitments. When Castles arrived in 2013, she describes it as landing somewhere in the early 1980s. By the time she moved into the consumer insights role, an enterprise-wide agility push was underway, and that created an opening to reframe what the insights function could deliver.
What Agile Research Actually Means
There is no single agreed definition. Castles describes it as faster ways to keep up with the pace of innovation and decision making. She frames it across three buckets:
- Faster data collection methods — omnibus studies, polling, DIY surveys via Qualtrics, and agile vendor partnerships. These have been around for a while but remain a core part of the toolkit.
- Faster research processes — technology platforms built by startups that remove friction from finding audiences, collecting data, and analyzing results. This is the main focus of her talk.
- Big data and advanced analytics — behavioral insight at scale. Still emergent at Lincoln, but the long-term frontier.
The Shift from Research Mindset to Experimentation Mindset
A key visual Castles references comes from CEB (now Gartner) and illustrates the spectrum from a traditional research mindset, focused on robust, statistically significant results, to an experimentation mindset built on trial and error, iteration, and directionality. The standard in the experimentation lane is not statistical significance. It is whether the insight is good enough to inform the next decision.
What is good enough? Good enough to help inform a decision that otherwise would not have leveraged any research or consumer insight to drive towards its ultimate end.
For a classically trained researcher, this felt foreign but also liberating. That framing, good enough to earn a seat at the table rather than good enough to be definitive, became the qualification test for when agile methods were appropriate.
The Platforms Lincoln Adopted
The agile platforms came into Lincoln not through the insights team but through innovation teams already using them to manage idea pipelines. Lincoln ultimately invested in two platforms: UserTesting (primarily qualitative) and Feedback Loop (primarily quantitative). These tools were built for non-researchers, product managers, UX and CX practitioners, people for whom insight is part of the job but who are not focused on research methodology. Out of the box they offer template-driven studies, managed sampling, and automated reporting.
Castles notes the landscape is evolving rapidly, with significant M&A activity and expanding capability sets. Lincoln is currently back in the market conducting an updated platform evaluation, two years after its initial investments.
How the Team Applied the Platforms in Practice
Early-Stage Concept Testing
The first live project was a concept that had not yet received any business funding. The team had a single one-page idea description. Using an unmoderated interview format, they targeted respondents based on basic demographics and investable assets and fielded a short set of open-ended questions. Results came back within a few hours. The recordings, actual consumer voice, were shared directly with the development team and with senior leaders whose funding support was needed.
Critically, the platform economics meant the team could iterate without reopening a budget conversation. When the team wanted to test a revised concept or a different angle, the answer was simply: let's do it again.
Moving Toward Hybrid Approaches
As projects moved closer to significant funding decisions, Lincoln built hybrid approaches. On the qualitative side, 60-minute moderated in-depth interviews replaced the short unmoderated format, with the platform providing access and the vendor partner handling moderation and analysis. That combination cut timelines by nearly 50 percent compared to a traditional approach. On the quantitative side, iterative agile testing condensed questionnaire design so that when a full custom study was needed, time to field was significantly shorter because so much foundational work had already been done.
Instead of saying hey that's great you've got lots of research objectives, we'll come back to you in six to ten weeks with answers, we were saying okay give us a couple of days, we'll give you this piece of it and then we're going to keep going.
Supporting a $30 Million Business Case
The toolkit was ultimately used as part of a market opportunity assessment supporting a 30 million dollar business case. Agile methods were not used to make the final investment decision. Rather, they built the body of evidence and consumer language insights that made the downstream robust quantitative study faster and more targeted.
What the Team Learned: Do's and Don'ts
Practices That Worked
- Plan testing in small, discrete chunks aligned to decision timelines rather than research timelines.
- Use agile methods as early as possible in the development cycle so consumer voice can shape, not just validate, a product or concept.
- Adopt an experimentation mindset: test, iterate, repeat.
- Simplify language aggressively. In insurance, translating internal jargon into plain consumer language was itself a major outcome of the process.
- Invest in unlimited platform capacity so budget is not a barrier to individual studies.
Mistakes to Avoid
- Do not try to compress a 30-question survey into a platform designed for three to five questions. Display logic, screening complexity, and branching cause fielding errors and erase the speed advantage.
- Do not treat agile studies as mini custom projects. The team initially produced studies that were too large, used too little of the platform capacity, and created standalone analysis burdens.
- Do not try to cover all research objectives in a single pass. The capacity is there to repeat and iterate, so use it.
- Do not position agile results as definitive for high-stakes decisions. Use them to build confidence and inform, then layer in more rigorous methods as decisions become more consequential.
- Remove corporate jargon and abbreviations from stimuli. This is especially difficult in complex financial products but essential for valid responses.
Stakeholder and Organizational Dynamics
Gaining internal buy-in required ongoing education. Skeptics pushed back with familiar questions: how many people did you really talk to? The response was to consistently frame agile results around the decisions they were informing and what stage those decisions were at, not to present them as standalone proof.
A turning point came when the broader enterprise agility agenda gave top-down support to the direction the insights team was already moving. The team eventually had a C-suite conversation about research methodology, which Castles describes as the first of its kind with Lincoln's CEO.
Getting into early-stage agile sprint meetings, even with highly technical internal teams like actuaries, allowed the insights team to surface consumer language issues week by week and build credibility incrementally rather than arriving at the end of a development cycle.
Where Things Stand
Castles is clear that agile platforms are one piece of a broader insights toolkit, not a replacement for custom research. The goal is to ensure consumer voice is present at the earliest stages of decision making, when it can influence design, not just position a finished product. The team is still actively learning, the platform landscape is still evolving, and the right balance between speed and rigor remains a work in progress.
Transcript
Read the full transcript
all right i think it's about that time um it's 11 a.m eastern so we are gonna go ahead and get started um i see some folks continuing to profile in um and that's perfectly fine but we'll go ahead and get rolling and folks can catch up as they as they sit down as it were um so welcome everybody to the july installment of the accelerant research virtual insights conference my name is bill mcgowe i'm ceo of accelerated research and i'm going to be your host for today's uh festivities um you know quick overview of these conferences and and what to expect um so basically we've got essentially just a series of hour-long virtual webinars um from some really good insights pros that are going to happen uh throughout the day um so every hour on the hour we're going to do a new topic a new speaker um and we're going to get started in just a few with our first yes who is liz castles from lincoln financial group um i'm gonna be your mc like i said i'm not gonna talk for very long uh that's not what we're here for so i am just gonna give a quick kind of overview you know this is now gosh we started these virtual conferences back at the beginning of shutdown last year um we've had several folks that have joined us for for several uh waves of of our events um other folks who are brand new so for their benefit we'll just give a quick overview and rundown of you know what this is all about um and we like to operate with a few pretty simple ground rules um number one being we're at the mercy of tech um we've got some folks that are starting to head back to the office um i myself am back in celerent's office uh but we've got a lot of presenters a lot of attendees who are still remote still logging in from from the living room bedroom you name it um so you know and we're at the mercy of whatever tech grid platform usage kids in the background dogs barking delivery people you name it um so you know we we just ask to be patient right we're all in this together we're all subject to the same interruptions and craziness um so we're not going to you know promise a perfectly polished you know wonderful presentation and show um accelerant research as a research agency we're not an event planning or conference organization so you know we are not going to apologize for anything that's a little bit less than you know completely polished no what we have is is you know a group of really good speakers talking shop you know sharing their own expertise in the in the insights industry and you know allowing us to to kind of follow along right um so we're we're excited to have that and to get started um you know second rule that we have along those lines is be respectful um because the virtual nature of this this conference we're not in person you don't run the risk of bumping into you know someone in the buffet line so you know there is at times the kind of you know anonymity that that folks can draw off of and you know the temptation to potentially troll um you know avoid that it's not something that we've really had to deal with but just you know keep in mind um you know these folks are volunteering their time um so be respectful but you know that's not to say that you shouldn't ask smart questions you know uh push back at times you know everybody that we're you know we have on our presenter roster is you know these are research and insights folks so we're used to having to defend the findings that we share and the things that we're talking about so you know there's no no harm in asking challenging questions um just again avoid the trolling uh third rule is uh to the extent possible you know this is a virtual conference we're not in person but as much as you can try to network engage interact with one another um not just our presenters but even other audience members um you know if you're logged into the zoom webinar you have the chat function you have the q a function that you can interact with but we've also got folks that are that are logging in from other touch points uh we've got youtube linkedin twitter that you may be accessing via and you know we ask that if you are on these social platforms uh use the hashtag ar vic that will help us sort of wrangle and keep track of of any chatter that's happening out there um you know introduce yourself to one another talk amongst yourselves um do these things possible shake hands virtually um yeah you know let's make it as engaging as possible right all right so we've got a good lineup of speakers today we're going to start off with those castles from from lincoln financial group uh at 11 o'clock which is right now so i'm going to spend just a couple of minutes we're going through our roster we've got our 12 o'clock speaker which is going to be dr chihan chobanolu apologies if i butchered that he's going to be talking to us about crowdsourcing platforms for market and academic research um coming in at 1pm is going to be christian kurtz from viacom uh giving us an overview of a talk called beyond 2020 global youth voices and futures and we're going to finish the day with sean good from jane.com batting cleanup and talking about democratizing research without jeopardizing your job so i think we've got a really really neat and compelling list of speakers um some diverse industries and diverse topics but i think you know all subjects that that we as research and inside pros can you know find great value in um so that's it for me i'm gonna kind of step off i'm gonna let liz castles get get rolling with her talk um i'm gonna hang out to you know to lead q a but you know you're not here to see me so i'm gonna shut down and let this do her thing thank you so much bill so while um bill and i do our swap here i just want to give a big shout out to to bill and his team um we are so appreciative um at lincoln for um this opportunity and and this forum um to be sharing um what we do and um have the opportunity to collaborate and you know meet with uh kind of under other industry practitioners i think to kind of share what we're learning really um and and really learn um from each other and so um i'm excited uh to share kind of our journey over the last two years to adapt and adopt an agile research toolkit but i want to start uh today with with a story um of how we came to that journey and i don't want to date myself um but uh you know my research career has spanned kind of the better part of the last three decades and um for for many of you who have been in you know in and around research during that time you know i've had the privilege i've worked on the vendor side um you know living through the transition of market research from phones to online to mobile devices i i worked on the tech development side um you know to to develop do do it yourself research tools and you know right now i'm i'm navigating what i find to be sometimes treacherous path right from small data to big data so um you know i'm a researcher who's open to change i like to try new things i like to innovate methods i really like driving continuous improvement both in process and cost efficiency so three years ago i had a fantastic opportunity to move from lincoln's b2b sales and marketing organization to lead our enterprise consumer insights function and i i did what any good researcher would do i sat down and i interviewed my customers to understand their perceptions their needs their challenges as it relates to the work that they did with this team and the message that came through loud and clear was not new um and quite frankly any researcher who lived through the changes over the last decade has heard the business complaint at some point that research takes too long costs too much and doesn't have the impact on decision making that it should um it's not new but it still hurts to to be sitting on the other side of the desk of a you know of a senior leader um or c level leader and hear these messages and so knowing that i had um a lot of catch up to do on the latest and greatest and kind of consumer research after taking a nearly 10-year hiatus kind of focusing on the b2b decision maker i dug in and what i found was an awful lot of buzz about agile research and now i started my career in silicon valley and spent a number of years working in consumer technology and i teach so i was very familiar with with agile right and agile development processes but when i landed at lincoln in 2013 a centuries-old insurance company right that provides long-term guarantees and commitments to its customers i knew i had traveled back in time um i wasn't quite sure where i had landed it took me a little bit lots of conversations with peers and colleagues but i had suspected that i had landed somewhere in the early 1980s where mainframe computers were still maintained to administer insurance policies sold and enforced you know since the 60s and 70s so i was really hard when agile and you know the technology transformation imperative finally arrived at lincoln and so it was timely and it was appropriate to use that buzzword as a way to transform thinking and perceptions about the value that my team could bring to an organization that was committed to a journey to bridge the gap with the customer um and so one of the um you know one of the reasons i like this quote it was one of the first things uh that had come up as i started your research right was you know this issue of speed right wasn't like i said wasn't new um but it was the real trade-off right that we as consultants first and foremost had to make um to say is the um certain and answer what's needed or is it the directional guidance and input and representation of the consumer kind of in the process what's needed so to really evaluate right the level um you know of urgency right the urgency versus certainty is is what stands out to me um as i kind of kind of embarked on this journey but what is agile research right there wasn't a pat definition there wasn't something you know out there that somebody could explain to me like you know okay what do i need to do to like start doing this agile research um the reality is is there isn't a defined set of methodologies or approaches or even one way to think about it the best way that i have learned to describe it is faster right ways to keep up with the pace of innovation and decision making and so the lowest hanging fruit were of course faster data collection methods which have you know been around a while nothing new there but they remain a really important part of our toolkit things like omnibus studies polling do-it-yourself surveys with qualtrics we're doing more of those um than ever but agile vendors like work that we do with accelerant and other partners where we have worked and partnered quite successfully to kind of reduce timelines and speed to deliverable on the other kind of end of the spectrum is what i referenced earlier right it's big data it's leveraging technology and advanced analytics to kind of understand behavioral insights and while that's still a very emergent area at uh you know at lincoln and i know many other companies it obviously gets us to a lot faster deeper insights when um when we can operationalize uh those models and and those processes um but what i'm going to focus on today really is in in this middle bucket here which is faster research processes and where a lot of new startup technology companies are upending kind of the research process as we know it and removing friction right from the most common research problems namely finding the right target audience collecting the data and analyzing and reporting the results so through my research and you know working with various industry partners i quickly came up to speed on um you know on a growing ecology of agile platform providers and these are mainly tech startups focused on solving the research problem for non-researchers and if you remember where we started this discussions these solutions kind of focus on speed cost efficiency and on-demand insight to kind of drive decision making in real time so now if you're from the it world or you're from a background and ux or usability research or even cx management some of these tools may look really familiar to you this is a space that's rapidly evolving there's lots of m a activity vc funded expansions and kind of an ever evolving capability set but after you know implementing two of these platforms just two years ago we're back in the market conducting an updated evaluation because of how quickly that landscape is emerging but what i'm going to focus on today is you know not so much the the tools themselves but rather what journey we've been through to adapt traditional market research processes to the agile toolkit and so this was an image that i came across early in my research from the folks over at chceb now gartner and you can see that i updated it because timelines for quick turn research have continued to compress in the three years since i started this journey but what i think this illustrates well is the shift from the research mindset which is as you know as i describe it an investment in a process a methodology that will deliver robust in-depth or statistically significant results to an experimentation mindset one that involves trial and error repetition directionality iteration and where the standard isn't statistical significance but whether or not those insights are good enough to help drive day-to-day decisions that ultimately have real impact on consumer value and experience um and this is something that um as a you know as a decades-long researcher um really appealed to me because you know you can often get caught up and i am the type of personality that you know is very interested as many of you know of the folks who come from a vendor background you know our um you know our salt is doing it right right um getting the methodology right getting the sampling right knowing that you've measured um you know variance and or error um you know in an accurate and complete way so to a certain extent um this felt foreign but at the same time was very liberating um and for me it really kind of stepped into that zone of you know what is good enough right good enough to help inform a decision that otherwise would not have leveraged any research or consumer insight to drive towards its ultimate end and and that became a really important uh distinction differentiation kind of qualification for when we would consider these approaches so what does it all look like in practice um we looked for platforms that would give us you know both qualitative and quantitative capabilities and these vendors came to us i can't take credit for bringing them you know into lincoln because they came to us through our innovation teams who were already using um them to manage their idea pipelines and so we looked to use them to start addressing business questions problems that weren't using any research at all because of the perception that research simply didn't fit into the development process and so we started with you know really low hanging fruit with things like marketing content um you know and then we moved on to things like product concept testing and more recently we've leveraged um you know this toolkit to tackle a market opportunity assessment um really supporting a 30 million dollar business case um but how we've done that how we've integrated it how we've um into a broader toolkit and how we have developed hybrid approaches and you know research um uh you know methods coming out of all of this has been the where the real learning has taken place certainly for for me as a researcher so just to give you a very quick idea of what comes out of the box with a platform like this as i said these tools were built for non-researchers product managers ux cx practitioners folks for whom insight is core to their job function but who perhaps are less concerned with the research process as as i described it so what they have delivered is fully technology technology-enabled research process where the user doesn't necessarily need to be concerned with sampling logistics reporting analysis um and the buckets described above are the templates right that exists um these are examples of templates that exist in one of our platforms that will enable our stakeholders to eventually self-serve i was really excited to see democratization of research kind of on the agenda today because this is all part of that process within my organization of giving and opening up access and capability to our stakeholders to gain direct access right to consumer insights in the context for example of an agile sprint right which is currently how not only our it team operates but increasingly our marketing and actuarial product teams are all leveraging those principles of agile to expedite and you know facilitate a speed to market that is still very new to our very traditional industry and so what this has looked like for us is really early on in our adoption cycle quite frankly this was something that we did as we were piloting a number of platforms and so we had an actual live project that um you know a lot of these platforms say hey try us for free and see you know see take it for a run and see how it works and so we leverage qualitative testing capabilities to get insight from consumers to help us shape an early stage idea that hadn't yet received any kind of business funding we had literally a simple one-page description of an idea you know from the team that we were able to test online in an unmoderated interview format uh to give the team um working on the concept feedback that came back literally over the lunch hour right so they had put together like i said this one pager um we put it out there on the platform we targeted you know at the time they had some hypotheses about who that target market might look like based off of some basic demographics age and wealth investable assets that kind of thing um and we just put it out there with you know um a short set of questions that said hey could you look this over and tell us what you think you know tell us what questions that are coming up for you tell us what doesn't make sense that you may need more information on um just to get a pulse on how is this sitting with with the folks that we think this could have some appeal for and we got that feedback um like i said super fast um and uh in a format that was engaging right we were ultimately able to take the recordings the actual voice of those consumers and share it out um you know with uh other folks working you know on the concept and or senior leaders who you are working to gain the funding support for and um given the the nature and the economics of these platforms i could as a you know insights consultants tell them like let's do it again right once they've made changes and tweaks and and or wanted to try a different angle like that was the really liberating aspect of all of this um is to not have to have the budget conversation every time they came to us with a question but rather to say hey we've invested in this capacity we want you to use it let's figure out together how to continue to test and learn that continued to evolve right so that was like i said one of our first experiences that was um incredibly enlightening for me obviously you know propelled us you know down a path of making some investments dollars that might otherwise have gone to custom projects with with vendors went into um you know funding a set of uh unlimited testing capacity on these platforms like i said for me it was an investment in tackling business challenges and problems where we weren't um earning our seat at the table for you know for a variety of reasons and so over the course of our journey we continued to experiment right um i continue to do my research our mind and you know with all of the statistics i wanted to see you know if we were going for something that wasn't as precise how you know what was that gap right what did that look like so i was running tests just to see if you know if we did a full custom study versus iterating on these platforms how different were the outcomes um you know what would it take to kind of you know close that gap how do we position that aspect of these platforms you know to to our stakeholders and so we continued to experiment leveraging the tools in a variety of ways we took a much more experimental approach to concept testing um where you know normally we might do you know focus groups um you know virtual yes but focus groups across the country with various you know audiences you know in some cases um you know parts of the team prior to the pandemic were still flying around the country to to sit in um on these kinds of discussions um or you know concept testing where we were still executing that you know with uh you know our our portfolio of vendor partners um and like i said lots of improvements there on being able to do that kind of work quickly um but we wanted to kind of you know really test the process see what we needed to do differently to really leverage the strengths of the platforms and really maximize what we could do in this kind of new approach and so one of the first things we did was really kind of plan out right the testing iterations um really thinking about things in very small chunks um and conducting much much smaller surveys right to help inform early stage decision making at the pace at which the decisions were being made and so instead of saying hey that's great you've got lots of research objectives we'll come back to you in six to ten weeks kind of with answers where in a lot of cases that wasn't a good enough answer anymore we were saying okay give us a couple of days we'll give you this piece of it and then we're going to keep going right um and and try to align our timeline to your decision-making timeline and so as the team move closer um you know to presenting their business case we move to a hybrid approach right so because as i said as you're working with the business um towards significant dollar investments the question about precision and certainty does start to shift right um over the course of that journey and so the question became you know as the team got closer to that um how do we start you know to incorporate these tools into you know what we might consider more kind of in-depth robust types of results and what we found is you know we were able to use some of the platforms and instead of you know kind of short five-minute unmoderated interviews um we were using them to do 60-minute in-depth moderated interviews where we were just getting quicker um easier access to consumers to to do those deep dives we continue to leverage you know our consulting partners our vendor partners to actually do facilitation and all those types of things but we were leveraging the access that we had through the platforms um you know to uh you know to to do the work that needed to be done and even on the quant side you know we work with a lot of vendors that do those full custom studies for us but we expedited and facilitated that process leveraging everything we had learned through the iteration on the early iteration and so in much cases the the questionnaire design process had been condensed significantly because we said hey our purpose here our specific objective is to validate right market sizing opportunity and so we're going to take these questions where we've been getting inter you know inter iterative feedback over the course of you know a period of time and we're going to consolidate that into a full-blown quant study um so that you know time to field and to get results was facilitated by the the work that we were doing in in the agile environment so what have we learned um and and many of these will be obvious right um as i think uh i hope that they're best practices that most researchers tried to strive for however i would say that in an agile process we have to take a more rigorous approach to things like simplification consistency and standardization as well as leveraging language that is super easy to understand and um you know this is especially challenging um in an industry like mine that's brought kind of new dimensions right to product complexity right we provide a simple benefit but we often do it with with lots of complexity behind it and we're terrible about not talking about that complexity um and through this kind of research we've been moving the needle on um you know the language and the education components and um you know the the consumerization of the language within the insurance industry has been a huge outcome from a lot of this type of work and so of course adapting to this experimentation mindset you know by taking that test and learn approach and iterating often using as early as possible in the decision-making cycle so that teams can build confidence over time so those are all the things that we learned and i won't spend too much times on on the don'ts because like i said you know i think they go hand in hand um but we also realize that we can't simply shorten right this isn't about getting our 30 you know 30 question question surveys down to 10. it's not that simple right these platforms don't have the robust um capabilities you know that a tool like a qualtrics might have or things like that just because um you know the the longer and more complex that you try to make that survey beyond the three to five question survey the more errors and problems with execution that we experienced and so there was a real balance that we had to strike of what is right and appropriate to field on a platform like this um you know these aren't fully custom studies we had to break ourselves um from our um tendency to uh you know to kind of map out all the objectives and try to cover as much as we could given that we were making substantial investments we don't need to do that anymore we're kind of you know freed from that financial constraint so what does that mean to break things down into really small testable chunks that can be woven together replicated repeated um that type of thing uh you know i think was it was a huge learning process for us um and that kind of comes through right don't tackle too much don't try to do too much keep it to what is just enough to get us to the next round of testing um you know we're not saying that these things should be used to drive the 30 million dollar business investment decision there's still a whole lot of other research tools in the quiver that are more appropriate when that's what's at stake and on the line but this is a really important component to like i said make sure that insights earns its seat at the table early when uh you know consumer insight voice of the customer can have an impact on the actual development and and design and before it gets too far down the path where i often said so much of our work was being putting lipstick on a pig it was built already and we had to then use research to position something whereas this is getting us in much earlier in the process um at least elevating um the consumer voice um you know a lot sooner uh so that we've got a better opportunity to kind of bridge uh bridge the gap and then goes without saying um to remove the corporate jargons and abbreviations but in our industry that remains kind of one of our our most significant challenges is to try to break this all down so i'm gonna end my presentation today um just by reiterating that these tools like i said are just one piece um not only of the agile research toolkit um but they're only one pillar of a comprehensive insights toolkit and i hope um that you've all gained something today from our learning journey and i'm happy to address any questions that may have come up so i don't know bill or team if there's anything that's come through through the chat but happy to take any comments shared war stories anything like that from the group hello all right i looks like we do have a question or two gathering um it's like lisa has of the agile agile solutions you explored which would be found most beneficial yeah um i have to say that we use both of the platforms that we invested in which were user testing and feedback loop quite extensively for very different purposes um i think one of the things that just personally i've enjoyed the most is the user testing platform and the qualitative because it really did rock my world uh to get results in over lunch um and uh you know to have that whole recruiting um and honestly the you know there's a lot of trade-offs um you know we have a lot of discussions about you know the professional panel member and all of these but the quality of respondent and the dialogue and discussion that we were having with folks was incredible um was a really high standard even in unmoderated interviews they're they're they are screened to be articulate they're on screen to be able to operate in their in the platform itself and to give feedback and as researchers as users of the platform we're able to give feedback back to them we're able to rate them as respondents we're able to you know say um you know i'm not sure this person met that profile but that doesn't actually happen that often and more often than not these are fantastic um like i said you know consumers uh that are in you know hit the money on uh target audiences we've even done some pretty tough recruiting we've actually done some b2b looking for decision makers and we have been able to you know we've been very comfortable um kind of with the quality of respondent that we've been able to find yeah i wrote down one one of your comments which i loved um in the presentation was you know leveraging platforms to get the work done um this tendency to you know just sort of adopt a new platform and accept it as it is or as it's intended um is something that i think we as researchers fall victim to too often right being able to you know adopt new tech and new platforms but then you know think of those out of the box ways to to use it for your own own purposes um i think that can be huge and to me like to me it's the fun part right you get really creative because every approach and methodology has its pros and cons and i think the more you know kind of the more options that we have from a toolkit perspective the more that we're able to like kind of bring together new combinations of products like i said some of the you know some of the you know more exciting projects for us is you know when we are able to work with a favorite vendor right who we love you know their facilitation capabilities and say hey we don't have time to do this over the next two to three weeks we've got to get it done this week but what do you say that you do the moderation give me your availability we'll hook you up um but you know you they still take on that you know that that that skill set which we value as well as the analysis that we value but we've expedited the process cut it by nearly 50 percent um to me those are the huge wins that we get when we've got these options at our fingertips yeah i mean it's it it blurs the lines between those those traditional roles of you know supplier and client that we you know als all used to like neatly fall into um but to the extent that you know both sides of the fence can be flexible and creative along the lines yeah it's it's an exciting time to be a researcher yeah let's see we had another uh another comment that came in from sophia hi liz thank you so much for sharing your journey embracing experimentation and building an agile research process you said these agile approaches came to the insights team via your innovation colleagues do you experience any initial reluctance from your team or stakeholders to embrace agile research and if so how do you overcome it 100 yeah right there's there's always um i i think resistance there's always um skeptics and there's always the folks that are skeptical of research period right so to there's no question that this gives those folks right a whole other set of you know challenges and how many people did you really talk to is that you know you end up back in the old conversations whereas you said if that was the answer we were trying to get we would be talking in another five weeks well after this thing had been put to bed right so there is a lot of education associated with this but the thing that um i think uh our business stakeholders um found most compelling was um you know was to get insight so quickly and uh you know our job was to appropriately position those insights um around the conversation and the nature of the decisions that were on deck right and to say look as you you know to really encourage folks to continue to say like if you know if there was still uncertainty and still areas that you know let's use this you know as input into a full-blown conjoint trade-off study that we're going to do at this point in the well that's going to take too long well we have we've gotten this far but if you want to be fully confident you know before we file you know that sec application then let's get that study done we'll do that you know we'll get it kicked off here you won't get it back till october november but it'll come in before you make that filing even though i know that that team was working to finalize the details well before that filing had to take care of and if we weren't there at that table at that time then that thing was going without any input from us and so balancing those two kinds of objectives um you know is is the art i think that we're still learning with our partners to like i said use this oftentimes to build the case to make investments in further research so agile is is very much a process right um any failures along the way or or broken eggs that have allowed you to make this omelette um that you'd like to share lessons learned that you sort of picked up oh yeah oh yeah um yeah like i said this presentation actually came out of um kind of a checkpoint you know that i did with my team you know probably about eight you know a year 18 months in because you know not only was this new to me this was new to my entire team you know we were rebuilding a team at the time right so i was we were hiring new folks you know we were getting introducing everyone to this and throwing them into the mix and at some point you know i kind of checked back in and said okay like how is this working and you know and what i saw was a lot of traditional approaches that we were trying to squeeze into um you know these new methods and so you know one of the things that we found especially in the quant testing um was just a lot of error right a lot of fielding mistakes right that the vendor was having you know certainly from a technology startup perspective there's always kind of a um you know degree of variability there but certainly the more complexity that we were bringing to the table from a design perspective because like i said they were built to deal with the non-research right so things couldn't have been more simple like when they said three to five questions they meant three to five questions like forget all of your display logic your screen like they were like like why are you asking me to do this that's not normally what we do so we push the limits of their capability because we wanted them to meet us somewhere and in that process there was mistakes there were errors there was the need to kind of redo to the point that you could look back and say was that actually faster than you know doing it the old way i'm not sure so there was a there was a lot of that and um you know breaking down kind of those old paradigms right of design and you know fielding and expectations around all of that is is a challenge right like you've gotta really experiment with what's going to work and um i think boiling it down right i think a lot of my team is used to you know getting a whole laundry list of research objectives and trying to cover as much as possible right in you know in a survey or in you know in a dialogue because you're in in that world you're maximizing the the value of your investment whereas this is a completely different economic you know model where you're investing in the capacity to be able to do it again and again and again and so you don't need to be comprehensive and in fact you need to be think small and think about repeating and or iterating um in order to you know in in order to really be making the most of the platform because one of the things we learned after the first year is we didn't use nearly the kind of capacity that i'd wanted to on the platforms because what we were doing was too big right we were doing like what teared into many studies that required a whole set of analysis and stuff on its own um and turned into like mini custom projects where i'm just like we've gotta push it even further um to where you know our stakeholders can consume the insights because it is that simple um and and so that's a that is uh an absolute work in progress that is going to be continuously um evaluated and and improve because like i said it doesn't come natural okay yeah i mean especially for someone who is you know the classically trained researcher this is this is a difficult thing to sort of wrap your head around and adapt um but you said it i mean it is very much it's an ongoing process and it you know you get there eventually or you keep striving to get there that's right striving right there's so much that's where all the learning happens is we're trying to get there i'm not sure that we ever will but um like i said we've got a lot more options to like you know and and our time to deliver is just has been incredible and and that conversation that i'm having with senior leaders this got attention at the c-level suite which i thought was amazing i think it was the first research method process conversation that we ever had with our ceo and so that alone um you know was motivating right that it that it was getting people's attention that we had a lot of top-down support for where we were moving because it aligned to kind of broader enterprise strategic objectives around agility um and so all of that you know definitely contributed um to our ability to gain support for this even though like i said in you know in practice in the field there's still a lot of you know challenging questions and skepticism that you've got to um you know that you've got to work through but the ability um like i said to deliver the voice of the consumer um to the table when the decision making matters most is i think that's what everybody's been super receptive to and along those lines uh rachel actually had a question any tips for getting stakeholders on board and joining the research conversation earlier yeah no um that's you know my team will say it's not always um you know the easiest thing and sometimes as we were still demonstrating these values you know it's particularly in organizations that quite frankly had written us off right that they they weren't doing this they had a pretty strong perspective about how valuable consumer insight was to their processes um and so um you know it took some work and some collaboration and partnership um you know talking about like you know resetting expectations about like okay um you're in an agile process you're moving really really fast um but if you you know include us right in those agile sprint meetings on a weekly basis we will you know do what we can right um to feed to you um feedback um through that process and you know it it was a real challenge i'm not sure that i would recommend it again because you know working in this case it was working with a bunch of actuaries who were highly technical um asking questions that we knew quite frankly consumers wouldn't understand right and you know they're always so reliant on the fact oh well an advisor will explain this but having done 10 years of b2b research i knew like no advisors actually can't it's like they're not experts they're not specialists in in our products and they don't know it like you know it and why we're putting you know all these kinds of things and so that in and of itself was a productive dialogue um but um like i said i think being able to come back week to week to that table with some insights that help them evaluate the the levers they were shifting and the kind of changes you were making i think did open the doors to further conversation um so that was the big win um still a lot of skepticism still a lot of you know healthy dialogue about what's the best approach to gather insight to inform the decision but um i in that particular instance um you know the the platforms the the certainly my team members earned their seat at that table we have a comment from john along these very same lines uh it comes down to recognizing and supporting leadership which has the humility to acknowledge they don't know everything and are open to research um it can be a baby and you know you've got a receptive organization liz um there are some who would be very closed off to such things and it is very much a you know a baby step and proof point type of process yeah one i mean i i no question that the broader enterprise dialogue around agility design thinking that was something that all the senior executives were going through and quite frankly you know one of the in those first rounds of conversations that i did as i stepped into the consumer insights role is talking to one of my key stakeholders in the distribution organization who said liz why do i need this when i can go down to starbucks and talk to 10 people and get input and i'm like oh my i god want the head of our distribution going down to starbucks and like taking a poll of 10 people and you know making a decision so um you know but that was you know that was the culture there was already a real receptivity to the need to do things faster i think though where it became very compelling um was when i was able to show them what we could do how much we could do for the same cost as one custom study so you know my budget is a consulting budget and so we have almost all the consulting dollars in our marketing organization goes and has been historically used for third party vendors and and things like that and quite frankly we used a very small chunk of it to get unlimited testing capacity um and so the amount of um our ability right to like i said remove budget from the conversation with our stakeholders um and just free them to come to us with questions in and of itself was really important to our business you know the the growth and the scaling of this process okay and when you can make it attractive to the first strings that's always always a good thing for sure that's right that's right um okay just last check i think that does it for q a liz you kicked us off great job thank you so much any parting shots uh nope just thanks again to you and your team for the opportunity for us to share and really looking forward to the rest of the agenda today because i think it it all is in this kind of same evolution and love to hear from kind of where others are really excited about the next session on crowdsourcing um because that might be our next frontier so awesome thank you so much um yeah audience uh give us about five minutes everybody check your email grab a beverage we will circle back in a few thanks a lot liz thank you
