Joe and Molly, a father-daughter pair with combined experience spanning quant and qual research across agency, in-house, and independent firm settings, discuss how the market research industry has changed over four decades. They cover shifts in methodology, client relationships, deliverable formats, sampling, the agency-versus-in-house debate, and the role of AI. The conversation is candid and conversational, drawing on personal career stories and practical advice for researchers at all levels.
Key Takeaways
- Agency-side work early in a career exposes researchers to diverse industries, methodologies, and problem types, which builds adaptability that in-house roles often cannot provide.
- Strong client relationships require operating at two levels: business-challenge conversations with senior stakeholders and methodology conversations with in-house research staff.
- A key technique for redirecting poorly designed research is asking the client what they will do with the findings before the study launches, which often surfaces whether the question will actually drive a decision.
- In-house researchers risk being farther from decision makers, which can create pressure to deliver findings that confirm what leadership already believes.
- Authenticity and self-awareness matter more than credentials when entering the industry; relying on AI-generated portfolios or resumes undermines the genuine voice that qualitative work especially requires.
- AI tools like Notebook LM can democratize research for budget-constrained clients, but the synthesis, contextual judgment, and human relationships at the core of research are not easily replicated by AI.
Questions & Answers
- What are the opportunities and challenges for small research firms now run by individuals from larger firms or in-house backgrounds?
- Molly noted that when more corporate leadership enters smaller firms, flexibility and researcher autonomy tend to decrease. Joe observed that large-firm experience shapes a very different selling approach, one focused on specific methodologies or practice areas rather than broad business growth conversations. The overall pattern has been consolidation rather than smaller firms expanding, so this combination is relatively uncommon in practice.
- Was there ever a desire to move from agency to in-house?
- Joe said no. He left a large bank specifically to escape corporate politics and never regained an appetite for that environment. He acknowledged that many capable researchers thrive in-house and it simply was not the right fit for him.
- How did you help clients focus on user or customer problems rather than jumping to solution-specific questions?
- Joe described having a direct conversation about what the client would do with each possible finding. If a given result would not actually change a decision, the question probably should not be on the survey. This hypothetical challenge often helps clients realize on their own that their proposed questions do not serve their actual needs.
- How do you advise a client to change a poorly designed research plan?
- Joe recommended gentle but direct challenges using hypothetical framing, acknowledging that sometimes the research still gets done in a less-than-ideal form. Molly emphasized a collaborative, empathetic tone rather than a corrective one, framing the conversation as building on the client's goals rather than overriding them. She used the sandwich approach: acknowledge strengths, suggest the change, reinforce the shared objective.
- What advice would you give early-career researchers about where to start?
- Both recommended starting on the agency side to gain broad exposure to industries, methodologies, and client types. Molly added that if you do not yet have a clear passion for a specific domain, agency work lets you explore before committing to a specialty. She also advised against using AI-generated application materials and emphasized staying authentic to who you are as a researcher.
Session Notes
Speaker Backgrounds
Molly studied anthropology, earned a master's at the University of Denver, and initially planned a career in museums. She entered research through a connection her father had at 2020 Research (now part of Schlesinger/Sego), then worked at Corona Insights, Answer Lab, and currently leads rapid research at Google.
Joe began in financial services as a regulator and banker. He conducted his first market research project in 1979 for a small bank. He later co-founded a small firm specializing in bank research, ran it for seven years, then joined GFK to lead its banking practice, followed by a similar role at TNS (now part of Nielsen). He retired from research roughly ten years ago.
How the Industry Has Changed
Client relationships and the rise of in-house research
In the early 1990s, most client companies did not have in-house research staff. Agencies sold directly to senior executives and were often treated as outside experts. As in-house research teams grew, the buying process shifted, and agencies increasingly had to communicate with both senior managers and internal researchers simultaneously.
- Early agency work meant positioning around business growth, not specific methodologies.
- As in-house teams expanded, agencies had to sell at two levels: business challenges to executives and methodology details to researchers.
- Industry consolidation has been ongoing, with smaller firms continually acquired by larger ones.
Deliverable formats
Joe described early reports as long, text-heavy narrative documents with a handful of graphics. The invention of PowerPoint shifted the industry toward visual, story-driven presentations. Molly noted that her career, which started in 2016, has always been defined by slide decks, with video clips and documents as occasional additions.
Sampling and methodology
Quantitative research once relied heavily on random digit dialing (RDD) for telephone surveys, as well as mail surveys. Regulations now limit phone outreach significantly. Molly noted increasing use of hyper-targeting enabled by large data availability, and flagged early discussions at Google around synthetic users and AI-generated respondents as a potential future direction.
Quant vs. qual split
Joe described being methodology-agnostic throughout his career, choosing the approach based on the research question. He observed that today buyers often come to agencies already attached to a specific methodology, whereas previously clients brought problems and left the design to the researcher. Both speakers noted increased specialization among researchers, which connects to the growth of in-house roles defined by discipline or industry.
Agency vs. In-House: Pros and Cons
- Agency work provides exposure to many industries, clients, and methodologies, which builds versatility early in a career.
- In-house roles offer the chance to see iterative research build over time and to be closer to actual product or business outcomes.
- In-house researchers can face pressure from non-researcher stakeholders, particularly designers and product managers, to deliver findings that confirm existing directions.
- Smaller agencies often allow more autonomy and creative flexibility; larger firms provide name recognition that helps when selling to senior executives.
- Joe found that small firm environments also allowed him to avoid corporate politics while still doing rigorous work.
Client Collaboration and Research Design
Both speakers emphasized that the core skill in client work is helping stakeholders clarify what decision the research will inform. When clients arrive with a predetermined question list, the researcher's job is to challenge them constructively.
Let's say we learn that 73% think that. What does that mean to you? What are you going to do with that?
Molly added that in low-research-literacy environments, such as nonprofits, the framing must be collaborative rather than corrective. She gave the example of a client who planned a statistically significant quantitative survey in a Colorado mountain town of 800 people, and described redirecting that plan by leading with enthusiasm for the client's goals and then offering an alternative approach.
Joe noted that sometimes the research gets done in a form the agency would not have chosen. That is part of the reality of client services.
Research Ethics and Avoiding Biased Outcomes
A recurring theme was the risk of research being used to confirm decisions already made rather than to genuinely inform them. Molly described receiving praise from clients for writing unvarnished reports, which she found both unsettling and validating. She also raised a well-known ethnographic case: a researcher hired by Nokia in the early 2000s conducted fieldwork in Korea and concluded that consumers wanted smartphones. Nokia did not act on those findings, and the company's market position declined sharply.
As researchers we have to know that we're right. But unfortunately we can't always be those decision makers and that's part of it.
The speakers agreed that even when research feels undervalued or ignored, researchers must maintain their standards and advocate for rigorous, ethical work.
Networking, Mentorship, and Career Development
Molly highlighted Women in Research (Wire) as a meaningful mentoring resource, and QRCA as an organization with accessible conferences, webinars, and a magazine, noting that not all content requires membership. She encouraged early-career researchers to reach out on LinkedIn to mutual connections even when it feels uncomfortable.
- Molly received the QRCA Young Professionals Grant and attended a conference through that program.
- Both speakers recommended building relationships intentionally at each employer, because colleagues move to Amazon, Meta, and other major organizations over time.
- Molly cautioned against AI-generated resumes and portfolios, emphasizing authenticity as particularly critical in qualitative research roles.
- Joe advised that understanding the business context of research, not just the methodology, is what differentiates strong researchers.
Lightning Round
Most overrated metric
- Molly: Net Promoter Score. Also flagged the general risk of forcing metrics onto qualitative research.
- Joe: Agreement on NPS, with nothing to add.
Underused method
- Molly: Triads. Three participants limits groupthink compared to full focus groups while still generating richer conversation than one-on-ones. Creates a more equal dynamic between moderator and participants.
- Joe: In-person, one-on-one depth interviews. Expensive and hard to recruit for, but the depth of learning is unmatched. The challenge is that findings are difficult to summarize quickly for senior managers.
Craziest research story
- Joe: A bank client wanted to use a word in advertising without knowing it had a profane meaning. A quick telephone survey confirmed the problem and the bank dropped the word.
- Molly: During a remote session with a Meta Quest headset, a participant accidentally cast their screen and had explicit content visible. Multiple remote sessions have also involved unaware participants walking through the background without clothes.
Recommended resources
- Molly: QRCA publications, webinars, and recorded conference talks, especially for those newer to qualitative research.
- Joe: Sailing books.
AI in the Future of Research
Joe expressed skepticism that AI can reach meaningful conclusions on complex research questions because it lacks understanding of the client's full business context, constraints, and situation. He sees human synthesis as irreplaceable for anything beyond the simplest questions.
Molly described the current AI moment as somewhat of a bubble but acknowledged its practical utility. At Google she uses Notebook LM to run initial analysis across session notes and generate early findings, which she shares with the team with explicit caveats that a full analysis will follow. She also sees AI as a way to make research more accessible for clients with tight budgets, citing nonprofit engagements as an example.
- Agency-side work often restricts AI tool use due to data confidentiality requirements.
- Molly does not want AI to cheapen the value of human labor in research.
- She expects that over-reliance on AI in research may be a temporary trend, with the industry eventually recalibrating toward human judgment.
- AI can democratize access to research for organizations that could not otherwise afford rigorous studies.
Transcript
Read the full transcript
[Music] Hello. >> Awesome. Uh let me jump into it for today. Um so today is the August 28th uh virtual conference. These many conferences that we are doing. Um I'm Brejuski. So I am the VP of research and growth here at Accelerant Research and we're joined by uh a great family. So both Joe and Molly. Uh we'll let them do their introductions. Um but before we do that uh some ground rules for for these conferences right uh we are at the mercy of tech. Uh so you know bear with us as things you know tend to go go arise at times. Uh please be respectful you know put some questions comments in the chat. Um but do be respectful for for other people attending uh as well as our our host today. Uh and then finally use this as networking engaging interact with each other. Um network with each other is a great way to meet other researchers. uh in in the field. But um and then finally these are all recorded so they will be on our website under the resources article uh and then going to conferences. But um without further ado, I will pass it off uh to both Joe and Molly. Um this is a little bit about their background, but you guys can really get into what you'll be talking here today. I'll stop sharing my screen so we can focus more on YouTube than you know having to to walk through any slide deck or anything like that. So, um, I'll turn over to you both. Feel free to to take it as you please. >> Awesome. Well, I will start. Um, so I am Molly. Um, I am currently a rapid researcher at Google uh, but have been before at Answer Lab as well as smaller uh, kind of nonprofit agency here in Colorado. Um, I'll turn it over to my dad before we dive into a little bit more about our backgrounds. >> Well, thanks. I um I'll start by saying I retired from research 10 years ago and have tried my best not to think about it ever since. Um but the scars are healing and uh I'm happy to talk about it again. >> Awesome. So uh a little bit about my own career path uh because our paths are very different. um you know like every parent's greatest dream uh I studied anthropology in college um so you know very high earning very good career prospects right um and then after getting my mers in anthropology at the University of Denver I thought I would go into museums uh I had this real passion for taking information and making it digestible for people so initially that was in this context of museum text panels and and educational programming. Um, but the job markets and you know opportunities just they happen this way. Um, and so through my dad's connections, uh, I actually was able to initially get introduced to Jim Bryson and Isaac Rogers at 2020 Research. Um, now is part of Schlesinger. I think they rebranded to Sego. Um, and since then have been uh at a little itty bitty company here in Denver called Corona Insights, not related to the virus or the beer. Uh, but that was a fun time to work there. Um, then I went on to work at Answer Lab for a little over two and a half years. Um, and then currently, uh, working on a rapid research team at Google. Pops, you want to tell your path? >> Sure. Uh I will note that Molly has always been fascinated by human beings. All her life even as a baby she gravitated towards uh books about people around the world. And so I should not be surprised where she's ended up. Um so my background in research um goes way back. I did my first market research project for this little bank that I worked for in 1979. I had a four-page report for the board which I uh prepared with colored pencils. Um so my uh background started in financial services actually was always in financial services as a as a regulator and a banker. Um, I found I studied finance and management. Um, but found marketing to be more challenging and intriguing and interesting. Um, I had the opportunity uh to go to work with a friend uh which ultimately led to a spin-off, a little company that did market research specializing in banks. I did that for um seven years. And then uh um we might talk later about the challenges and benefits of small firms versus big firms, but I decided to join a big firm and I went to work for a company that is now uh GFK, which I think might be part of an even larger company uh where I ran the uh banking practice and then uh jumped over to TNS which now is part of Neielen um also running the uh banking practice. It it was a great time and we I did some good work and had met some had some wonderful clients. So uh it was a rewarding career. >> Great. So, I mean, obviously we have these conversations all the time, but what do you think or what have you kind of noticed in terms of what's what's changed since you've started your career Dad? >> Well, everything's changed. Um, so I got into research really about 1990, notwithstanding the little project I did for the little bank. um after having worked as a banker and a consultant to banks. Um it was 90 or 91 and at that point the industry was you know mostly small companies and we were um selling our services to um the the senior execs of our client uh companies. Um and we were often the gurus because the these companies our client companies did not have in-house research firms or research staff. Uh so the buying process and the nature of the relationship was a a lot different. So that's one thing. We can talk about lots of things. >> Yeah. I mean I think that's interesting. Um, I I assume that our guests here are kind of all over the board in terms of what size companies from our independent uh consultants all the way to the big the big guys. Um, you know, something that I have noticed over the years is there is this kind of conglomeration or monopolization of research companies and we're kind of you see especially some of the big guys buying up smaller firms and kind of consolidating the offerings. Um, so I wonder like those engagements that you had with clients when it's a smaller more personal experience. How did that differ from when you were at TNS or GFK in terms of the relationships that you had with your clients? >> Well, um, honestly, I look back and when I had this little bitty company, I mean, I'm talking like three employees, right? Um, I think I did some of the best research I ever did uh at that time. And we could often be uh more resourceful, more creative, faster, cheaper um than the the big guys and there weren't many big guys out there. Um but ultimately I gave it up because um research uh client companies were starting to uh have uh in-house researchers and um selling services to the seauite is much easier from a big well-known company than it is from a a small company. So that's, you know, just a pros and cons, I guess. >> Yeah. Yeah. Um, I mean, another thing that that makes me think about is kind of it's not just about the interaction, but it's what we deliver our clients, right? Uh so since I've been in the research world which started in 2016 so going on nine years um it's always slide decks you know sometimes you get to do a document um video clips maybe if you're lucky uh but it's it's very kind of graphic heavy you want to tell a story you want it to be visually appealing. What were your reports like? Has that shifted at all? >> Oh my gosh, it's changed so much. We used to write these uh long narrative reports that had a handful of graphics to make key points. Um and um you know it's very heavy in text and analysis and it started you know with uh methodology and goals and objectives and then we present the findings and then you know a couple of pages about um uh conclusions right and yeah uh somewhere in the early years of my career somebody invented PowerPoint and all of a sudden the whole world changed um But um you know the essence of the the job at hand is the same. >> Ah I remember you sharing like a VHS report that you had delivered and it was like a focus group like video and that is what you and had you know the text overlay and just very different. I see some questions coming in. So Brett, >> I was going to ask I mean if you you're fine to field them, but yeah, if you want to ask your dad as well. Um kind of you know Joe, you've you've done a lot in agency side. Uh was there ever a want or desire to kind of flip into an in-house role as those started to materialize >> for me? No. Um um one of the things that I liked about um and the reason I left the the big bank and went to my own firm or to a small firm which became my own firm um was to get out of the corporate politics and uh you know of a big organization and I um and to this Hey, I still uh I I have no appetite for that kind of big corporate environment. But that's not to say that there's not great people doing that kind of stuff who know their stuff and, you know, are trying to do their best for their um for their clients. Uh it's just not for me. I'm I I mean, you say like big corporate politics, but like doesn't that exist at these big companies? Because I can relate. I'm not good at playing office politics. U gotten me in trouble. So, >> you mean uh uh at a big research firm? >> Yes. >> Yeah. They tend to be siloed though in broken down into into practice areas, right? So it might be u organized by uh industry uh which is my focus or it could be um broken down by you know discipline customer experience marketing communications product development whatever by methodology and so um there certainly is and you know as I as I've gotten old I've you know developed a few skills um you still have uh bosses you always have bosses. Um, and you know, that's just that's just the real world. >> But a a market research firm, even a very big market research firm, doesn't I think have the political um dynamics that some of the biggest uh buyers of research out there have. >> Gotcha. Gotcha. I mean, also methodology. Uh, I remember being like very little and doing a phone survey for one of your clients and I think I got a $5 bill maybe in the mail or maybe you just gave it to us. I don't know. Um, I think we can all relate to to issues with recruiting and and you know, especially when you're dealing with kids. >> Um, I mean, methodologically, how did have you seen things change throughout your career? Oh, it's massive. U Yeah, you probably got a $5 bill for filling out a mail survey because that was like a uh Yeah, we used to do mail surveys. Um and phone phone was the dominant methodology for quantitative surveys. Um and this, you know, we didn't really even have an internet or it was in its infancy. So, uh, methodology has changed a whole lot and um sampling, uh, is much different than it used to be. Uh, and I can't even talk about that intelligently because I'm sure it's changed so much in the last 10 years that I don't even I don't even know. >> Yeah. >> Yeah. I mean, I think there's laws and stuff about calling people now. Like, it's definitely harder. Um, you know, it's uh it's it's interesting to I think some background that maybe we haven't mentioned is uh you did quantitative research mainly. Um, and I really specialize in qualitative, right? Um, and so, uh, even just the way we approach research, you know, I'll tell you about like projects that I'm working on or something. uh we kind of approach things different in terms of research questions and uh you know methodology obviously. Uh have you always seen this split between quant and quall or did it used to be more kind of mixed methods? >> Yeah, I mean um I was always uh agnostic as to methodology, right? So sometimes we did call um which could be you know focus groups or one-on-one interviews or we could do quantitative surveys. Um I think nowadays and I you might comment on this. It seems like nowadays buyers are looking for a methodology sometimes. Back in the day they came to us and they had a problem and didn't have any preconceived idea about how to solve it. And so I think back in the day, as they say, um we had more freedom to design studies that uh served the objective more more effectively. That's my take. Now, I don't know if it's correct or not. >> I I think it probably depends. I'd be very interested to get the audience reaction, but you know, as someone who's recently been on the job market last year after a reduction in force, you know, when you you're competing for these limited roles, I think there is some increased specialization, there's always somebody that's applying for the same job that specializes in that industry or that methodology in particular. And so I think that specialization of researchers also kind of goes to the point about in-house researcher like being a thing now. Um you know that didn't used to be the case. >> We we had a question about that from YouTube. Um you know both of your thoughts about uh small research firms like current small research firms who are now run by individuals from larger firms or in-house. um you know what are your thoughts about you know kind of the the agency takes on the flavor of the person in charge right um you know what have you seen Joe or or Molly in your experience where you know you're you're at a a smaller firm and there's someone from a you know GFK or there's someone from a you know a Apple or Google that comes in and and is oversight of those larger firms good thing bad thing What's your opinion? What have you seen as the benefit or drawbacks of the variation in size? >> Oh, you want to take this first, Dad? >> I'm not sure I get the question. You talking about the uh a small firm competing against a bigger firm? >> Well, I think we can take it in that lens, right? You know, what is the value prop of those two? But then also in terms of more of a office politics I think this is related to of you know you've done your time at GFK and and then you've also had your own agency right uh >> and obviously within those two spectrums the larger firm you know let's say you you were to move over to a smaller firm and it's run by someone that came from a major corporation um you know how do you deal with that kind of bureaucracy those office politics how do you navigate that Um and I guess the the question is really you know what are the opportunities of small research firms who are now run by individuals from larger research firms. >> I mean I I think I kind of have a little bit of this experience. Uh I was there kind of through a transition in leadership at answer lab. Um really saw our staff balloon and then reduce. Um so when I think you have these the big people come in I find it to be less flexible. Um you know when I was at a small company uh there were 10 of us at Corona Insights. So and I think they've grown a little now but it they I was very much encouraged like go after work that you want and like do what you want to do and have that autonomy. And I think when you get kind of that more corporate side of things that that flexibility and autonomy for the researcher can not go away but diminish some. Um I don't know. Dad, what do you think? Well um I mean I think um if you're if you're in a big firm and you're accustomed to selling um your unique selling proposition, right? you're we have a uh unique approach to solving this problem, a unique methodology or um um and so you're selling to a research staff. Whereas if you're with a smaller company, at least my experience, we were often selling to um managers marketers executives and uh we talked about the business differently. So we would What business are you in? We're in the business of helping you grow your business versus we're in the business of uh doing product development research or of doing basian analysis or or whatever. Um, so I don't know if that gets to the politics at all, but um, I mean I' I've never been in a situation where I'm in a small company and um, it's run by, um, somebody who's previously been in a big company because like many industries, research has consolidated over the years, right? You've got smaller firms that keep getting acquired and become bigger firms. Um, and that's that's been the the pattern. I guess another tangent and a followup of that, you know, uh whether it was advice for Molly as she was getting, you know, started with her career or there's many, you know, people trying to get into research just because of how lucrative some of these jobs can be, right? In-house jobs can pay six figures from the start, you know? >> Yeah. >> What would your advice be or what was your advice to to your own daughter or to young researchers starting out of where would you want to start, right? Is it is it in-house? Is it agency? Is it small, medium, large? What what what would you say and why? >> Well, I think there's virtue in having both experiences just because of the perspective. Um I have never really worked as an in-house researcher. I was the marketer who bought research. Um but of course I've interacted with many in-house researchers. I think it's great. uh experience um and I think they get often get uh exposed to the business issues that in today's world uh researchers in big companies uh agencies don't really get. So an issue might be how at if you're an in-house researcher, you might be part of a team that's that's focused on how do we improve the customer experience in this area? Uh whereas if you're in a um agency, the question is more like what's the best way to uh ask customers uh these questions? And so it's, you know, it's a it's a different focus. >> Yeah. To add to that, I think I mean obviously I started in on the agency side and so kind of am in that that space to a degree. Um I really liked it at the beginning of my career to be exposed to different topics, different industries, different methodologies. Whereas in my current role, you know, it's fun to see kind of iterative research build on one another and kind of see that that outcome more being inhouse. Uh but you're not as exposed to much. So I think I would say for young researchers and and even people that I've mentored going into the industry um is expose yourself to everything. If you if you don't know, if you're not like passionate about financial services in particular or something, um going on to the agency side really allows you to explore uh the breadth of the industry as opposed to being more kind of pigeonholed to a specific product or or team. >> Certainly working on the agency side teaches the teaches you the real world, right? So, um I mean I I am sure many of your colleagues have had this experience where somebody comes to you and says, "Here's what I want to do, and it should only take two weeks. I mean, how long can it take?" Well, they've got no clue, right? >> I think we can all relate to that. Um, you know, I do think on the agency side, I've definitely been in the position of like you get a brief or something from the client and you're like, "This is bad research design." >> Yeah. So I think that is a pro of being inhouse is is having the ability to kind of put that expertise you know uh especially in the qualitative world we see a lot of kind of diminishing of our value and the rigor of qualitative research and and how valid it is. Um and so really being able to not just moderate studies or or whatever, but design it and really give that special um know knowledge I think is really uh beneficial especially once you've kind of developed your career and you got to know who you are as a as a researcher, >> right? >> Um I see other questions coming in. Um, one thing that, uh, we kind of touched on earlier as well that I thought was was interesting was, uh, sampling. Uh, you know, you mentioned like way at the very beginning that like sampling is totally changed. Uh, so how like, okay, you have a client, they need a survey. What how did you approach sampling back in the day? >> Oh, well, it was RDD. I don't even know if that term is still around. Random digit dialing. Um to do a telephone survey. Oftentimes we did uh customer surveys which required getting lists from banks. Nowadays there's lots of regulations about that uh which makes it complicated. Um qualitative involved recruiting which is difficult and expensive. Um and I guess still is it must be right. I I think so. I think I mean I've definitely um at Google there's been discussions uh about synthetic users. Uh so when we think about AI and kind of the future of things um I I think it's I think we're kind of on a precipice and we don't really know what the future looks like um in terms of sampling. But one thing that I've noticed is like hyper focusing, hyper like microargeting, you know, like nowadays there's so much data everywhere uh that you can find, you know, that person that definitely >> uh shops only three times a month, that one store, you know, as opposed to kind of more just broad customers, >> right? Yeah. We're we're getting a lot from uh you know relationship of agency and and you know into client relationships right so a couple different questions there of one is specific to you know Joe with your experience uh how have you kind of helped solidify or found your clients into customer or user problems versus uh you know being specific to solution orientation, right? So, what kind of tactics, what kind of techniques have you been able to utilize that really solidified, like you said, the agency approach is they came to me with problems and I had to kind of get them back to it was a user problem versus, hey, we need to sell more shoes or we need we need this, right? Um, how do you how did you get your clients to uh go back into that that customer centric problem? What agency client collaboration tools or techniques did you use to do that? >> Yeah, you when you say user, you're talking about the the in-house researcher perhaps. >> I'm talking about So, if a a client came to you, you know, let's say it's Nike or or whoever Yeah. >> comes to you and say, "Hey, we want to sell more shoes. We're not selling specific brand. >> Joe, help us." Right. >> Yeah. um how would you utilize, you know, your skills, your technique, your experience to, you know, collaborate with those clients and what did that look like? What were some of the things you do to see success with that? >> Yeah, I mean, you just have to have a good relationship and a good conversation about um you know, what what do you need to know and what are you going to do if you know that, right? So they'll say, "Well, let's ask customers or uh prospective buyers this." And uh you know, sometimes you have to challenge them a little bit. So So let's say we learn that 73% think that. What does that mean to you? What are you going to do with that? Um because it's easy for people to think clients have a tendency to know what the questions are, right? That need to be on the survey. Um, and yet they don't know what they're going to do with the answers when they get them. So, this is the sort of collaboration and uh challenge that you need to have. What do you need to know to decide to do A versus what do you need to know to decide to do B? Um, what's going to what's going to what's going to drive the decision that you need to make? I I think that's an interesting output in terms of like you kind of you said already there's a lot of specialty uh in research now right of why this methodology versus what you're sounding like you reverse engineer of >> what's going to be successful how do you want to see it and how can you make the decision >> and then going backwards to how do I get you there right versus you know at times there's agencies that are out there that say hey I have this program six weeks you got to buy it and you know you just got to buy Right. And what's the output doesn't really matter to them. They just want to sell you on this program. Right. >> Right. Right. >> Um other aspect, there's a conversation or a question over Zoom uh that dealt with your relationship with clients. Right. Obviously, you have some clients that you're trying to invest in and uh create long-term relationships with and then there's some other clients that you know you might have strived to just keep as a short-term relationship. Why might that be? How did you decide on that? and kind of what was your approach uh to differentiate you know those different kinds of clients? Well, um, you know, running a a a practice area like mine, which was banking, um, inevitably a a focus was on business development, and oftent time that involved dealing with the the buyer, which was was oftentimes a uh in-house researcher. Um, so we had to, you know, me and my staff needed to be able to talk at two levels. We needed to be able to talk to the researchers about methodologies and sampling and timelines and all this stuff. Um, but we also needed to be able to communicate with senior management about growing the business. And oftentimes when we could, we would try to interact with those people. um directly establish relationships, but oftent times that was just politically not viable um to have a relationship with a senior manager who was the boss of a researcher. You just sometimes that's just too sticky. And so we would our uh marketing communications which would involve um uh email blasts and uh social media to a less extent. Um would sort of target those senior managers and would focus on business challenges like um retaining customers or getting uh repurchase or broadening relationships. Meanwhile, our uh one-on-one conversations with researchers um would focus on, you know, this methodology versus that methodology. So, we had to be able to do both. >> I think I mean, at Corona, we we talked about that a lot of of that consultant role. Um and the people that I worked with at Corona were were academics. I think I was one of the only people that didn't have a PhD. um and and they were very uncomfortable in that space. Do you feel like you saw like your employees like I mean thinking about like Kelly um and Julie like did you did you feel like as the owner of a business or like the boss that you needed to help build those skills outside of just research skills? Well, so we were focused on banking. So we talked a lot internally about the industry and trends and you know because we did want to talk at two levels, right? Both at the business challenge level and at the research methodology level. I mean you have somebody like Kelly who's brilliant, absolutely unbelievable. Um and you know it was not hard for her to grasp these things, right? And the advantage of having a small company is you can get them exposed to uh you know everything. Um which is which is nice. So yeah >> go on >> bit of a change of a topic right networking. Um Molly you mentor we talk what bi-weekly if not monthly right you you mentor a bunch of researchers. Um, so would love your take on this in in terms of what you help with, but also Joe as a hiring manager, as someone that led major corporations in in terms of GFK and whatnot, but also your own agency. Um, you know, someone that is going to transition into research, right? Uh, what are you looking for in a portfolio? What are you looking for or or what advice do you have or strategies you might have in, you know, developing certain skills? uh or you know where where should they focus in, right? Um obviously it's uh do it all and and learn all these methodologies and network and and be able to see the macro and the micro and um but if you had to give someone some advice on some areas in order to strengthen up to really stick out, what would those things be based off of Molly, both your mentor experience and then Joe, your your uh management experience? >> Um so this might be an unpopular opinion. I don't know. Uh but what I would say is I have always like I go back and talk to students in anthropology departments you know about career options and things like that I always really emphasize the importance of like staying you um I mean when I was on the job market last year the amount of like LinkedIn posts and services of like let us AI generate your your resume and your portfolio that's not authentic and I think especially in a qualitative world like that authenticity is is is key. Um, daddy might disagree with that. Uh, but I I think for me it's know who you are because that is going to then make sure that you're doing the research that you want to do and like you're kind of carrying those morals and values throughout your work as opposed to being in a a role where maybe you feel not hesitant or but like persuaded to p to give certain outcomes like research findings that are like you know maybe you're fitting the data to the out to the hypothesis as opposed to um this delivering pure insights. But as a hiring manager, Dad, you tell me. Well, I don't know. I mean, I think yeah, we have to understand ourselves and we have to be honest. Um, I think, you know, certainly the growth of in-house research staff is is a real thing. And I think one of the there's lots of good re lots of good things about that, but one of the downsides of that I I'm afraid, is that um the researchers are now farther away from the decision makers than they once were. Um, and sometimes the end user of the research, the decision maker has an idea about what it's going to what they're going to find in the research and that unfortunately sometimes influences what the in-house researchers asking for. Um, so there's trying to trying to make the boss happy by uh delivering a research result that the boss likes. I I think a key theme that you know Molly obviously talking with you over the past year plus um and Joe from what I've heard from from your conversation is it's research isn't just about research it's about relationships right >> uh and you know it's we can create these really rigorous research studies and programs uh but if the framing is not right if the relationship isn't there then you know it goes into the ether and it might not see the light of Hey. Uh, but I think you know the success you've had with your career, Joe, and and Molly, you know, you've jumped around and you've you have a really good pedigree and what you you offer and skill set uh beyond many. Uh, is the the relationships you guys have had. And I think Molly, one of our first conversations that you talked about uh with me was, hey, I was at this place and what I wanted from there was to extract as many network and relationships that I could get um because I know people are going to go other places, right? uh someone's going to go to Amazon, someone's going to go to Meta or Facebook or Walmart or somewhere or even just another agency. Um and the relationship really helps you establish, you know, how you can get your foot in the door if you are making that pivot, right? Not so much about your research expertise. >> Yeah. I um there's an organization called Wire as well, women in research. Um, and they have a mentoring program that I I was the mentee in this case and was paired up uh with a amazing researcher, Susan Suite, uh, who was out here in Colorado as well. Um, and to have that mentor outside of my job, you know, like she was worried about my career developing, not about the company I was at necessarily. Um, and she exposed me to a lot of resources. of of kind of learning, networking. Um, QRCA is an amazing organization. Um, she encouraged me to apply for the Young Professionals Grant to go to a conference one year and I was awarded that and went um there. And I mean, you just meet so many people. And to your point, we're all moving around. We're all, you know, we're kind of in this shakeup time right now. And and so yeah, maybe you are not best friends with somebody, but like feeling confident enough to I think reach out on LinkedIn or or whatever platform and saying like, "Hey, I noticed that you were also at this company or I noticed that we had this mutual connection." I think it's it's kind of awkward especially for us younger people but uh being confident enough to reach out and make those connections is really going to help give you that confidence in interviews in your client interactions um and all of that. So I think that's also really important. Uh find those resources whether that's professional orgs or books or just plain old talking to people. >> Yeah. Uh, another question uh an offshoot of kind of the conversation we were having earlier but uh yeah Molly we have talked about this countless times but uh this is coming from from YouTube you know at times you're forced the question goes at times you're forced into uh using the client's poorly designed research plan um you know something that we've we have had over over the course of years but uh you know how can you advise the client client to you know change their approach. What tactics might you have of obviously you know you both come with so much research rigor. Um how have you tackled this in the past where you know your client or uh colleague you know stakeholder might be wrong. Um what would you do? >> Dad I'll let you take this one first because I feel like we probably handled this very differently. Well, um sometimes I mean I will say sometimes you have to be a little forceful. Um but there's a balance. Um um you have to think of it as it's an opportunity to um educate your client. Um but I think with you know properly phrased uh challenges you can get there almost almost every time um by asking you know the the hypotheticals. Let's say we learn this what what you what are you going to do with it? Um, and then sometimes it'll dawn on them well gee it kind of that's not going to really drive our decision after all, is it? Um, I mean, it's a dance and some people are better at it than others, but u and sometimes sometimes we sometimes we did research that was just not the research we would have wanted to do, you know. Um, that's just the real world. >> Yeah. I feel like an asking clients what the the outcome or what they're going to do with the research is definitely key because sometimes they don't know and sometimes we just see research that's checking a box, right? >> Um you know it's not really purposeful. It's just someone has told them that they need to do this research. Um, in in my situation, I've encountered especially like in nonprofits, you have a low research literacy, right? So, they've been told this is what they should do. Maybe they have a grant that they've written uh that they wrote out that they were going to do a quantitative survey in a tiny mountain town in Colorado with a population of 800. like no, you can't really get a statistically significant sample there. Um, so I think that goes back to to my like passion for making information digestible and being able to communicate it to people that that maybe don't have as much experience in research and being like, hey, this is not me trying to just like come down on you and like I don't not porative but lecture. Um, it's like let's collaborate and like I'm really excited about what you want to do. Uh, here's how I would approach it differently and having it more conversationally as opposed to that just client um consultant relationship. >> Yeah, >> I love your background, Molly. You know, you talked about, you know, not having a PhD, right? And I think there's there's that whole debate or the war on PhD now, right, of uh those that do have it versus when we got started in the industry. Uh it was you were looked down on, right, if you didn't have an advanced degree like that. Um you know, I think the the humbleness that you approach so many of your conversations with clients or colleagues with um and you truly are doing research on your research, right? How do I frame this? How do I show this off? How am I communicating it? Um you know, as a researcher, right? you're constantly asking how you can improve what you've done. Um, but always something I've admired from from the work that you've done in the past. >> Well, I think it goes back to like being in elementary school and your friends are like, "Oo, what do your parents do?" I could not explain what my dad did. I I think at one point I started saying marketing research because like people understood what marketing was. Um, and now I'm in that position. My friends have no idea what I do. Um, you know, I try to explain it and they have no idea. And so even just kind of using those informal conversations as a way to learn how to communicate with people, I think, can be super helpful because you don't want to people don't like to feel dumb, right? People don't want to feel like they don't know something. So, it's all about like, hey, like I want to share this with you as opposed to I want to teach you if that makes sense. >> Can I tell a story about Molly? >> Please do. Yeah. >> So, she brought a piece of paper in and said, "Here's my homework assignment. What do you think?" Then it was a drawing of a woman with a broom sweeping the floor. And I said, 'Well, what was your assignment? She said, "To draw a picture of what you're going to be when you grow up." And I drew a housekeeper because I didn't know how to draw a scientist. >> So, there you go. >> Kindergarten. Um, and I'm a very bad house cleaner. So, sounds like we uh ended up where we should be. Um I I'm seeing this comment that came in about the pressures of in-house and delivering specific outcomes from research. Um I would say okay a really great example of this is in a former role on agency side. Uh, I would get the hardest clients um and they would be like, "Molly, we wanted to work with you because you're going to write a mean report and they just meant that like I'm going to write what the findings are and I'm not going to like sugarcoat it." Um, and that was like kind of horrifying to hear, but also like kind of boost your ego a little bit as well. like um so now that I am more in-house I have found the tension and I would be very interested if other people is the tension between like designers and like product managers versus the researchers. I would say when I am dealing with other researchers there is no pressure I feel about like delivering specific findings. I would say sometimes when I'm working on projects with there's not another researcher on it and it's just those more you know technical roles that is where I do feel a little bit of that pressure and again I think it's the the sandwich method of like compliment fix this compliment uh to kind of soften the blows and again making it more collaborative like you're not doing terrible but hey um you can be even better. Um, I've also noticed a shift of like we don't really say challenges anymore, we say opportunities. Um, so I think that we're seeing that softening. Um, and maybe maybe that is part of kind of in a reaction to that those pressures. >> I I think right now with the political climate, with the economic climate, um, there's a lot of eyes, there's a lot of scapegoating, right? uh and you know we have we have felt it in our industry and research across the board with you know I think over the past two three years there's been you know hundreds of thousands of jobs that have been eliminated from these major tech corporations right um you know I think the conversation that we've had today uh has always been one with lead with empathy two it's ask the question of you know what are you going to be using this for and you know it might be a lack of translation where uh your your understanding of what your client is asking you to do or what your in-house colleagues are asking you to do might not actually result in what you think it is and asking them how is this going to be utilized what is the framework that can help you position it to uh be more of of um something that's in line with research rigor and ethics and morals and and however you want to lead as a researcher versus having to fall into a system of the weaponization of research at times, right? >> Yeah, definitely. And I also think I mean maybe this goes back a little too far, but um keep research to that like just checking a box, you know. Um even in those scenarios, you still have to lead with empathy. you still like these these kind of values still have to be there even if you sometimes maybe feel like your work is not valued or you know you're you're giving a client very clear-cut insights and they're doing the exact opposite. Um there's this great TED talk about this ethnographer u who in like the early 2000s was hired by Noia uh to do ethnographic fieldwork with young I think it was in Korea um about phones and like smartphones were just getting started. I think like the iPhone had been released and so she does all of this this ethnographic work and she's like yes people want smartphones. Noia did not listen to that and they listen to big data without that human touch uh to it. And we've all seen kind of how that business has changed, right? Everyone had a Nokia phone back in the day and now it's it's very rare. So I think advocating even when you you feel like yeah your work is not valued or it's not like being used in the way that you would is still an important thing to hold on to because I mean she was right. You know as researchers we have to know that we're right. Um but unfortunately we can't always be those decision makers and that's part of it. Uh that's part of the role of being a researcher. And uh it's humbling. It's humbling. Uh especially for us that always think we have the answers. >> I'm going to shift to a lightning round, right? We have about seven or so minutes left. So, uh about, you know, let's say 60 seconds per response uh for both of you. Um or even shorter. Uh but we'll start with uh Molly. Most overrated metric. I mean net promoter score uh like why are we doing that still? Nobody is recommending your product to anyone. Okay. Like I I I think that um in the qualitative uh side of things um I think having metrics in general with qualitative research can be can be problematic. >> Chuck. >> Oh gosh. Uh I have a similar opinion about net promoter score. Um I I'm not sure I've got anything to add honestly. >> NPS. All right. Um next aspect uh method you would use or you would wish teams used more. Some kind of methodology that you don't see utilized enough or that you have strong positive opinions of. >> Triads. >> Triads. Why? Um, I think we get a lot of flack on focus groups about group think and all of this stuff, three people, even diads, you know, I think it allows people to kind of rift off of one another. You get a more robust conversation going. Um, but you're limiting like I'm one person versus eight. You know, if if I differ, maybe I'm hesitant to speak up. So, I think it just creates a much more equal environment between the moderator and the participants where everyone is more comfortable. You're building a relationship, you're building that rapport, and you're getting better insights from it. >> Yeah, I I would say one-on-one in-person interviews, which are expensive as all get out. Um, and recruiting is key, but man, the richness of the learning is just amazing. I mean, some of the studies I did using one-on-one interviews, and of course, the the findings are difficult to sum up in a in 90 seconds like some senior managers want, but um, man, you can sure learn a lot. >> Uh, we'll start with Joe for this one. Uh, craziest research story. Um, well, I don't know if I could say this on family television. Uh I had a client come to uh when I had my own firm and want to uh ask what a word meant because they wanted to use it in their uh advertising and um and honestly I didn't know all the meanings of this word but we had to do a little quick uh I think we went telephone survey and asked what does this word mean and it has this very uh you know uh profoundly profane meaning that uh the bank decided was not appropriate. >> Uh I feel like mine is also inappropriate. Uh it's two of them I would say. one. Um, I did a project with the MetaQuest headset. Um, and the participant went to cast their screen um, and had porn up. Um, and then several times actually I have had remote sessions where there have been like naked people walking in the background. So, I don't know if this is like a me problem, but it does feel like it's happening a lot. the the over comfortability of uh work from home. Yeah. >> Yes. Yes. >> Two two more questions here. Um a book, paper, or talk you'd recommend every researcher to to watch you have >> I mean us right now. Duh. Um I don't know. I would say qualitative uh any QRCA things that come out. They have a magazine. They do webinars as well. Um, and then they also film or record their conference talks as, you know, and that's also great. So, I think if you're especially maybe not familiar with qualitative or want to learn more, it's a great resource and you don't have to be a member necessarily to access everything. >> I got nothing. >> Been a long time. >> You're you're you rather just focus on sailing now. Yeah. >> Yeah. I'm sailing. Got some good sailing books I'd recommend. >> Uh, last question here. Um, and Joe, we'll start with you, but uh, you know, role of AI in the future of research. Where do you see it going? What do you see it used for? What do you see it where it shouldn't be used for? >> Uh, well, I have very little experience with AI. Um, and you know, guy my age, holy cow, how what do I know? Um, I do think that the one thing we should, as little as I do know, I I don't think AI can reach conclusions unless it's the simplest of questions. Um, because the synthesis of an understanding of the client's situation, challenges, constraints, uh, I don't see how AI is going to ever have that. They're only going to have what they what we input, right? Um, that that'd be my that'd be my uh answer. >> I think we're having an overreaction to AI a little bit. I think it's kind of like this bubble right now. Um, being on an agency, I was often not allowed to use a lot of AI tools because of like data confidentiality. Now, at Google, I got I got all of it. You know, I can use uh a lot of tools. Um, I would say Notebook LM is a great resource. Uh, not just for research, but if you're a student or you got a kid in college or something, it's amazing. You can upload sources, videos, and it'll kind of do um, you can like make a study guide from it or like a briefing doc. Um, I think that's great for getting those initial insights. I mean, I did this yesterday to to my team. I I sent the initial findings for my sessions that I had yesterday and had to say keep in mind this is based on a very cursory analysis um and things may change in the final report uh once a more more thorough analysis has been completed. So um yeah I definitely think it can be a really great tool. Uh I do not want to see it cheapen the value of the human labor uh needed in research. Um, and I think that we're going to see situations where AI is going to replace research. And I I I think that's going to be a short term thing. And by short term, I mean like it could be 10 years, but like I think we're going to come back around and um not be so reliant on it anymore. But it does make it more like it does democratize research in some ways, making it more accessible to people because it can be cheaper. I mean, I'm thinking about nonprofit clients I've had where, you know, you got $20,000 and you've got to make a lot work. And in that that scenario is a really great strategy to help a client get what they need um in a budget and time that that they have. >> We are at the hour. Um so, any final last thoughts? Um, I know this was really meant to and love both of your perspectives, but um, Molly, you know, obviously growing up and seeing your dad, uh, have his firms and his agencies and move around a bit. Um, and then your your perspective on research and and likewise Joe, um, you know, seeing Molly mature as a researcher and the different positions she's now had. Uh, any final thoughts of, you know, how does how has that been uh, over the the course of the years? the pressure >> the pressure to like be good, you know, like Jesus, what if like I get introduced to somebody my dad worked with and they're like, you're awful. You like don't know what you're doing. I think especially earlier in my career, that was a fear. >> That would never happen. >> Let's go sailing. >> Yeah, I I appreciate the time. Um, thanks for everyone that joined. This has been probably one of the most engaging uh, conferences we have had to date. So, both of you guys, I really appreciate the conversation. Uh, we do have another one that is going to be scheduled, so be on the lookout for our socials. Um, but Joe and Molly, thanks for the time. Um, and, you know, hope you guys have a great Labor Day weekend. >> Yeah. Thanks so much. >> Thank you everyone. My