Dr. Ari Zilmanow, a former police detective turned insights professional, makes the case for prioritizing qualitative interview quality over data quantity. Using cases from his detective career and the near-failure of Febreze, he argues that a small number of well-designed interviews can surface insights that large datasets and algorithmic tools routinely miss. The talk covers research planning, interview design, fieldwork technique, and how to deliver findings as actionable counsel rather than raw data.
Key Takeaways
- Focus on quality, not quantity. A handful of well-recruited, well-conducted interviews can reach conclusions faster and more reliably than large datasets, and at lower cost.
- Recruit deliberately. Controlling for sample variance on the front end, by distinguishing customers, potential customers, and non-customers, is what makes small-sample research credible.
- Avoid asking 'why.' Use probes like 'tell me more,' 'how does that make you feel,' and 'what made you decide to do X' to surface emotional drivers that respondents cannot articulate directly.
- Stop at saturation. In applied settings, themes typically repeat by 5 to 12 interviews. If you are not reaching saturation by 12, something is wrong with your recruitment or your guide.
- Deliver counsel, not data. Researchers should provide a clear point of view and a specific recommendation, not a menu of options. Most business decisions are reversible, so commit to a direction.
- Record and transcribe every interview. Reviewing transcripts and coding for themes is the only way to capture detail missed in the moment and to share evidence with stakeholders.
Questions & Answers
- Will the slide deck be available after the presentation?
- Yes. Zilmanow confirmed he would provide a PDF version to be shared, and the conference organizer noted it would be posted on the conference website along with the archived video.
- Can you finish the story from your opening case?
- Zilmanow confirmed the child was found to have been abused and neglected. The child had some developmental delay but not enough to require diapers at age nine. He ultimately charged the mother with homicide. The grandmother was approximately 50 percent lucid; the mother was fully lucid. The mother was convicted. Zilmanow described it as one of the most difficult cases of his career and expressed gratitude for being able to close it and hold the responsible party accountable.
Session Notes
The Case for Small Data
Zilmanow opened with a child abuse investigation from his time as a detective in St. Louis. He walked through six pieces of physical evidence from the scene, then asked: how many data points did you need before you were convinced? The point was that decision makers often believe they need enormous amounts of data before acting, but a small number of high-quality data points is frequently sufficient.
He reinforced the argument with the Febreze case. Procter and Gamble launched Febreze as an odorless odor eliminator and sales failed immediately. Rather than turning to big data or machine learning, a small number of in-home interviews revealed the core problem: people who live with persistent odors, such as cat owners and smokers, become nose-blind and do not perceive that they have a problem. Febreze was being marketed to people who did not know they needed it. The pivot to adding a fresh scent and repositioning around finishing a cleaning routine drove the product to $200 million in its first successful year and $1 billion annually today.
More data equals more problems. Data rarely helps you find a needle in a haystack. In fact it only creates a bigger haystack.
He also flagged the risk of spurious correlations in large datasets, citing the near-perfect correlation between Maine divorce rates and per capita margarine consumption from 2000 to 2009, and cautioned against over-relying on black-box algorithmic guarantees.
Five Customer Interview Types
- Futurecasting / trend interviews: identifying emerging signals before they become mainstream.
- Competitive intelligence interviews: understanding how customers perceive and choose between alternatives.
- Business originated interviews: refining or optimizing a product, service, or brand already in market. This is where most applied research sits.
- Consumer originated interviews: exploring an unmet need before a product exists, used heavily in new product innovation and early-stage user experience work.
- Customer interrogation: eliciting testimonials and social proof from satisfied customers for marketing use. Zilmanow noted this is the only legitimate use of interrogation in a research context.
The talk focused on business originated and consumer originated interviews, as they represent the majority of applied research work and are foundational to the other types.
Deductive vs. Inductive Reasoning in Interview Research
Business originated problems call for deductive reasoning: you start with a product and a hypothesis, observe customer response, and confirm or refine. Consumer originated problems call for inductive reasoning: you observe a customer segment, look for patterns and unmet needs, develop a hypothesis, and build toward a product. Both approaches require Bayesian agility, the willingness to update your hypothesis as new evidence arrives, rather than researching only to confirm what you already believe.
Building a Research Plan
Zilmanow argued that even a small interview study deserves a formal research plan. Key components:
- Project name and background: what is the business context, what are the knowledge gaps, and what will the business do with the findings? If you cannot answer those three questions, do not start the project.
- Research objectives and key results: frame each objective as something you are trying to accomplish, then ask 'as measured by' to define the key result. Keep objectives specific, measurable, actionable, and relevant.
- Approach and sample: specify who you are talking to, how many interviews, and the timeline. Distinguish customers, potential customers, and non-customers. In B2B contexts, also distinguish decision makers, economic buyers, influencers, recommenders, and end users.
- Screening criteria and quotas: be precise about inclusion criteria and define explicit termination or exclusion criteria to keep the sample clean.
Conducting the Interview
Discussion guide structure
- Use 5 to 7 core questions per session. Aim for 30-minute interviews over video conference rather than longer sessions with fewer participants.
- Open with a 2 to 5 minute warm-up. In video interviews, the person's environment can serve as a natural conversation starter.
- Close the loop at the end. Tell participants what happens next, just as detectives do with witnesses and victims.
Question technique
- Avoid asking 'why.' People are poor at explaining their own motivations. Use indirect probes instead.
- Six why-replacement probes: 'Tell me more,' 'How does that make you feel,' 'What made you decide to do X,' 'Walk me through that,' 'Can you give me an example,' and a reflective restatement to invite correction.
- Focus on 'what' and 'how' questions. 'How did you choose between product X and product Y' is better than 'why did you buy my product.'
- Avoid leading questions and yes/no questions. Leave room for explanation.
- Use silence. A pause that feels long to the interviewer often prompts the interviewee to expand. Suspects in police interviews use silence too, trying to make the detective fill the gap. Do not take the bait.
Field principles
- Record every interview, with explicit consent from the participant. Zilmanow reported only one person ever declined recording across his entire career.
- Separate roles: one person conducts the interview, one takes notes. Do not look down and write while someone is speaking.
- Follow the discussion guide as a roadmap, not a script. Diverge when something valuable surfaces, then return to the guide.
- Ask questions in non-linear order when the conversation calls for it.
- Prioritize revealed preferences over stated preferences. What people do matters more than what they say they intend to do.
- Recency matters. People recall recent events and emotional peaks and valleys most clearly.
When to Stop: Saturation
Saturation occurs when interviews begin to sound largely the same. In applied settings this typically starts between 5 and 7 interviews and is usually reached by 12. If you have not reached saturation after 12 interviews, something is wrong: you may have recruited the wrong people, missed a key variable, or structured questions in a way that scatters responses.
Analysis and Transcription
Zilmanow recommended transcribing all recordings using tools such as Temi, Otter, Rev, or freelancers via Fiverr or Upwork. For analysis, he suggested Delve as a practical, lower-cost qualitative coding tool for applied researchers who do not need enterprise platforms like Atlas.ti. Coding involves going through transcripts to identify themes and patterns, then comparing those patterns across participants.
Delivering Research as Counsel
The final section addressed how to present findings. Zilmanow drew on the Godfather metaphor of the consigliere: the advisor who gives direct counsel, not just information.
We don't deliver data and we don't deliver insights. We deliver counsel.
A research deliverable should include three components:
- Point of view: a clear, direct recommendation. Not a list of options. Researchers who hedge by presenting multiple paths without a recommendation leave the business no better off than before the research.
- Executive summary: a brief narrative supporting the recommendation.
- Reaffirmation of the decision: remind the client that most business decisions are reversible. Amazon's framework of reversible versus irreversible decisions applies here. Advice is given based on the facts and circumstances available at the time, which is where interview research has an advantage over backward-looking data sources.
Transcript
Read the full transcript
off our our next presentation um thanks to everybody who's just joining uh thanks to everybody who's been here um all morning uh our next presenter is going to be dr ari zilmanow and he's going to be giving a really interesting talk i've seen him present in person remember in person conferences that was a thing um and he's just very captivating speaker and i'm i'm excited to hear this next talk so what i'm going to do is just shut my mouth you know i'll i'll let that be the intro and i'll let let ari take it from there um topic of of his discussion is going to be how to interview like a detective um fascinating so again i'm gonna be quiet i'm gonna let r do his thing i'm gonna stop sharing so that he can and are you can take it away great thanks um so i'm gonna share here i need to make sure the first question i'm gonna have is can you see the slide and nothing else just the slide okay i'm working with two monitors for the first time um you know well since covet hit so i'm going to jump right in because i've got a lot to share today um and i want to see if i can get through it in an hour it's going to be tough but i'm going to do it so my father told me every good story begins with once upon a time so i always like to stay once upon a time but um the reality is is that what i want to talk about today is interviewing and the problem is is that interviews get a bad rap um and and a lot of it's about there's a lot of reasons uh for that so i want to start by having everybody go through a paradigm shift so if you didn't know i was a police detective um and when i s in st louis and one of my assignments as a detective was in the crimes against persons unit specifically focused on child abuse so on a warm day in may i was called to the hospital about a suspicious death of a child and so i go to the hospital to find a very very dirty like a like not a thin layer of dirt not just some dirt but like a layer of dirt across a nine-year-old child who had pretty severe injuries um and i can't see everybody's face or i can't ask for a show of hands but if you've ever seen the movie misery the child had ankle injuries consistent with being hobbled so both ankles were broken so my partner and i go to the house to investigate further and here's a picture of the house pretty nondescript you know like many houses it's fine um and so we go into the house and i look into the fridge and um as you can see this picture of the fridge the fridge is dirty and you can see the zest of crackers down in the lower lower left of the screen and you can see the amount of food in the house okay so i've just given you a data point i'm going to give you a second data point i'm like okay well we need to go through the house let's go to the bathroom so i go to the bathroom and i find this this is a uh this is the water they didn't have running water they used this as water i mean you can see that it's pretty dirty um you can't really see the toilet very well but it doesn't function um remember i told you the child's nine so this is the bathtub and the shower so i ask you for a nine-year-old child if you look carefully you'll see those are all diapers so why is a child a nine-year-old child wearing diapers and then finally we go to the kitchen um and this is the the kitchen of the house and you can see the sink is full and there's trash and stuff everywhere so let me ask you this now i've just given you six data points how much more evidence do you need to see before i can convince you that this house is unlivable and this child was abused or neglected i'm betting i'm betting i don't have to give you any more data points and this becomes one of the problems with interviews and convincing stakeholders of things is that they feel they need a ton of data to be able to make a decision but i've just given you six data points so those are kind of at odds so i'm going to give you a few other things because i need this paradigm shift to occur to understand really how and why interviews work the crux of this is to focus on quality not on quantity so this paradigm shift is why interviews can be a very effective form of research for businesses so the other argument for this is that if you compare to that budgets that big brands have for research they can spend millions of dollars dollar for dollars smaller businesses can't win um and and big big businesses are still stuck in an old way of doing things so i always like to tell the story in the business sense because you're like well that's neat sorry but that's like detective work so how does that really work in the business space and so what i would say is consider febreze fabrice started in a laboratory with a scientist who was working he was a smoker and he was working on this formula and one day he um sprayed it on himself went home and his wife said did you quit smoking and he said no why she goes i can't smell any cigarette odor on you and at that point he goes back to the lab he tells febreze that he found this thing that actually kills odors and they immediately like wow this is this is a biggest breakthrough ever we're going to create this and launch this and they they get everything ready the brand teams go out and buy ferraris everybody's really really excited and what happens is they launch and this is what happened according to internal png report febreze is a procter and gamble brand they disappoint they should consider terminating the product sales gold unmet every week um and what they did it wasn't go back and look at big data they didn't look at artificial intelligence and they didn't look at machine learning it was it was and it wasn't huge customer surveys in fact a customer survey wouldn't have even captured the right data what it was is a few interviews they went to a house in arizona and they talked to this woman who had a bunch of cats so as they approached the threshold the researchers approached the threshold of the door one of them literally gags at the odor the scent the odor of cat so they go in they sit on the couch and they talk to her and they say so what do you do about this cat smell she's like that's funny it's usually not a problem they're like well do you smell it now no isn't it wonderful my cats hardly smell at all this data point is what what really pivoted febreze from being an odor killer to being a um an odor killer that is has a fresh scent so remember febreze was odorless when they launched it but the problem is is that people that smoke and people that have cats and people that have houses that smell different than others um they don't smell that they become accustomed to that odor and so they don't notice it so febreze was being sold to people who didn't even realize they had a problem well once procter gamble discovered this from a group of interviews sales exploded 200 million in one year 1 billion a year today they had made a huge marketing error they were trying to sell something to these people that didn't have a problem and so you can learn now how to get in front of this problem today with one of the with a few of the research pathways i'm going to show you but again it starts with focusing on the quality of of the data and not on the quantity another big problem with data is this when you consider big data solutions fancy algorithms machine learning you have no idea what's inside right unless you work for the company and you're actually involved in the development of that but even if you are a mathematical machine learning coding wizard and you're presented with another company's thing they don't show you what's inside this black box it's a secret um and so what could actually be inside the black box is just this so if you ever um and finally a lot of these a big data or machine learning or ai solutions they offer a guarantee and i have to show you a brief video of of what i think of when i think of guarantees why would somebody put a guarantee on a box hmm very interesting go on i'm listening here's the way i see it ted guy puts a fancy guarantee in a box cause he wants you to feel all warm and toasty inside yeah makes a man feel good of course it does why shouldn't it you figure you put that little box under your pillow at night the guarantee fairy might come by and leave a quarter am i right ted what's your point the point is how do you know the fairy isn't a crazy glue sniffer build a model airplane says the little fairy well we're not buying it he sneaks into your house once that's all it takes next thing you know there's money missing off the dresser and your daughter's knocked up i've seen it a hundred times but why do they put a guarantee on the box then cause they know all they sold you was a guaranteed piece of that's all it is isn't it hey if you want me to take a dump in a box and market guaranteed i will i got spare time but for now for your customer's sake for your daughter's sake you might want to think about buying a quality product from me okay i'll buy from you so as you can see guarantees why would some guarantees are only as good as as the uh the guarantee so focus on quality not on quantity finally um interviews are really well suited um for looking examining problems that are more recent so consider the problem of induction i don't want you to be a turkey so this turkey every day he lives outside the farmer comes out gives him water feeds him takes care of him he has open range to walk around the farm everything is great so the turkey has become accustomed to looking at the world and understanding the world from the things that have been happening day after day after day the turkey isn't really looking at the now or what's ahead and this works until you look at the thousand and one days in the life of a thanksgiving turkey so food water up until day a thousand and then the thanksgiving day hits and it's the surprise and that of course is what happens so interviews capture things in real time or near real time so i would urge all of you to take caution when trying to predict things from the fast from the past focus on quality not on quantity also more data is not always the answer in fact it's rarely the answer more data equals more problems data rarely helps you find a needle in a haystack in fact it only creates a bigger haystack and you can find the number of spurious correlations correlations that don't don't really matter to the number of variables increase for example if you look at the divorce rate in maine and per capita consumption of margin they're highly correlated between the years of 2000 and 2009 99 correlated in fact so does that yes we all will agree that correlation does not equate to causation and there's probably no relationship but when you add a lot of new data variables this is these are the things you can find and they're probably unrelated focus on quality not on quantity and finally i would tell you sometimes small data will re will beat big data and reach the right conclusions faster more reliably and lower cost so again focus on quality not on quantity so then i would say to this okay well ari what do we do with this well today i'm going to show you how to interview like a detective thinking like a detective and interviewing this way gets us closer to small data it gets us closer to people and people is really where we live so there are five primary customer interview types right there are interviews that lead to forsything trends there's interviews that help us explore competitive intelligence there's business originated interviews which exist when you already have a product service or offering in in flight and most of the interviews that we in consumer insights and market insights do fall into that space there's consumer originated this is where you don't have a product developed we're developing a brand new product not a new product extension but a brand new product this is a lot of in in the tech space this is a lot where user experience sometimes lives in the early stages but it's really focused on new product innovation and then there's customer interrogation and of course that gives everybody pause because i would tell you there's only one occasion where you interrogate customers so the difference as a detective between interviews and interrogation our interviews are fact-finding interrogations are looking for an affirmation of guilt so i'm affirming that i'm guilty or um and i'm trying to get you to to make a statement uh to that effect so customer interrogation for for researchers really falls in developing case studies developing marketing marketing collateral and developing social proof so these are the times when you know a customer's happy with a product but you're trying to elicit a statement to that so you can use that in marketing that is the only time customer interrogation is works since we have such a limited amount of time today i'm going to focus on two the two areas that that give the biggest the offer the biggest punch those are business originated and consumer originated the reason is pareto right it's the 80 20 rule that this is this is where most of the effort and most of the things start from it's also foundational to all of the other interviews um as i told you there are more interviews that are better suited to refinement so in the marketing space you might be looking at brand awareness positioning and product uh you might be looking at innovation for a product or a product extension so customer development product development uh you might be looking at segmentation um this is also an area where we can help businesses with funnel analysis so in customer experience where you're thinking about the sales funnel awareness consideration preference purchase loyalty and advocacy and this is where customer satisfaction lives as well and then finally you can also offer services because you're interviewing like a detective to engineer sales conversations so you're understanding what prospects need to hear to understand to increase a close rate and then when the consumer originated you could be hired or be tasked with how we want to develop a brand new product we need to understand that so understanding that there's these two areas matters and it's a matter it's it the reason it matters is one falls under deductive reasoning and one falls under inductive reasoning now sherlock holmes always it's always been said that he deduces things well he actually induces things but let's not mince words here let's say that we're going to do both business originated problems exist when a product is developed you're then going to develop a hypothesis which is identifying segments or jobs so you're looking for where does this product fit you're going to test it and then you're either going to have success huge success or you're going to go back and ultimately you'll probably continue to refine that inductive reasoning exists when you're like i've got a customer segment there's an unmet need in that market i need to explore and understand what's happening i'm going to uncover why people like hire so when i say hire i mean right now if you're taking notes you're taking notes with a pencil you're taking notes with a pen you might be taking notes with a tablet or you could be taking notes on your computer each one of those things has been hired by you hired meaning you've chosen it to do a specific job for you over the others that is a job and that that's under jobs to be done theory which was popularized by uh harvard business professor clayton christensen um and it's still widely used in innovation in tech today um and you're gonna do that and then you're gonna develop a product and then that from that product you're going to go back and then it becomes a business originated problem where you're iterating on it so going a step deeper it's a matter of reason deductive reasoning is top down so you're starting with a theory like you have a product we think that i think that this is going to do this we're going to observe confirm inductive is bottom up we're kind of observing the market looking for patterns developing a hypothesis and then having a theory the point with interviews and why they're so effective is that it allows you to move relatively quickly um a conclusion can always seem true at one point but further evidence can change that and all of us as consumer insights professionals and detectives have bayesian agility which is the ability to modify a probability of a hypothesis being true as new evidence is supplied so when we're researching something we shouldn't oh we shouldn't be researching necessarily to prove something unless it's your hypothesis it should then be to prove or disprove it you can have hypotheses but you should be going with an open mind and the ability to be agile on this so again i want to remind you all to focus on quality not on quantity so how do we operationalize this well i wanted to i know that we have a wide group of people um watching this some already have a deep understanding of some of these things but some some may have less and so it always starts with if you fail to plan you should plan to fail okay so you should always have a research plan even if you're just doing interviews and i said just but i don't think of interviews as just so you want to name your project give it a name i want you to always provide some background relative to the problem you're trying to solve and this is going to become really clear why soon when i when i take you through this process but why are you doing this project like is it a business originated problem is it a consumer originated problem like what are the what's the business case what are the gaps in the knowledge and then what's the business going to do with what you learned because if you can't answer those three things you shouldn't be doing the research project to begin with then i want you to think of research objectives and key results or questions so if you notice i've entered something into here that probably hasn't been seen a lot before yes research objectives and questions have but key results haven't key results are things like i want you to think of an objective as i'm trying to accomplish this and a key result as replace that with as measured by so an objective is something you're trying to accomplish but i want to know i want you to think about how will i know if i've accomplished this objective and so that that should be as part of this document i want you to outline those things you want to keep your objectives smart like specific measurable actionable and relevant to the project um there can also be primary and secondary research objectives and then i want you to go through the approach so your approach obviously in this case is uh we're going to conduct five five five interviews with uh this group of people i want you to identify who the sample is and then i want you to kind of figure out what a timeline for your project should be once you get this dialed in and you're interviewing appropriately you can be doing customer interviews for for clients or if you're in house you could be doing rounds of customer interviews in weeks two weeks um now as a detective the single most important thing that we look at is who who is important who is the most important detectives focus on three primary groups of people witnesses victims and suspects as applied researchers we're focusing on three primary groups people customers potential customers and non-customers knowing the difference is key because by doing that you're able to control for variance of your sample on the front end this is why small sample research works it's because you're not you're not throwing things out and hoping to get to capture enough sample that you're seeing some repeatable things and you're getting this bell-shaped curve which by the way doesn't really exist in applied research anyway but that's a conversation for another another day we are controlling for the variables that we think influence things on the front end who is so important and then i always feel like i i need to take a moment to say it's b2c or b2b customers but there's a famous marketer who leads uh digital marketing his name's ryan dice and what he says is b to c or b to b doesn't really matter it's h and he's right it's human to human no matter what whether you're talking to business business to consumer or business to business you're still talking to people but b2b isn't a single person there are decision makers there are economic buyers there's recommenders there's influencers and then there's users i'm going to have to change that word i don't like the word users i like customers better or consumers but either way and then along that whole spectrum there's saboteurs there's people that will sabotage a b2b buying process throughout knowing how that paradigm works or how that works within a business helps you help your clients understand marketing decisions or research decisions because sometimes you might be doing research on decision makers and other times you might be doing research on influencers it matters being very specific matters which brings us to the next point which is um the the screening criteria and quota section so with screening criteria in quotas you want to be very very specific so prior to my role at panasonic um i worked at twitter and we were doing some work on on on developers and as you can see there there were some screening criteria we used uh to decide who we wanted to speak with now this was because we were looking for a broader range if you're if you're looking for very specific people you might say we want this number of people with these criteria and then do that and then as you can see below there are termination criteria or exclusionary criteria those are things that as you screen people you're screening them out and there are others that you're going to either establish quotas for or you're just going to have some sort of descriptive statistics but you want to develop your screener and then um and then we go to the interview so the interview is an interesting part and as a detective um as you'll s you'll see here also i love alice in wonderland and being a detective is basically chasing the white rabbit you're kind of investigating things seeing what pops and doing that and so we like to ask why a lot but we don't as researchers applied researchers want to use the question why people are terrible at telling you why i know i should exercise more ask me why and i'm going to come up with reasons but they're probably not the real reason why so instead of asking why i'm giving you six why prompts to get to the core of that one of them tell me more um ari it's interesting you said you want to exercise more can you tell me more about that um yeah i want to exercise more but i feel like i don't have the time how does that make you feel and do you see what i'm doing there is i'm now getting to the emotion of something if in business in sales and in marketing people do things emotionally and they justify with logic or ration after the fact if if we only made logical and rational decisions nobody would buy bentleys right i mean they're a very expensive car they make you feel a certain way and then you rationalize and justify it after the fact which is fine i think it's great that we do that but we have to recognize that as researchers this is the way people work all right okay just so i have it right ari did i get any of that wrong no no i think you got that right or it gives them a chance to fix it what made you decide to do x that's better than asking why did you do x they're not going to know why but they could tell you the steps and the things and as a researcher it's your job to put that puzzle together and then um i want you to use silence to your advantage don't respond and let the silence prompt people to keep speaking so that's a pause or that's a pause they feel long to me but they're probably very short but when you introduce silence into the equation it encourages people to keep speaking and and and expand on what they're saying as a detective we would do this a lot so what would happen is um and we would do it we would use silence to let people go but suspects would also use silence so or they'll they'll use silence and then they'll they'll use a prompt so you would ask a question and they they would say what and then there'd be silence waiting for you to fill in the gap but as a police officer if i use silence they heard the question i'd say what's your name they'd say what and then i let silence drive them to tell me their name because i knew they heard they were trying to buy some time use silence to your advantage so um i love alice in wonderland as you can see the second second uh quote from this alice asks what do you tell me which uh tell me please which way i ought to go from here that depends a good deal on where you want to go said the cheshire cat i don't much care where alice responds well then it doesn't matter which way you go as long as i get somewhere alice added as an explanation oh you're sure to do that if only you walk long enough the point i'm making with this when we talk about an interview is if you don't have a discussion guide you're going to get somewhere but might not be where you want it to go but it's going to be somewhere discussion guides are important and i like to use the rules of five interview university the group that i use to teach interviewing and interviewing skills i teach that you should have um five to seven interviews with five to seven core questions that's the starting point so you start with five to seven per issue five to seven core questions five to seven interviews with video conference you wanna aim for 30 minute interviews i know as applied researchers there's a history of us using taking an hour or there's a history of us doing other things an hour on video conference can be hard you want to aim for a 30 minute interview i'm going to tell you at panasonic we just did a big round of interviews they were 16 minute b2b interviews we were able to to land those if you can get them and they work for you great but i teach like start start by aiming for 30 minute interviews doing more interviews with shorter time periods than longer interviews or less interviews with longer time periods um and i and some of that also matters if you're trying to go for depth or breath but i want you to be your discussion guide to be focused on an area of interest five disorder seven chord questions be flexible also start with a warm-up and then um a warm-up what i mean by that is you don't want to jump right into an interview um and the cool thing about video conferencing is generally speaking um or interviews with people's houses you can look around you and you can find things of interest and then you can use those as discussion points spend two to five minutes warming up with somebody and it's going to pay off huge throughout the interview because they're it's going to be more conversational and then at the end of an interview you want to close a loop you don't want to just get up and leave you don't want to just stop the interview you want to like let people know what what's going to happen next detectives we did this too witnesses want to know like what's going to happen victims definitely want to know what happened and suspects are curious too they want to know am i going to jail am i what's next here so now i'm going to give you some rules so in in the detective world or in uh in in interviews revealed preferences i notice there's hashtags it's my it's a call back to my days at twitter data validity will be maintained by collecting what people do not people what people will say they do it's the difference between revealed versus stated preferences revealed preferences are what i do stated or was that what i want to do i don't work out but i aspire to recency matters people will have a good recollection of things that have happened more recent than longer ago and that recollection becomes such where they're going to remember really high points and really low points they're probably not going to remember a lot in the middle unless you have hit on a point that in their mind they've made some sort of collec connection to you so there's semantic connections in everybody's brain if i say something it might bring you back to your high school days you might bring remember something in your high school days and then etc etc go non-linear don't always have to you don't have to ask every question in the exact same order so as a detective we would ask there's some standard questions we would ask like i want your name address date of birth but then i would ask other things and then i could always circle back remember i want i told you to be conversational you aren't joe friday we're not we're not interrogating people we want conversation and the interview to feel like they're talking to a friend the more comfortable they feel with you the more they will open up and the more they will share the more guarded they feel the less they will open up to you and the less they will share i want you to give pause short pause is going to feel like a long time to you um interviewers interviewees sometimes have to think about things if you jump right in they're going to stop thinking about it and think about the next question those pauses matter give pause slow down give it time for the record i want you to take notes and record interviews these there's lots of golden interviews and sometimes you'll be asking a question listening for something and you'll miss something else the only way you're going to get that is if you've recorded it you also want to be able to review and share these later now i want to take a minute here to mention note taking in an interview when i say you're taking notes i don't want you right this is why you record i don't want you looking down writing and taking notes you're missing things you're missing body language you're missing other things by doing that that is why we record interviews you should record 100 of your interviews unless the person being interviewed rejects it and then you got to go to teamwork makes the dream work ideally one person is conducting the interview the other is taking notes remember saturation is what we're trying to get to here you're talking to participants until they all sound about the same in the applied setting i find that this happens but starts happening between five and seven interviews sometimes as many as twelve if you're not hearing it after twelve something's wrong if you're either interviewing on on a topic where people are all over the place you've recruited the wrong people you're missing a variable something is wrong if you're not hitting saturation at some after 12 interviews and then remember interview not interrogate you're there to learn about people's experience not get a confession or convince them that your product service or idea rocks remember this what who what when where and how not why aim to ask what and how they're bad at telling you why why did you buy my product is a shitty question why did you choose between product x and product how did you choose between product x and product y better what about product x made you think that to be a good match that's a good question remember no matlock you're not cross-examining people during a trial don't lead your interviewees isn't it true ari that you ate a bagel this morning for breakfast now you've got me saying yes or no and avoid yes or no questions and listen leave room for explanation and listen and then i want you to dig deep the best stuff comes when you dig a little deeper on something that strikes you so the script is your roadmap remember it's a plan but it's okay for you to diverge from the plan there's no reason you can't return to it after some period of time and what's really interesting is let's say you diverge and you get a bunch of gold and you only hit two questions on your interview guide well there's two things that could happen either a the interviewer's gonna be like hey i can go a little longer it's cool or the interviewee can go a little longer i mean or b okay you hit it on the next interview um but you've gotten something that you wouldn't have gotten otherwise and the whole purpose of this is to add value to your conversation or or to your research remember focus on quality not on quantity this is a reminder all of your interviews should be recorded unless the person being interviewed does not explicitly um give consent for that you want people to consent to interviews don't record people without them knowing but you want to ask and get consent i have had in my history of doing doing this work in the corporate setting i have had only one person ever displayed discomfort being recorded one and i've been all over the world like it um i want you to transcribe your interviews here are some options there's trent there's temi there's otter there's rev um and or if you really want to get fancy about it you can go to fiverr or upwork and hire a freelancer send them the recordings it's like the flavor wave oven they'll set it set it and forget it and then once you get the recordings back and the transcripts you can analyze with it cool i i like delve it's simple it's really good for for uh applied researchers uh who don't have don't want to spend a ton of money on like uh atlas ti or um any of the other major tools uh qualitative analysis tools and you you code the the interviews and by what i mean by coding is you're going through and you're looking for themes and similarities between all the interviews you're pulling out things that are important and then you're going to make comparisons across them and then ultimately you're going to take that now something i teach all my interviewers is this i'm a huge fan of the godfather right the godfather sitting there on the right and on the left is tom hagan and he is the consigliere he's the advisor to the godfather i tell my researchers this we don't deliver data and we don't deliver insights we deliver counsel we don't deliver data we don't deliver insights we deliver counsel we give advice to the business you give advice to the business and then you tell the business how they're going to operationalize what you've done if you don't do this what ends up happening is you create a bunch of data and information that's just not getting used so at the end of your research there are three things that i require my researchers to do and i would encourage you to pick up today one is the at when we're delivering research the first thing we're delivering is a point of view that's that is our recommendation we're saying from the data and what we know about the business and all these things remember we're going back to the background information that i made you put on the research report in the beginning from all of this this is what you should do this is what you should do it's a really uncomfortable thing to start doing because we like to hedge our bets people like don't want to be wrong so instead of being wrong they'll say well you could do this or you could do this but the decision is kind of yours and by saying that you're basically the business is like well i don't know i mean tell us what this all means after you've given the point of view i want you to give a summary and the summary is like okay here's the executive summary here's what you think you should do here's an executive summary supporting it and then i want you to reaffirm this is the decision you should make amazon teaches its its people that there are two types of decisions there are reversible decisions and there are irreversible decisions 99 of the decisions we make are reversible you'll be able to reverse them so you you're going to give them a so uh some advice and the reality is is in the police world we use this this phrase it's the facts and circumstances available at the time so we're giving you advice with the facts and circumstances available at the time which is real time because we just did interviews it's not backward looking it's not from from a year ago i mean the reality is is nobody could have predicted the business impacts from kovid before kovit hit the way it hit because nobody was if you were looking backwards to get that data you'd never have gotten it so there are reversible decisions and irreversible decisions remember that most decisions are are reversible for the irreversible ones we want to be a little bit we want to be a little more conscious of those but remember the the key here is to focus on the quality not on the quantity so i know i blasted through that relatively fast i wanted to make sure i got through it and leave some time for questions so i will remind you it has been long it has long been an axiom of mine that the little things are infinitely the most important but to a great mind nothing is little um and so with that i would open it up for if there are any questions or if there are any uh comments or disagreements or or agreements hey all right it's bill hey bill all right looks like we had well a couple of questions uh will oh will this be available will your slide deck be available after presentation um i don't see why not yes i will make it get a pdf version to you and you can share it out as you see fit yep yeah we'll post it on the on the conference website along with the archive video and all will be available uh second question was can you finish the story from your attention grabber please ah that's funny so the child right uh because i i had a good few good stories i thought but um the child abuse case he uh i ended up uh we ended up investigating that the child uh was definitely abused and neglected i gave you the six data points that i think proved that um but obviously as detectives we dig a lot deeper the child did have some some developmental delay but not enough to be put in a diaper i ultimately charged the mother um lived with his mother and grandmother um i eventually the grandmother was uh about 50 lucid the mother though was completely lucid and i ended up charging her with homicide uh for the for the death of the child he uh it was a really really sad case it's probably one of the most touching things that that i've ever had to go through um and she ended up getting convicted of that um it's a sad story there's no there's no never a good ending to those cases um but i'm grateful for the opportunity to have like served and gotten to help close that up and hold the people accountable for for those kind of bad actions it's funny ari i always ask people you know the question of how how did you get into research and i think you you get the gold medal for for one of the most interesting paths because i feel like all of us sort of make our way into this industry from a lot of different different backgrounds um yeah it was an interesting path but i think it's yes it was an interesting path and i i don't know many others that have done it although i think detective work and insights work is very very similar in a lot of ways all right that may be it for follow-up questions just checking one more yep it looks like that's it um any parting shots for us harley no i think um obviously you can uh always uh visit my my if you have questions you can visit me um at my website which is arizona.com i put my email on this last slide if you have if you have questions and then i offer um some sort of not some sort of i offer some interview uh training that helps professionals become better interviewers through interview jump start and interview university that's that's the interview jump starts a good starting point but interviews interview university as well um those are those are the pathways all right fantastic um well that gives us about it's like 15 or so minutes before the top of the hour um let everybody take a break grab a beverage or lunch if you if you see fit and we will see you in just a few ari thanks a lot really appreciate it


