What about the rule of three, as mentioned during the talk?

Leslie described it as a guideline she was taught early in her career: for every one minute of interview recording, budget three minutes for analysis.

To what extent are you using these techniques in a hybrid fashion, combining interviews with surveys?

In unmoderated self-interviews, Leslie follows up a participant's open response or demo with a Likert scale question in the tool. This gives her both qualitative texture and a metricized response she can communicate to stakeholders who prefer quantitative data.

Have you done anything out of the norm to recruit participants during COVID?

In a previous education-focused role, the team offered Uber gift cards to college students who lacked cars. At Starz, incentives have included free subscription access for new shows and branded merchandise tied to popular programs. Leslie also emphasized that reliable, on-time compensation and clear communication of expectations makes participants more willing to re-engage in future studies.

How much do these remote practices carry over after a return to normal?

Leslie expects remote research to remain standard. It enables broad geographic sampling that local recruiting cannot replicate, and participant availability often falls outside business hours, which remote tools accommodate. In-person methods may return for specific use cases, but hybrid approaches are likely to persist.

Have you used the make-up/break-up exercise outside the Americas or Europe, and how might it apply in other global markets?

Leslie has found it works well in North American and European contexts. She noted it is less likely to translate in Asian markets, where personal relationships tend to be more private. It also works better with younger participants, roughly college age through early 40s, and tends to lose relevance for older demographics who are less embedded in the dating context.

Tips, Tricks, and Techniques for Conducting User Interviews

Leslie Salazar Bushell, a senior UX researcher with over 20 years of experience, walks through practical methods for making user interviews more efficient and analytically rigorous in a remote work environment. She covers a time-stamped indexing approach that replaces traditional note-taking, consistency techniques for managing organic conversations across multiple participants, and hybrid methodologies that go beyond the standard remote research toolkit. The talk is grounded in real constraints researchers face, including limited tools, reduced team support, and restricted access to participants during the COVID-19 period.

Key Takeaways

Stop taking verbatim notes during interviews. Instead, use a time-stamped indexing system tied to your question outline so you can navigate recordings by topic rather than listening end to end.
Use a coded shorthand (icons, letters, symbols) during sessions to flag clips, insights, and quotable moments in real time, which cuts post-session analysis time significantly.
Write interview questions in conversational format and use a physical checklist to confirm topic coverage. This keeps sessions organic while ensuring consistency across all participants.
Print your indexed script after each session and use it as a quick-count sheet. Tallying how many participants raised a given issue from the script is faster than mining software logs.
Expand beyond the standard remote research four (analytics, usability testing, surveys, interviews) with hybrid methods such as self-interview videos, the make-up/break-up letter exercise, cognitive walkthroughs, comparative analysis tasks, and before/during/after session structures.
The make-up/break-up methodology works best with younger demographics (roughly college age through early 40s) and in North American and European contexts. It may not translate well in markets where personal relationships are more private, such as parts of Asia.

Questions & Answers

What about the rule of three, as mentioned during the talk?: Leslie described it as a guideline she was taught early in her career: for every one minute of interview recording, budget three minutes for analysis.
To what extent are you using these techniques in a hybrid fashion, combining interviews with surveys?: In unmoderated self-interviews, Leslie follows up a participant's open response or demo with a Likert scale question in the tool. This gives her both qualitative texture and a metricized response she can communicate to stakeholders who prefer quantitative data.
Have you done anything out of the norm to recruit participants during COVID?: In a previous education-focused role, the team offered Uber gift cards to college students who lacked cars. At Starz, incentives have included free subscription access for new shows and branded merchandise tied to popular programs. Leslie also emphasized that reliable, on-time compensation and clear communication of expectations makes participants more willing to re-engage in future studies.
How much do these remote practices carry over after a return to normal?: Leslie expects remote research to remain standard. It enables broad geographic sampling that local recruiting cannot replicate, and participant availability often falls outside business hours, which remote tools accommodate. In-person methods may return for specific use cases, but hybrid approaches are likely to persist.
Have you used the make-up/break-up exercise outside the Americas or Europe, and how might it apply in other global markets?: Leslie has found it works well in North American and European contexts. She noted it is less likely to translate in Asian markets, where personal relationships tend to be more private. It also works better with younger participants, roughly college age through early 40s, and tends to lose relevance for older demographics who are less embedded in the dating context.

Session Notes

The Remote Research Environment

The COVID-19 period created a specific set of constraints that changed how UX researchers work. Leslie framed these constraints before presenting her solutions.

Most research is conducted fully remote, through conferencing software and online testing tools.
Teams are smaller in practice: researchers often handle data collection and full analysis alone.
Budgets may not have anticipated software costs for remote conferencing or testing platforms.
Access to participants is reduced because users are also at home and harder to reach.
Methods that depend on in-person co-presence, such as interactive design sessions and focus groups, are largely unavailable.
Interviews have become the default methodology because they are data-rich and achievable remotely, but they carry their own costs.

The Core Trade-offs of User Interviews

Data richness vs. analysis burden. Interviews generate a lot of material. A common guideline: budget three minutes of analysis time for every one minute of interview.
Depth vs. consistency. The organic nature of interviews allows for productive tangents, but following one participant down a path that others were not asked about creates gaps in the data.
Tool flexibility vs. feature gaps. Researchers work across everything from full-featured UX platforms to basic conference calls. Not every tool supports the same capabilities.

Tip 1: Stop Taking Notes, Start Indexing

The traditional approach produces two data sets that must be reconciled: the recording and the handwritten notes. Reconciling them can double analysis time. Leslie's solution is to stop capturing verbatims and instead use a time-stamped index.

Write your script in outline form. Organize by main topics, then questions within each topic, then anticipated follow-ups. This structure becomes your indexing document.
Log timestamps, not words. Before asking each question, note the elapsed time from the recording. If a participant answers a later question early, timestamp that reference too and skip the question when you reach it.
Use a coding shorthand. Replace written notes with icons, letters, or symbols. Examples from Leslie's practice: a smiley face flags a clip worth showing the design team; a star marks a key insight; a Q marks a quotable moment.

This approach lets you move through recordings by topic across multiple participants rather than listening to each interview in full sequence. It also reduces recency bias: instead of your impression being dominated by the last interview you conducted, you can compare all participants on the same question at the same time.

Tip 2: Be Organic With a Checklist

Consistency across participants matters for identifying patterns. But a rigid script can make participants uncomfortable and close off unexpected directions.

Write questions in conversational language. The goal is to cover required topics while maintaining a tone that feels natural to the participant.
Check off topics as they are covered. Use a literal checkbox in your document. When five minutes remain, a quick scan of unchecked boxes tells you exactly what still needs to be addressed.
Leave space for unexpected questions. When a participant raises something unplanned, log a timestamp and a keyword, then decide whether to add that topic to the script for subsequent participants.

Tip 3: Compensate for Tool Variability

Because tools vary in features and availability, the index script acts as a consistent layer that works regardless of which platform you are using.

Use whatever features your current tool offers (timestamps, flags, annotations) and let the script serve as a map to that content.
After each session, print the script and use it as a quick-count sheet: mark how many participants raised each topic, how many clips or quotes exist per question, and any patterns emerging.
Use the printed sheet to pull numbers quickly when writing the report, rather than navigating software to count occurrences manually.
After every study, note what could be improved in the process. Ongoing iteration applies regardless of experience level.

Expanding Beyond the Standard Remote Methodology

Leslie identified four methods that most remote researchers default to: Google Analytics, usability testing, surveys, and interviews. She encouraged experimenting with hybrid approaches that combine elements of these methods.

Self-Interview (Unmoderated Video Response)

Participants record themselves answering questions, similar to a vlog or confessional format. Questions are posted through a testing platform or delivered via survey, and participants submit video responses. This eliminates scheduling and still produces rich, qualitative data. If a dedicated platform is not available, participants can record on a phone and submit via email, with compensation issued on receipt.

Make-Up / Break-Up Letter

Participants are asked to personify a product, service, or feature and write a letter either making up with it or breaking up with it. They explain the relationship history, what went wrong or right, and why the decision is being made. This method humanizes product feedback and produces data that is easy to communicate to stakeholders, because the relationship metaphor is universally understood. Analysis can include word clouds, phrase frequency, and the ratio of make-up to break-up responses.

Everybody has that connection to the idea of making up or breaking up with somebody. It really contextualizes it quickly for all the stakeholders, users, and researchers involved.

Note on applicability: This method works well with younger demographics (roughly college age through early 40s) and in North American and European cultural contexts. It is less likely to resonate in markets where personal relationships are more private. It tends to lose relevance for older participants who are less embedded in the dating context.

Cognitive Walkthrough

Instead of asking structured questions, invite the participant to narrate a real task from their natural environment. For example: it is Friday night and you want to watch a movie. Walk me through how you find one. This surfaces behavioral data, such as whether users browse by image, read descriptions, or rely on saved playlists, that direct questions would not reliably capture.

Comparative Analysis Task

Have participants step through two competing products or features side by side. The comparison itself tends to surface unprompted preferences and feature gaps. Participants naturally reveal what they want when they can point to what one product has that another does not.

Before / During / After Session Structure

Divide a session into three phases. First, capture the participant's expectations before they engage with the product (based on marketing materials, packaging, or first impression). Second, run a cognitive walkthrough as they use it, noting reactions in real time. Third, debrief after use to ask whether the experience matched their expectations and to gather reflective feedback. This structure provides context for the new-user experience that a single round of questions cannot.

On Remote Research After COVID

Leslie's view is that remote research is here to stay, even as in-person options return. Remote methods allow for geographic diversity in sampling that local recruiting cannot match. She noted that participant availability is often outside business hours, and remote tools accommodate that. In-person methods may return for specific use cases, such as co-design sessions, but hybrid approaches are likely to persist as a standard part of the research toolkit.

Transcript

Read the full transcript

all right we are at the top of the hour so i think we're gonna go ahead and get started looks like we've got some attendees uh filing in so what i'll do is kick us off do a quick intro and then we'll get to some some speakers uh my name is bill mcdowell i'm ceo of accelerate research uh welcome everybody to the may installment of the accelerant research virtual insights conference um this is your first time joining us welcome if you're joining us uh again welcome back um quick overview of these these webinar conferences that we've been uh hosting for the past gosh year plus now since pandemic set in accelerate research we've been hosting a series of virtual insights conferences we um as an industry haven't been able to get together in person for quite some time looks like you know things are beginning to to loosen up so thank goodness for that but we've been as an alternative hosting these uh virtual insights conferences and you know the the format is is pretty simple basically you know just a bunch of research nerds getting together talking about their experiences and expertise in zoom webinar format we've got a few ground rules that we've been operating by for for these conferences um just you know the quick reminder that we are at the mercy of the tech um our speakers are still you know in large part uh logging in from home and are at the murphy of you know whatever bandwidth connectivity dogs barking children crying uh issues that might come along so uh let's be patient with them um you know i don't anticipate any issues um everybody's been quite gracious uh with these you know since since we've been doing them for the past several months um you know that brings me to the the second point which is you know just try not to to troll i know these aren't in-person uh webinars so there is the anonymity factor and the you know potential that you know you don't have to bump into somebody at the the buffet line like you might at a normal conference so um just you know continue to keep keep your manners about you um not a problem that we've had but just a reminder um and also again not an in-person conference but to the extent possible let's try to uh try to network with one another try to engage with one another both um our speakers as well as you know audience members amongst yourselves um if you're in the zoom webinar definitely feel free to take advantage of the the chat function the q a uh we've got other folks that are logging in and accessing via youtube uh linkedin twitter so if you're on these other platforms um feel free to use the hashtag arvic post about you know what you're seeing ask questions with one another reach out to one another and again network to the extent possible shake hands virtually uh if you can um i'm gonna spend very little time talking and hand off to you know the folks that are giving us the wonderful content here but just quick overview of of our speakers for the day um we've got two hours worth of uh insight content coming your way uh we're gonna i'm going to go backwards on the agenda here so uh next up at 3 p.m eastern is going to be molly molsom from schwab but our first presenter of the day is going to be leslie salazar michelle and she's going to be talking to us about tips tricks and techniques for conducting interviews and making the methodology work for you so we've got a couple of uh user experience researchers presenting today um but on some some very compelling topics that are obviously applicable to you know no matter your your industry or you know what type of research you're getting into this is some some very interesting practical um techniques and tips and tricks that are going to be talked through so very excited to hear the content um honestly that's going to do it for me i'm going to shut up let leslie take over and and you know do her thing i'm going to keep an eye on the chatter on the q a um and help you know lead q a um toward the end of leslie's presentation like i said definitely feel free as we're going to to log into the chat to the q a to ask questions uh log comments um you know be as interactive as we can um but that's it for me leslie i'm gonna let hand over to you and let you do your thing well um hello everyone let me get my screen share set up can i see there we go make sure everybody see if i can allow for this [Music] okay there we go okay hold on just a moment can you see my screen now not yet okay hold on i think that there might be an issue hold on is your screen sure there can you see it now yep we got it perfect um hello everybody my name is uh leslie salazar bushel um i wanted to share with you i was talking to bill about when he asked me to present what would be some compelling topics to talk about and um i gave him some options of some things i could could address in the in the webinar and uh this is the one he says yeah i think that that would be the most uh interesting for the audience that we have who attends the conference in the webinar calls so hopefully you'll get something out of the my presentation today um let me go in here um you're probably wondering who i am um i just want to say welcome to you all my name is louis ella bush again and i'm a i've been a ux researcher for with over 20 years of experience i work in industries such as communication government security education and now i am uh the senior ux researcher at stars um i've worked for fortune 500 companies i've won to some design awards and i have a couple of patents that i filed um from my work um and the largest reason that i'm here is i just believe in community mentorship and contributing to the greater good and helping my fellow ux researchers whenever i can if you want to get in touch with me um there's my email i'm also on linkedin so please feel free to reach out to me um after the conference if you want all right so um in talking about interviews um i wanted to kind of address the elephant in the room and about interviewing in the climate of today of of covid and working at home that we've all had to do and so in thinking about that i started to think about how we are working with the constraints that we have around us we're all working remote right now i mean there might be a few people who get to do in person but for the most part um everybody that i've talked to who's working from home um they're conducting research online through emails and conference softwares and end user testing tools because we are remote there's less support even if our we're in a larger team so we're largely independent um we have minimal team members helping us or taking notes for us or helping us set up or help us with technical issues things of that nature we tend to be an army of one doing both the data collection and the full analysis our resources are probably limited you know if you were doing mostly in person interviewing or testing um research beforehand um you probably didn't budget in for software's or uh conferencing or things of that nature and so you may be working on a limited resources time access to users that's been cut off just because they're they too are working from home or not out and about and then the methodologies that get that you can actually do remotely are highly limited um you know we um you know we're we're at the mercy of the technology again just as bill said for the presentation but we're also at the mercy of of of other things going around uh and just being able to get the most work um that we can with the things that we have um suddenly things that we were able to do maybe like an interactive design session or a focus group those things are out the window because we have the limited technology um and so you know a lot of us will default to the interview process because the methodology tends to be really data rich but there are a lot of good and bads to the methodology that we have to take into consideration so when you look at doing user interviews um why we make why we choose this mythology and what makes it hard is that you know interviews have lots of data you know you have a set time with a user to ask them the questions um and then you know get the wealth of information that they have it's like drinking from the fire hose um but with that the yang to that is the extensive analysis needed you know the rule of three like for every minute of the interview capture give yourself three minutes for analysis that's what i was always always taught um and so that can be extensive um you know interviews allow for deeper questions and follow-ups um you know participants will go on a tangent you can ask them like oh can you explain that a little bit further um oh that's something that came up that i didn't think about let's go down that path um but with that but with that ability to go into deeper questioning and follow-ups um there's inconsistencies in the data it's hard to maintain a question set across all of the users that you're going to be interviewing with so making sure that questions and topics are covered by all the participants through the through the time limit that you have making sure that things that come up are addressed in the next users that come through the process all of that creates uh inconsistencies in the data that you might have at the end and the last thing is tool variability i've conducted interviews on everything from you know go to meeting to usertesting.com to uh uh lots of other tools uh users you can go now um just you know to basically some some tools have these these features some tools have those features some tools have no features you may just be doing a conference call where you they're connected and the only thing you can do is record um so you know while there are great now that we have the ability to we have the ability to do our work at home the fact that there isn't the tool set maybe not be present for us to utilize as we do that work from home it becomes very manual so i'm going to go through and kind of deal with those each individually and one of the ways that like when i go through and i deal with a large set of data and data analysis in an interview there are usually two data sets that come that you are working with when you're conducting an interview one is the actual recording and feedback that the user is giving you um you know so there you have it recorded they're telling you exactly what they're feeling and that recording is being captured for posterity the other data set is the notes that you're taking in the interview session um those notes i mean i as a person as a personal person who um you know i tend to capture information by writing things down you know i can make a checklist of groceries that i need to get the store once i've written that down i i internalize it and so i will remember everything that i need on my list usually i say usually um but um you know that as you're writing through your notes and doing interviews you're creating another data set and a lot of the time that you spend afterwards in analysis is reconciling your notes to the recording that you have before you and that can actually take a lot of time um you know that can actually make your analysis time double um is what i found so how do i do deal with this issue i stopped taking notes and this is my first tip so stop taking notes start indexing um one of the things that i learned early on is that you know like i would spend a lot of work trying to frickin sell my notes with the interview and try to pull the insights and it just wasn't working i found that one of the things that i really wanted i really helped was creating a list of questions but doing it in an outline form um and making sure that things were like you know this is my main topics that i want to touch on these are the questions for those main topics these are those follow-up questions that i might have and basically thinking of it in an outline um once i did that i started to like write notes against that but i was still finding reconcile issues reconciliation issues so what i started doing is i stopped writing the the user was telling me because i knew it was being captured and i started doing um indexing with time stamping so in all the recordings if i um you know it will take the recording and it will tell you how much time is elapsing um as the session is going even the most simple of softwares will do that if the software didn't do that back in the day i would just set up a clock and as soon as i hit record i would watch the timer go off and what i do is i ask questions i go through and i add a time step so when i ask a question about a right before i ask it i write the time and then i ask the question um even if a question even if i have a question on my list like let's say i ask a user a question and i ask them okay you know what about this and they actually answer another question that i have on my list i'll go ahead and actually timestamp that as well and then i just won't ask them that question further on this helps a lot with analysis because you're going back and coding the information the third thing that i started doing is instead of writing a lot of words i started coming up with a coding system i'll use letters numbers icons um to indicate quality of responses uh to track like the metrics things that i'm trying to measure in the study um and then i'll instead of writing extensive notes i'll just put the little icon on there to give you an idea what this looks like um i created a sample um and my picture is in the way minimize that um so i created a uh a sample of what this might look like so like when i'm doing an interview you know i'm working with user three i'm going through the interview process with them as i'm going through the thing i'm looking at the time stamping and putting the time stamp next to the question um as if they address a question multiple questions or more than one question i'm putting that time step next to those questions um if somebody does something that i feel would be interesting to show to the design team i will put a smiley face and that's a clip if they say something really profound that i feel like wow that really shows me something that i really really need to internalize and ingest as a team i'll put a star by it and if they say something that's super profound um just like wow that sums up the issue in in one sentence or in one way i'll put a q next to it in quote um you know as i go through if there's a question that that suddenly i realized doesn't apply to them um i'll go ahead and cross that out like that's not a question that needs to be asked because they addressed that in a previous question and i make those notes um oops sorry about that the next thing that um as we go through interviews that i usually deal with as i'm interviewing users is dealing with with the follow-ups and the consistencies basically making sure that the data is consistent across all of the interviewees um you know the great thing about interviews is that it's organic in nature um you know you uncover things that you don't even maybe even release that were an issue or didn't even occur to you or your team to ask about you know it really gets up at the heart of of how users are feeling um and you know you can even if they bring something up you can kind of go down on a tangent but just because you go on that tangent um conversation you may not go down that count that that in that rabbit hole with all the users and so a lot of times you want to show magnitude of an issue so you want to make sure that as you meet with future ish users or future participants you kind of make sure that that isn't something that also addresses them as well and you know and if you go on a tangent with one user and don't go off you know go down that rabbit hole with them with the next user it's hard to identify um patterns in the data that you're having so the way that i address this is that i try to be organic with a checklist which sounds uh um counter-intuitive but what i like to do is like when i go through and write my script and my in my outline form um i will actually write my questions in conversational format so that as i go through i cover the topics that i want to but i still maintain a certain tone that is more organic and more comforting to the user and that really helps a lot so that makes sure that i'm covering the things that i need to but then if as we go off on it go down and grab a hole of information i'm able to come back and still maintain that air of conversation that comforts the user um and then as a user covers a question physically check a box um i will use in google docs reward and like in the outline table i will literally create a physical box that i check as the question is answered or participant covers the topic um i love checklists and maybe that's my type a personality but the feeling of actually going and going okay that's been covered um especially when you're at the end of like you have five minutes left and you want to make sure that you get to everything it's really reassuring to see as you go through the list go yep covered that yes covered that yes covered that and then making sure that you're hitting the points that you need along the way so that the time that you're spending with the participant is meaningful and you're getting the information that you need um and then the last i plan for questions that i didn't think of or didn't ask sometimes when they when you users give you information that you didn't even think of um or give you unintended answers to your questions um you want to be able to capture that again um and so what i'll do is i'll leave space for those questions i plan for them that come out of the questions that come out of the blue and then i'd also i'll time index those i'll use a keyword to remember what the question was about or what the top the information was about and then i'll consider about adding that into the script for the next user and just to give you an idea what that looks like um so here is my modified list kind of showing you those features that i've added in so here i have a check right my outlined checkbox and i've checked all of the questions that that apply earlier i crossed out a question that didn't apply but having that question crossed out and then having it checked shows that me that the reason that it's checked is crossed out is not because i didn't i forgot about it but because i it wasn't something that i needed to worry about um and i can clearly go through here and see like okay i've covered all of this area let's move on um even if things don't you know happen in the order that i have outlined i can still go back to my recording um when i go to do analysis and say okay at 3 18 this is when they talked about what services they didn't like um and then at 406 this is why they didn't like them um but when like when they talked about the services they're using how long and what the one and they start talking about some of the ones that they really like i can actually go back and and jump through my recording um appropriately without having to worry about listening to it over and over and over again and it really helps me get to the information quicker and be able to you know move through the content and find things oftentimes if i want to analyze things in math i'll go from one just listen to one user at that point and then go to another recording listen to user and they're reporting and kind of go by topic by topic and then the one the questions that come up that when we get something that is unexpected you can see that i've have some lines that i've added at the end of each section i put the time stamp i check it that this was covered and then i put a quick word about where what was covered during that thing so when i talk about the services that they're happy with um they go on a tangent of the things that they love like like extremely enjoy and so i put word love and you know i put my little uh notations that you know this is a clip worthy this is insight worthy and this is um you know a quote that i actually want to share with my team um again and then when i ask them what service you know what streaming content devices do they use you know i have a user that randomly says game console and it's like that's not something that i thought about so i'll add that in and i'll put the time that they talk about that that as a source um the next thing that tends to be an issue when you're doing interviews is tool variability i've worked for uh government offices that had no budget and i had to use the best tools that i had of it you know the cheapest tools that i had available to me or free and i've had um companies that gave me an extensive budget so i could have a really nice um analytic tool that i can go in and and do notations and editing and uh transcription and it's great um but you know with all of these wonderful tools you know you can vary um depending on the project that you're on the environment that you're working on of what you can and cannot forward and then every tool doesn't have the same features um you may just not you know you may have a you have liked a feature in one but it's not available to you at the time so you know how do you compensate for the tool vira uh variability and the analysis that you have to consistently do across those tools um my tip would be use your tills fully um but maintain written scripts so like the doing using the the indexed script that you have you know um i would say go in and you access the ux tool that you have and use it use the features to timestamp use the features too and then use your script to help you do that indexing in the tool but you know annotate add your notes you know once you go in afterwards and you know a lot of tools you can go in and add your notes like more detailed notes add them in add your the things that you're noticing um that you want to make notations of and add that to your time stamping but the index script that you're maintaining still provides a way for you to get to that content still quicker um because you can basically have the script in front of you go into your tool and say oh yep there's the flag for that um you know this has a hashtag this has a flag this has a a word that i put in there or an icon and it's really easy to go back and forth and working in your tool and it's kind of like a map to with content that's in your tool also i'll print off once at the end of the session i'll print off the script again and i'll use it for really quick counts that i'm that i'm doing so like um you know four people brought this issue up um three people um you know talked about this or demoed this and i'm able to say you know four people said that they'd use this service two people did that i can actually will print off the script and i will actually start using my script as an indexer of of those quick counts because i'm going to actually when i type up my report i want to pull that information up quickly i don't want to go through no matter how robust the tool is i don't want to go through the tool and say okay this user okay that's one that's two and then three and then have to write that out i can just use my a version of my index script to have those counts readily available as i'm doing my report um and then adding notes about the process um as i'm going through that final sheet of that index script in my accounts i will make i might add in something um that i didn't think about like a data point or add something into my coding index that oh this this was something that users did and so i'm going to indicate every time that they did that and i'm going to add that as something that i'm going to report on um you know every time i go through a process i or a study i think about the process that i utilize and how i could improve it and so there's always room to improve how you're working to become more efficient and you know no matter if you have two years experience or 20 years experience um being um self-aware of of how you're going about things really will help you be more efficient over time so the way that this looks so here is a version of my check of my script and you know in here instead of having the time stamps like i'll print this out separately you can see that i've kind of gone through and i'm starting to go through and say okay how many streaming services are currently being used well they range from three to six you know there was a couple of clips that i want to pull from these for people answering this question i have about four insights and there's one really good quote what services are they using um netflix um they're disney they're hulu i keep account every time they say that they're using a service you know how long everybody's averaging about two years and i'm able to go through here and add like i said quick counts and quick notes um what services are you happy with you know in you know in the asking that question i had two users that demoed the actual app in the interview and said hey this is what i like um and this is what i really and they showed me and so i can actually take that and say okay that's a demo and so basically take those and pull that nut there's kind of like a special clip to show my team um here you know i may have not formalized a question about what they love about a product but i still have i kept it on there and i still keep that as a count and like three users you know talked about things that they loved and i accounted for that as far as clips insights and quotes and then you know a couple of people talked about game consoles and so i went ahead and added that as a full-time question i highlight it as a it's an afterthought question so i made it a different color and then i i continue with the counts of what people what platforms people are using and so i i keep those counts there and so when i'm going and doing my report it's really easy to pull those up i don't have to even though i have an awesome software or an awesome service i don't have to worry about uh navigating through there and finding those i use this as a quick sheet to get at that information okay um and then you know as we are in what i call the covid research rut i find that there are kind of four methodologies that we tend to kind of trudge through so you know what do we have available to us we have google analytics um so you know we're using the data that our company is collecting to see insights and behavior because that's easy to get at it's cheap it's it's available to us and we can look at this and see like is there a behavior that a pattern that we're seeing um you know usability testing you know a lot of us have usability platforms and testing platforms that we can basically do your standard usability tests our designers are creating workflows um we're they're making prototypes we're testing them things are going to beta we're running usability tests on that we're testing them you know we can all run usability testing interstate because that is at the end of the day our bread and butter um surveys you know everybody's at home we've got time to waste hey let's send out a server to survey to our end users or target customers um surveys are you know whether it's a five question intercept survey that you're sending through your tool or you're setting it out you know uh as an incentive via social networking and your marketing team you know those are that's a really easy methodology to do and again the interview you know if we have to get a lot of data from somebody and get some insights before the design team starts working why you know yes we can do interviews because we have conferencing software so um the way that i'm dealing with this you know mundane methodology um focus um is kind of you know just taking stepping back and saying you know i'm lucky to be able to do the work that i do from home um while maintaining social distancing protocols but i want to do something that is fun and interesting not just for me and our participants but also for my team a lot of times my team is involved in watching the results and seeing the data that's coming out of the study so how can i get out of that you know four-step process that wheel of these are the four things that i'm doing while working from home um my encouragement is to try something new there are lots of what i would call hybrid methodologies that takes advantage of the basics of usability testing surveying and interviewing that can be changed based on how you look at the data so one of the things that is easy to do is a self-interview in today's day and age users are more and more comfortable with the idea of vlogging of doing a tick tock of doing um a social media post of them just talking and and giving information and so a lot of tools are now avail there's a lot of tools out there um that you can have the user you know basically you post the question and they answer that and it's unmoderated and the great thing about this is that you can launch a study and people are still doing an interview um confessional um a deeper type data give but uh you don't have to schedule the time there's a scheduling that goes away and you're still getting that rich information um and so a lot of times you know scheduling an unmoderated self interview you know you can use a standard usability study software like usertesting.com userzoomgo anything that you're using to do that you know if you don't have access to those softwares you can have people sign up for you know through a survey and then have them submit a video to uh an email and then once they you get that confirmation you know they've completed the task and so you know everybody has a phone that has record let them use that phone and then have them send the videos and then you compensate them after you receive those videos so you don't even need a fancy testing software but you know utilizing what people are already doing and are comfortable with um allows you to get at that information and then that will freeze you up to do other tasks another methodology that i really like is the makeup breakup ladder um what this does is allows you can use like a software uh survey software um and basically propose the idea to somebody to humanize the products that they are using thinking about a collection of things or features and then ask them you know like based on your your your interaction with this product or these services or these features name something that you would either make up with or break up with and then tell me right like write a letter to that person to that feature or product or service like you're going to like you're going to make up with them or you're going to break up with them talk about the issues that you've had why you're doing what you're doing and why you think that the making up or breaking up with them is necessary um it really humanizes the feature and it makes the user it's fun for them and then the data that you get back is really quite interesting because you really get at the heart of why something does or does not work um and so that's a fun um i do that when i we have interns in both in this my current role and my previous uh roles we would have interns and i would work with the interns and like and this was always a fun methodology that they utilized um when they had to to write up their their test plans because you know it's it's a super interesting way to look at the data um next you could do a cognitive walkthrough you know instead of asking users to you know do tasks or um review something just have them do a process so for example i work at stars you know you know tell me you know you're gonna it's friday night and you want to watch a movie how do you find a movie that interests you um using the platform and just have them you know have them sit in the chair that they're um that they normally sit in or the air part of that be in the part of the house that they're normally at and say okay i do this i like to you know have them describe the environment um what are the things that they're what are the features that they're using are they do they have an established playlist are they going through something and saying reading are they reading the descriptions are they using the information that's on the page are they reacting to the images that's all important data that you would you can't really surmise in a questions so you know letting them take you through their what they're thinking and have them walk through that process as opposed to just asking a question and and writing the answer asking a question or writing the answer um another thing that you can also do is have them do the comparative analysis you know a lot of times we look at products and say okay this product does this this product does this this product is that have them go through the fact of two products that they're using and then have them compare because a lot of times when they do that comparison they're going to tell you i like this i don't like that and it's kind of an unintended uh benefit of doing having them to go through that comparative analysis so i'm like oh yeah i'm looking at this feature they don't have this i really wish they did oh this one has it but i don't like the way that it's presented you know when you get back at like okay so we need to offer this feature but we need to present it in that way um and that really is a fun way to have people kind of like give you their feedback and they and they feel like they're more invested like oh you value what i'm saying in my opinion um the last one that i like to do is the before during and after so um you might set up short shorter interviews with users and say okay let's talk about your expectation i'm to have you use this what is your expectation going in what do you what do you think this is about you know giving them either the uh marketing materials or the write-up of the product um and then seeing what you know what is the infra what is their expectation going in with what information that they have or just seeing the packaging or seeing the the service for the first time and they'll say like oh i think it's going to be able to do this this and this you know and then have them as they go through using the product the during do like a cognitive walkthrough where they're using it they're they're reacting to it um and you're writing down like okay you know oh excited you know 10 15. um you know ooh confused you know at this time and then um you know step there you know confuse the step five um you might put that in your notes and then you're you're actually looking through as they go through and then afterwards after they've used the product or um gone through the process you can say okay you know tell me how you felt you know um and they can respond to their actions afterwards um you know and then you know you want to ask them like was this did it work the way you would expect and they can say yes no and then it kind of explain why and it's a different way to get at the information um than just asking them you know the standard questions of do you do this do you do that why do you do this why do you do that um and so it and it really gives you context of um a new user coming in and like what they're actually experiencing as they uh see if each a feature for the first time um that is all i have um i'll open it up to questions that anybody may have about the information that i presented let me see if i can go in here where is this let me see okay um i have let's see one from heidi um never heard of the rule of three learning something uh already so that's good um any questions anybody hey leslie hey bill yeah and as as we're waiting for questions um i had a couple of thoughts comments um yeah number one is just a big amen to the sort of iterative process you described in terms of you know chronicling and recording as you go as opposed to you know at the end right you know it makes life so much easier with qualitative research to you know have this sort of traveling set of notes observations takeaways that that you build as you go as opposed to you know trying to go back and rewind the tape for for everything that you've learned over the course of these series of interviews yeah um as i mean i've i've started doing interviewing and it was always something that i would go through and like okay you know and i i am a person like i said i internalize things by writing things down so being able to let go of writing everything that i heard was a really huge hurdle for me um i don't know how other people feel about that um but for me when i started saying like you know i think the biggest when i would have time stamped transcripts i really loved them but then i started like well how can i get that time stamping as i'm going through and then when i just let go of capturing verbatims um and just started saying okay just i just need to reference it um you know i started working more efficiently and i'm like oh wow i really like how this works um i really like you know being able to go through and like go tap not only just have to go through interview by interview but be able to go instead by task so like hey they started they did they you know we're going to cover this question i can go through all the questions of all my users at the exact same time with really with a little you know little effort and with ease and say okay this is the theme that i'm hearing and sometimes you know you get that um bias of of last heard or you know recency of the one the the interview that you just last conducted as the way that information is going this way you can actually with being knowing where your timestamps are you can actually listen to them in in as a group and then you don't have that recency bias because now you're looking at the data from a perspective of oh okay i've just listened to these all together as opposed to individually over time and i'm i'm sensing a different trend than maybe what i was sensing after i talked to the last user so that's another i think benefit of that that i've seen over time and i know you mentioned you know surveys as in i guess an alternative or potential you know add-on tool um it's what extent are you using some of these techniques in this more of a hybrid fashion um yeah so you know um in unmoderated testing i really like that because after you ask a question especially when you're doing that um that self interview where you're having the user give them they're walking through the questions that you've programmed as you know normally your program is the task well you just put a question to them and they're answering or demoing what they need to do afterwards you know you still in a lot of tools you can still ask that follow-up question in a lacquer likert scale so i will have them explain and demo and then maybe the follow-up that i have for them in the tool is a likert scale so that now i have a metricized response so they they gave me how they feel but then now i can compare that on a scale and i can communicate that metric to people who are more data driven as opposed to um a qual you know quality driven as far as what people are actually saying and more emotional versus you know more data versus emotional um this box of the world if you will so i try to do that as well um and again like i said um the couple of times that we've sent the makeup breakup letter as a survey um people i've had a couple users like so you just want me to write like a love letter or a makeup i said yeah have fun with it you know they've reached out to me afterwards and they're like okay you know and we just try to assure them that this is a survey but this is the data that we're getting and then we'll take that and we'll you know go through and look for word you know we'll do a word cloud we'll do look for phrase analyses of things that are common um you know how many people make up with the product versus break up their product um and yeah like i said it's a lot of fun for our interns as a methodology it's been a lot of fun for the users to use and then when you convey that to even you know executives they're like everybody has that connection to the idea of making up or breaking up with somebody we've all been in that relationship where we've either oh i'm sorry you said you're sorry i want to i want to continue this relationship or like no this isn't happening and so it really contextualizes it really quickly for all the people stakeholders users and researchers involved yeah indeed that's a great projective exercise it really does not just the participant but even you know the folks internally the in in receivers of the research see we have a question in have you done anything out of the norm to recruit appropriate participants during code um so um when i when i worked before i worked at stars i worked at a pearson and so a lot of times when we um um we're trying to recruit college students you know we did a lot of um you know um um social media um you know we would post like um give them uber rides like so a lot of times they don't have cars um and so we'd say like hey we'll give you a free uber um here and then anywhere else you want to go afterwards and then like uber gift card um that was really helpful for in person from from a remote standpoint um i've been and i've been working at stars during most of kovid um we've kind of you know we've been able to reach out to our users and give them like free um hey you know we we're this show is launching um we want you to watch it we'll give you a free um you know subscription to our service and you know you go through and utilize the service and you can watch you have a streaming platform free for three months and so that's been a really good incentive uh to get them involved um you know we have some swag that we've shared for some of our more popular shows that you know hey if you participate you get a free t-shirt or you know some type of something that they can um wear about their you know their favorite actor or their favorite show so that's been actually really nice that i didn't have it at the previous positions that i've had but um a lot of times it's just the incentive and the time constraint you know um and making sure that they can participate to get that and just following up on that a lot of a lot of people that i utilize you know when they realize you're going to pay them quickly and on time and as as promised and you're communicating with them what's expected of them um a lot of times you when you have when you have to go through the pool of users um it's easy to engage them again so and these these sort of coveted practices that you you talked about um how much do you think they'll be sticking around you know after back to normal happens i think it depends on the type of data that you want so you know before covid we use we use actually both services um like i said i worked for an education company and so we would do in-person um uh sessions like we would do design participate designs with users with our designers to kind of get some fresh ideas but then we would do usability testing remote usability testing because we wanted a good sampling of you know the college student that goes to a ivy league school in the northeast is not the college student that's going to a junior college in you know the middle of america and those are two very different experiences and so when you want to make sure that something's reaching all of those those demographics um you're going to want to do a wide variety of sampling across the us and that's when remote remote remote research just helps get get at that information and so if you can you know sometimes i'll even do a couple of local and then you know like if i have to do five people two in local and then three remote just to make sure that i'm getting and making sure that the end user especially if it's if it's across the us or across multiple countries and so i don't think remote is gonna go away in that context and then you know people you know if they have time and they can make a little extra money uh some of the services that we've used you know they they they get points for participating in things and so they'll they're willing to participate because at the end they're earning you know cash from the the service and then you know as they have time they're doing them and and so at you know i sometimes i get i'll have a i'll post a study and i don't get responses until 11 at night so the people that i'm targeting you know they're not sitting down to answer their tests and i don't work at 11 at night but my users that's when they're they have time to do it so i feel like um remote is here to stay even if once we get into in-person data and rebecca had a question about your makeup breakup exercise um have you used it outside of the americas or europe um specifically how how might it apply in other global markets um that's a good question um i feel like i said in us um in a european american um north american context it works really well um i in asia i feel like that is a little bit like they're because they tend to be very closed about the relationships that they do so i don't think that it would work very well there also um it work it doesn't really work with older people i've noticed it works more with um people who are still in that dating a realm so um um younger participants to you know 30s or 40s people that are they're really in the context of dating and that tends to taper off as people get older so that's why it's a really good technique to use with college students with our interns because it's that demographic tends to really respond to that just one last check really trickling questions i think that might be it all right um leslie thank you so much this is nice really good thank you and i said reach out to me my uh email is ella special gmail.com or on linkedin and you know if you want to follow up or talk to chat about interviews i'd love to do it with anybody that's interested so thanks thanks bill thanks for the opportunity