Operational Excellence and an Outcome-Driven Approach for UX Research

Hi everyone. It's the top of the hour, so it's time to get started. My name is Christa Reid. I am one of the senior project managers at Accelerant Research, and I'm very excited to introduce our next round of speakers. Um, some awesome researchers coming to us from Zillow. So, before we get started, there are just a few ground rules that I want to cover. We know that with being quarantined during this time, we're all having to leverage technology, which pretty much means we're at the mercy of technology. So, let's just keep that in mind. Try to be as respectful and considerate of one another and any background noises or interruptions that might occur. Um, I know in the Charlotte area, the wind has been pretty rough. So, if you're joining us virtually and have any of uh mother nature impacting your connection as well, I totally understand. Um, let's use this time to network, engage, and interact with one another. It's hard to connect in person because of the current times, but the great thing is that we do have technology and it's our resource to be able to stay connected. So, with that, the formalities are out of the way. Today we have uh Jenny, Lorenzo, and Lauren coming to us from Zillow. They're going to talk about operational excellence and outcomedriven approach for UX research. And with that, I'm excited to uh take the screen off from me and move it on over to them. And I'll be back later to help field some of the questions we get from all of our participants. So, thanks everyone for joining. And thank you Jenny, Lorenzo, and Lauren for your time today. Awesome. Well, thank you, Christa, Bill, Accelerant Research um for having us here today. We're really pumped to represent Zillow Group and share with you a framework that um we we built back in the fall of 2019 um and we hope to get a lot of value out of. So, just bear with me for a sec here. I'm going to start sharing my screen. Here we go. All right. Can everyone see the title slide? Perfect. Okay, so to start um wanted to introduce ourselves. So my name is Jenny Mangum and I am a senior UX researcher at a Zillow group and there's 22 of us on a big centralized team at Zillow and um I specifically work on a small set of that team that's focused on ops related things. We do a little bit of measurement moving forward um doing more research enablement type of things that we're calling ourselves working on tools, processes, frameworks, etc. to help all the other researchers on the team um you know be more effective and more efficient at their jobs. I'll hand it over to you Lauren. Hi, I'm Lauren Spore and I am a principal UX researcher. I've been at Zillow for about six years now and I actually started my career uh on the customer insight side. So I was number two on that team and see that grow to at um I think we're probably on that side about 20 as well. Uh and I have been on the UX research side for about two uh a little over two years now. And I am focused on uh primarily understanding sellers within the Zillow offer space. So that's our line of business that focuses on buying and selling homes directly with customers. Great. Um morning or good everyone. My name is Lorenzo and uh I too am on the Zillow research team. Um my coverage area is mainly the course search and so that's kind of the bread and butter of what um Zillow is and has been. Um essentially when you come to the site um I help design and research pretty much the web as well as the the apps. And hard to believe but I was actually the third researcher when I joined Zillow. There were only three um other researchers. we've grown to 22 which is really hard to believe but um I've been with the team for uh almost three years now and um yeah really looking forward to sharing our framework with you today. All right. So we have a ton of content we're going to cover today and we're going to start by talking more about um what UX research operational excellence is and the need for it. And um even though we're drawing attention to UX research, we're all part of a UX research team here. We really believe there's application here um for folks in market research, data science, really any discipline where you have to ask questions, go get those answers, and then share that with partners, stakeholders, clients, etc. So, we really believe there's widespread application for this. Um but we're going to start there and then we're going to um go in depth how we've tackled this problem, our approach to this um you know our approach to building this outcome focused framework and then talk more in depth about some of the applications that we've used it um so far, how we hope to use it in the future and then just some final thoughts key takeaways um and get into any questions we didn't address along the way. All right, so let's do it. What is UX research operational excellence? This is a pretty big question and it might um bring to mind a couple different things. Some of these are things like process, having efficient processes, having effective processes, things like rigor and high quality. When you think of rigor, you might think of doing methods by the book, having big sample sizes or the right sample sizes, things like cleaning your data, having really really high quality data, and of course being user centric. Um, you know, that's the bread and butter of what we do. We're user centric. So any way we we attempt to define operational excellence has to focus on user centric, customer centric, that type of thing. These are a lot of different words, a lot of different concepts. They kind of fit together if you squint really hard. Um, and you can take it down to that next level. Okay, so you have an individual. So you have a single um, practitioner, an individual UX researcher. what does it mean for them to produce high quality work. What does murderer look like for them? Then you bubble that up to a team. Um, you know, you have you're introducing some more complications here. How does that differ from individuals? What what makes a team operationally excellent versus an individual? You can bubble that even higher to an organization where you have multiple teams perhaps. Um, where you have new things to consider, um, different types of politics, all that good stuff. There's just a lot to consider when you're thinking about operational excellence. And so for a little bit more context here, um what we were tasked with back in the fall of 2019 was trying to unpack how our UX research discipline at Zillow Group could be operationally excellent. And at the time um we were divided um or organized into three different teams um you know according to three different lines of businesses. So we had our Zillow team which had its own set of stakeholders and partners across design and product a different line of business premier agent same thing different different partners that they were working with different stakeholders as well as Trullia which you might also be familiar with there and so at the time we had these three teams and we're using similar tools somewhat similar processes despite our different partners and whatnot um you know wouldn't it make sense for us to come together and talk about and align on what it means to be operationally Excellent. So that we're not repeating the same thing three different times. We're not putting energy and effort into building out these processes that perhaps other folks are already doing within Zillow group. So that was our big question is what does being operationally excellent mean for us as a discipline? And yeah, we had to go figure that out. So how do we do that? Well, it all started with an outcome driven approach as our foundation. Um, and you you might ask, what does that even mean? What is an outcome driven approach? And it's pretty simple really. Um, what it really boils down to is articulating, you know, users end state, their user needs, and what success looks like for getting there. You may look at that and say, okay, like that's a ton of buzzwords. What is going on here? And so, we're going to break that down a little bit more. So, really, it's just about solving for the most important user problems or jobs. This is what we do on the product side is uh we do a ton of work to articulate and figure out what is that most important need or unmet need whether whether defined or undefined. Um you know figure that out and then we go solve for that. And you can also you know think about this in terms of jobs. I know there's a ton of literature around jobs to be done and purists with that theory might kind of ah like scoff at at mixing that with outcome driven but they're so similar and for you know sake of simplicity you can just think about this as solving for the most important things people do their jobs and breaking that down a little bit further you have a who you have a person doing something which is the job and some measure of success okay so this is the type of you know thinking and approach that we were already doing on the product side. Our director of New York's research, Andrew Mayor, um, you know, really really um, you know, worked hard to to bring us up off the ground and bring it over to Zillow and really um, evangelize this way of thinking across the organization. And so on the left here um, you see, um, an outcome movers find homes based on state of criteria. You have that who which is the mover in this case and you have that what that job which is find homes based on data criteria and then you can add on that you know what does it mean to do well piece to it in this case accurately and efficiently and who's doing it well us Zillow we're doing this so we are that tool that solution that's helping a mover find homes based on the criteria that they state and so we have over a hundred of these outcomes that we've built up um across selling buying renting being a landlord, being an agent. We've organized these into maps that we have for each of these key personas that chronologically list the most important outcomes all these users are trying to do. And so the team, you know, Lauren Lorenzo built these up from scratch and um you we're already using these as you know, throughout all phases of our research. So why couldn't we take this approach and apply it to ourselves as researchers? And that's exactly what we did. And so what does that look like? Well, if we go back to that simple, you know, syntax of an outcome, you have the who, you have the job, you have some measure of success, and this is all building up to designing products that are operationally excellent. And so in this case, we have the who. We're looking at ourselves. We're looking at our own discipline. We're looking at UX researchers. That's our who. We have our end goal in mind. We want to figure out what is operational excellence for our discipline. We just need to fill in those middle parts. Sounds really simple. Um, you know, we should know our jobs. We should know what we're doing and we should know what it means to do that well. But breaking this down in this way um really helps give us just a place to start and to to start doing this. And so taking a step back here, why bother doing all this? Why bother taking this outcome driven approach applying onto ourselves? Seems like a lot of work. Don't really know if it's necessary. Like what are we doing here? Well, there's a couple different benefits. And one is that this is just inherently solution agnostic, which um is so exciting. So you could think about a researcher's jobs that and and build these outcomes that are totally method agnostic. So whether you're doing a card sort, whether you're doing a giant survey, whatever you're doing, you're still trying to answer some sort of question regardless of that method. You can also think about it just being totally or agnostic. If we're doing it in Trullia, if we're doing it in Zillow, if we're doing it in Premier Agent, in theory, we're still trying to complete the same core job so it could apply across all these different teams and organizations, which is pretty exciting. Had a lot of potential to scale. Another reason is that outcomes are scoped and measurable. So in having um you know a collection of these units, we could actually analyze, oh my gosh, we could actually gauge our success, measure our progress, and do so many cool things with that, which uh teaser we actually did. Um there's more to come on that later. And then the third piece here is that, you know, like we mentioned, we're already doing this on the product side. was familiar to a lot of the research practitioners on our team. A lot of the IC's management are already pretty well steeped in this outcome driven thinking and within Zillow we were calling these experience outcomes. So you see the short form exos there. Um that's that's just the lingo that we were calling these on the product side. So with this in mind um this already being sort of a share team foundation there's really minimal onboarding and minimal work that we would need to do to get folks um you know thinking this way. So tons of benefits. So we plow forward. So thanks Jenny. So our next step was to actually build this outcome driven map. Uh and so we kind of had a a highle um process of what we did to get there. So we're going to walk you through that. So the first thing is we actually had a core team of lead individual contributors uh on across our different UX research groups come together and think about how we could create a solution for what it means to be operationally excellent. So uh as we were talking about different uh potential solutions or frameworks, we kind of had this moment of oh my gosh, what if we use the experience outcome framework? And I remember us kind of sitting in a collective room and we just went like, you know, oh my gosh, this is mind-blowing. What if we did it to ourselves? We've been doing it to customers, you know, we've been applying this to to sellers and buyers and agents. And then we also recently started um doing this with some of our internal operators to better understand, you know, their their needs and their goals of their jobs so we could apply um you know, solutions for them. And so it was really again this slight bold moment where we realized we could turn turn it completely on its head and apply it toward ourselves and our jobs. So, in terms of the process, obviously, you know, we wanted to review it with our management team. They were super jazzed, excited, kind of gave us the big thumbs up. Uh, and actually Jenny was the person to say, you know what, I've got this. Like, I'm super I'm super jazzed about this. Let me take on and develop a draft framework for us to start from. And so, that was really beneficial because we weren't starting from scratch, but we knew that um we wanted to get a lot of the team involvement as well. So, we used an upcoming uh UX research off-site. and and we say offsite, but it was actually onsite, but now that's seems like a dream where we could come together in a room. So the we were kind of pulling together photos from a year ago and we're like, remember when we were together? Remember when we could share food? Like it just feels so foreign. But um yeah, it was just a year ago where we came together. Uh next slide I'll go through uh kind of our core steps. So the first one is we wanted uh as a group we just kind of sat down again together in that room but we could do it um virtually as well if we were uh creating it uh this year but we really wanted to align on the highle phases of our uh framework. So things like framing the research, constructing research design execution analyzing and ultimately inspiring and uh you know getting others to act on on our our findings is uh the high level things that we wanted to get uh alignment on and then really is there anything missing. So we wanted to make sure that we didn't have any big glaring gaps. And then we actually created a team activity where we had in independent brainstorming via post-its. So you know we're always a big fan in design sprints or various things like that of using post-its just to kind of have individual thought uh and and coming together. And really this was breaking down you know when you think of your role your day-to-day either currently or in uh times past what are common common jobs or tasks that you as a researcher go through. So, um, when you're thinking about framing the research problem, you know, what are different things that you do? And so, we really wanted to use this as a way to get that comprehensive, uh, and and, um, robust, you know, feedback across all of our, uh, both individual contributors and managers when we were thinking about this. And this slide really just kind of encapsulates uh, the process. So, in the post-its, we asked what are you doing and why are you doing it? So some examples um for the first what is well you know I've got to reach out to initial stakeholders and additional stakeholders to gather my questions. Um why well I want to make sure that my research is broadly applicable. So we just again applied that um just similar thought all the way through the different phases. Once we had that, we actually did a different uh team activity where we broke uh folks out into teams of two and each team of two focused on one specific phase. So, one uh group got frame, one group got construct. Um you can see Lorenzo there pointing he's really working uh deeply with Antonio uh on I think it's probably is it frame? looks like an earlier one, but we really used um the full uh you know room to the advantage to kind of put um things together to start affinitizing those postits and grouping them under um phases and subfases. Uh and once we did that, we were able to kind of recheck again Jenny's initial draft to say, "Okay, this is really great. It looks like we've got a lot of support for, you know, this specific job or oh, you know what, I think we missed this component, so we might um reframe this job um just slightly to incorporate that new feedback. And then the last one, we actually had each group of two uh present to the team and then folks took individual notes and then we shared uh those additional questions, comments. We kind of just like pushed and stretched and pulled a little bit on it. And ultimately uh each uh group had one person responsible for kind of aligning that feedback. And then um we had our core team uh inclusive of Jenny and Lorenzo and I and some other folks uh really figure out um how to uh finalize this and make sure that it is um where we wanted to kind of land off. I do see one question come up. Uh the jobs to be done framework is absolutely an input for this. Um I would say we use p primarily um the experience outcome framework that we had used on the product side but definitely it is born out of um jobs to be done um as one input for sure. I'll hand it back over to Jenny to introduce um what we came up with. So now we're going to walk you through um the framework piece by piece that we built. And um a great place to start with this is in that execute phase like this is the core of what we do as researchers if you're going to start out say as a junior IC you know this is this is the place that that you go you're you're collecting data you're collecting whatever is the necessary data that you need to answer your question and so the two outcomes that we have here on your collect data are capture behaviors so things like observations or events and you can think about this in so many different ways. Um whether you're doing observation in usability study, observation, um doing some ethnography observation, um be collecting events of the data scientist, etc. So that's one flavor of it as well as all that attitudinal data. So um you know things like self-report, whether it's through a survey, a site intercept, um in an interview, a diad, etc., etc. All all the tons of different methods can help you tackle this. But at the core when you're executing research, you're capturing behavior and you're capturing attitudes. And from here you can start building this outwards. So all right, you've executed and collected some data. Now what uh you can analyze it and there's a bunch of different phases involved here. Um you know the first step would be assessing that quality of the data um through through the outcomes that we have listed there. And then you're extracting findings from that collected data. And to do so, you're organizing it, identifying patterns. You're mining the data for facts and findings. And then you're capturing those findings and insights. And again, because this is a solution agnostic um you know, following the similar jobs to be done uh framework that Lauren mentioned there, we're not telling you or prescribing how. It's just that you know, the act of capturing those findings and insights, it's identified as one of those outcomes. and then moving down the line and determining the implications for those findings. So now that you have those findings, now what do you actually answer that original research question? Um how do you you know synthesize those findings with all the existing knowledge that you might have in that space? Aligning those findings within the broader business context as well as articulating actionable next steps. So it's one thing to just say what happened, but it's another step to say okay now what? So what uh with those findings and then you can bolt on that phase that comes before execute. Um and this is what we're calling that construct phase. And you might think of this as designing an approach and whatnot, but design is just a murky term especially as as you work with so many design partners that we landed on construct. And so as part of that initial construct phase that comes before actually executing the work, you're doing a bunch of different things. So you're designing that approach to answering the question which involves tons of different steps. So determining what methodology could be used to actually answer that research question, determining what materials you need to carry out the research. This could be uh writing survey questions, building a discussion guide, um you know getting your hands on a prototype, whether you build it yourself or need to work with partners to get that all those steps to just determine those materials and then actually producing them. So doing the work to to make those come to life once you identify them. And then of course recruiting the right people, the right participants to ensure that they are representative of your target audience, whatever that might be. And then another piece of construct that follows that is setting up uh the research activities, whatever those might be. So um that could involve preparing the research participants for data collection. And depending on what type of method or what type of work you're doing, that could be so many different things. um providing context um you know hey show up here at this particular time here's a video link providing homework priming them in some way so many different ways that that you could do that um to then preparing uh your partners stakeholders clients etc for that data collection and that could be things like rules of engagement um again show up here at this particular time here's a video link etc and then two more outcomes here is preparing the research environment for data collection nowadays. That could be just closing the tabs in your browser. It could be um you know ensuring tools are ready etc etc. And then um the last one that we have there is preparing the structure to collect data. What this is really doing is setting you up for execution and analysis to try to expedite that. So so many different ways you can do that. again whether it's building a spreadsheet ahead of time um making a template in a notebook etc. And so we have these three phases. Now we can still bolt on two more at the end. Um you know once you've collected your data, analyzed it ultimately you want to inspire. So you want to drive action with partners and stakeholders. And we have two outcomes listed here. Um which is communicate findings, insights and implications to partners and stakeholders which is a lot again. you can take you take a finding build up to an insight and then distill next steps and implications for them. So that's that first outcome and then the second one is helping partners implement those insights and um as you probably noticed at this point you know these outcomes might not all be necessary for you tackling a given research problem. They might not all be necessary for you in your role and or at your level in your career etc. Um but nevertheless, if you were to do it all, this help partners implement insights is a key piece. And then lastly, frame. So this is that first step that comes before all the rest where um you need to figure out the research need. It's a pretty big important step. It's that very first step. And in doing so, there could be a bunch of different outcomes involved here. So determining partners and stakeholders um determining the goals and questions that they might have um determining what decisions uh will be informed by research and you know what what happens many times uh for myself I know others as well is you might you know start down that path of determining partners determining goals and questions and then determine that it's not going to impact a decision. So then you stop right there and that that stops that process for you. Um anyways, like moving down that list, uh triangulating articulated needs with existing knowledge. That's a big one. So, you know, you have these goals and questions coming in. Well, what do you already know when you when you hook up all those things together? When you synthesize all that, now what? Um what's the question? What's the goal? How does that change? And then lastly, under the figure out the research need um determine whether or not to move forward with new research. So, I mentioned that earlier. um you know maybe it's a time where you pause and you just you don't move forward in the process because there's not a need um to to actually conduct and execute the research. And then so once you figure out that need in general then you can move on to defining the questions to be answered. And we have three outcomes listed here, which is trans um translating those articulated needs and questions that partners and stakeholders have into research questions that you can go tackle and think about what methods you would use to do so. And then there's a prioritization element to that. Many times we're not going to go tackle the dozens and dozens of questions that are top of mind for folks. So we have to prioritize those and um you know distill down which ones we can actually address in the upcoming research. And then lastly here, identifying constraints. Um so a lot of times um you know there's so many things that pop up. We're not just working in a bubble. Um there's timing, uh budget, etc., etc. All these things that we need to balance and juggle um that that really inform how we then construct our approach. And so this is a full end toend research process. It's a lot. And um you know it when we were building this, we wanted to represent the breadth of what we could do if we were to truly do a project from end to end. And again, these might not all be relevant and that's okay. You might you might do a couple of these. Um you might start framing, then you hand it off to someone else, a different tool, a different person, a different agency, etc. to do the constructing, the executing, etc. Um that's kind of the beauty of it being solution agnostic is we're not saying how you do it rather these are all the steps that you do. All right. But that's not it because then we have these wonderful things uh that we call key enablers. And so as you um as you were getting the super speed read of our framework map um a couple things might have popped up of all these additional factors that are really important that or factors I call outcomes that are really important as you're going through and and doing your research process that just aren't really addressed there. These are things that are horizontal and in theory like show up at every single phase. These are things that are catalysts to completing those outcomes. and we group them into a bunch of different buckets here um in terms of organizational influence. So those are the relationships that you have with partners and stakeholders. Um the confidence that your stakeholders and partners have in you um whether it's as a researcher or as a team um and then you know your ability to align and drive consensus. That's a skill that's an outcome um that that takes time to to build up and and time to build. So we have organizational, we have technical competence, same type of deal. Um practical skills. So things just like coordinating time, reviewing work, um communicating intentionally um and having a sense of work life balance. That's an outcome. We really wanted it on here. It's important um especially um you know to and our team. And then lastly, a couple more here. Uh we have adhering to regulations and standards. that didn't show up in that outcome map. I think it's really important that we do that um to be to do ethical and and you know follow the our company policies as as we do our work um you know ensuring that others uh you know can find insights. So this is talking about more more um ops related things. So maybe you go and do that end to end uh you know research but now what what if people can't find it? um you know there's there's all these little hops type things that are pretty important here. And so that is it. That is um our research at work UX researcher framework. Um now I'm going to pass it over to you uh Lorenzo to talk more about how we've used this framework. We we did see a quick question on there and we wanted to note um it was about collaboration and um you know if we have any ideas about that and I think we have a a section later on about collaboration which we'll fold into. So no worries we definitely saw the question uh we just didn't want to break up um Jenny's presentation but we'll be sure to address it. Great. So we've gone through this process. We've talked about the framework we've created. You know, we've we've reflected on, you know, what we do as a discipline, the the different phases, the different activities, the different tasks, the different jobs, if you will. And I'm sure many of you are wondering now, you know, what's the purpose? Like why why are we doing this? What was this activity really for? And so I want to dive into some of the applications that we have um come about from this exercise and really dive into some of the more specifics about like how we're using this and how we're continuing to use this. Now keep in mind that many of these are works in progress. Um it's been a year in the making but uh you know it's still being evolved and will continue to be evolved but um some of the things that we've done really talk about you know how do we activate this? How are we measuring you know quality? We also want to talk a little bit about how are we using this in our hiring practices and you know bringing in candidates and uh you know formulating questions that really ladder up and align to these different phases and these different activities that we've identified we as a discipline you know should be providing for our clients and our customers. Um Lauren's going to talk a little bit more about how we are using this framework to really assess our overall um team's performance and then we're going to talk a little bit about future applications. Now when we think about um you know how we're planning to use this for measuring quality, it's really about like what is research quality look like? And so when we came out of the um the activity, you know, we'd been in a room for one day, we had a couple weeks followup to really identify, you know, what this framework would look like. We wanted to know, you know, what is the output of this? What is what is quality? And really the questions came about if you go to the previous slide. Can you go to the previous slide, J? Sorry, one more. The goals were really around, you know, how do we develop an approach and really set a bar for what research quality is in our team. And some questions that were coming out from management and from other team members were really how do we, you know, create a quality bar for what we identify as quality research? How do we ensure that we're maintaining this as we go throughout our practice? As we go throughout these steps and activities that we've identified, and really how do we work to improve the overall quality of our team? Now the first part of this process is really trying to identify like what does research quality look like? Um you know for each one of these phases where we've got you know all the way from the beginning where we're meeting with team members we're starting to engage with them gather questions all the way to the very last phase where we're really looking at that aspect of you know activating those findings those insights and working with the team to really you know get those implemented. And so a follow-up to this was a series of workshops where we brought our team together to really gather input on what high quality looks like. Now being a key component in the development of this framework, we really wanted to maintain that level of collaboration. And this was super critical to get team alignment and also team input on like what quality is. Next slide. Um now as part of this activity we also wanted to determine what does it mean to be successful each of these phases in our framework. Now if you remember Jenny presenting a slide on you've got the the activity itself for each one of the phases. This example here is when we start to design the approach to answering the question. What we're lacking for each one of these is really how do we know we're successful? And for that we held a workshop. We did a bunch of posted affinity exercises, those type of things to really get at, you know, our level. When when are we going to know we're successful? And so if you think about when you're designing the approach to answering a question, success is often seen as when we're confident we've selected the most appropriate method or for example, when we've done due diligence, we've reviewed previous research and consulted with experts and leads on the team. Next slide. Um, as part of the next step for this, we wanted to really create, gather, organize, and really take inventory of, you know, what exists within our current team and even beyond. You know, many of the um members of our team come from diverse backgrounds. They've worked in different companies, different industries, but they all bring a wealth of knowledge. And so we really leverage that to, you know, start collecting like what activities exist out there, what templates, what are some examples that the team can provide that really exemplify high quality. And we started to get those together. We started to gather them. We started to take inventory of what we had and what we didn't have. We ultimately sought to kind of start filling in the blanks to really understand, for example, um, you know, when you're constructing your research. One example was to collect um artifacts that we as a team had for like study plans for different types of studies. You know, we all have these kind of hidden away in our um you know, our own personal drives, but the effort was really to try to collect these to try to gather them and then ultimately as the next slide displays, we wanted to build a way that we could share these out with the rest of the team. And as you'll see as part of the next process, this involved really um placing these in in an area where they were easily accessible and also bringing about awareness that we do have these practices, these guidelines, these templates, these examples, you know, that we've created over the years and that, you know, as researchers, we never start from ground zero. You know, the wheels already been invented. So why not leverage that and evolve to the next level? Um the next step with this was really to ensure that we have processes in place to set standards and expectations with the team and maintain high quality over time. Now this is still a work in progress but some initial steps we've made to start activating the framework is really um building processes. One example would be to ensure that you know all stud study plans we have are reviewed with peers prior to launch and this isn't a a heavy set rule. It can be more organic in peer meetings but we should always start with senior manager level or one level above just to start to maintain you know some level of quality as far as what we are producing um within our practice and I want to talk a little bit about the output of this and the next steps. Um so as I mentioned a couple times here this is evolving. We've uh we've only started to activate this, but our efforts are really um targeted towards continuing the activate the framework and part of that's going to be identifying other processes that can serve to facilitate this high quality and also um just exploring and continuing to explore other ways in which we can maintain quality you know throughout the activities that our teams are performing. Now another uh part of this effort um that came out this is super important um was we were approached by our managers to um to start thinking about you know how can we use this for interviewing candidates you know bringing in candidates understandably the problem is we've got as I mentioned a very diverse group of people each we were starting to notice there was a lot of inconsistency within the questions that were asked um invariably that can lead to you know complications when you're evaluating a candidate. And so the questions here were really how can we be more consistent in the questions we're asking? You know, how can we be more equitable, objective in how we're evaluating candidates as we're bringing them through? Then ultimately, how can we make this process more efficient so that we're not having to, you know, dig through files and pull together our own question sets and really just, you know, make it as effortless as possible. Um now with that in mind um you know we we wanted to try start identifying success here and really that's you know to make it easy for our hiring managers to you know you're hiring for a certain role what's a question set that you want to use that and then ultimately how can those align back to the framework that we've created. And to really start to accomplish this, we leaned heavily on the framework that we designed. Um, now within the framework, as you've noticed, this identifies things that we should be providing for our clients, for our teams, for our customers. And so, we start wanted to start gathering and synthesizing the most common questions we asked during interviews. This was an effort that really took you know bringing all these different templates these different guidelines these different sets of questions we had from all our different managers and also we looked to uh forums and other sources um you know within the community to try to see what are some common questions and ultimately how do they identify you know people who are good within the framing within construct within execute and all of these different phases and activities that we have deemed to be important as a discipline Our next step was really to start aligning these with each phase to make sure that we had coverage within each one of these different phases and if we didn't to start beefing up or understand if we needed to beef up and have questions for each one of the phases that we didn't. And ultimately we started to look at you know if we have a set of questions how can we start to understand leveling for each question you know how can we ask or do we need to ask a question for someone who's entry level versus someone who may be senior or principal and if so what do those questions look like and how do they align to the framework and we worked with our team leadership to start vetting questions aligning on leveling and this is one we've started about a month ago and we will be continuing to activate um in our upcoming candidates candidate interviews. Now, the output for this one is super critical. We want to bring in the highest level of quality candidates that we can that really meet the standards that we've determined to be important as researchers. And we're going to continue to start using the question bank as we interview candidates and ultimately evolving it through usage. Um, this will include the addition and deletion of questions. we'll start revi refining our criteria for leveling and just really determine what works and doesn't work. So now I'll pass it off to Lauren to talk a little bit more about another way that we've been using the um framework. Yeah. So we wanted to use this to develop a better understanding of where we're at collectively with our skills on the team and uh we uh kind of brainstormed all the different ways that we could evaluate oursel. that's always kind of fun to do. Not even been just research on research. It's research on researchers. Uh and uh we had a couple of different uh leadership frameworks and and things uh used at Zillow and as well as um externally where we developed um uh this uh three kind of components of evaluation. So the first one is competence. So on a scale of one to five uh rating your own uh level of ability. So you can see an example there um is you know one very low to none all the way up to very high again on uh competence is the first one u motivation is the motivation for you to grow in your own confidence. So how excited are you um how much of the kind of intrinsic uh inspiration is is coming from you that to say you know I'm not there yet but I I'm really excited to do that. And then the last level is just about a confidence measure. So confidence you can grow given your current opportunities. And uh that was a little bit open-ended and we kind of did that intentionally. So whether it was your specific role in the team or your um you know level of ability uh given you felt like where you were at versus where you wanted to be um just general sense of confidence and we wanted to share kind of high level uh findings uh for for you here. Um and we shared uh ultimately we shared this um with the managers just to kind of give them a sense of of where the team is at and we're still kind of working through potential opportunities which we'll talk about in the next section. But some high level takeaways in terms of competency uh we really measured ourselves highest uh in terms of qualitative research. Uh we are a group uh predominantly uh with qualitative methodological backgrounds um and particularly in the setup phases. So when Jenny was talking about uh you know people coming in and feeling strongest in the um execute and construct um phases we see that kind of reflected in the data. Uh we measured ourselves lowest in quantitative areas particularly in assessing the qualitative data. So that um is identified as a potential gap in our team uh again with open questions to managers as to the solutions for improving on that. You know is that okay? Is that something that we feel that um you know another team data science or um customer insights would maybe take on that type of methodological approaches or is that something that we want to um kind of fill the gap on on our team because we want to be more um robust and comprehensive from a motivation uh perspective. We are motivated to increase that organizational in influence and also just grow our general practical skills. Hello. Um it's interesting that we um the improvement in areas where we rate ourselves as competent um plus setting up uh quant research is where um we definitely want to improve and our motivation or motivated to improve. And then the last piece is confidence. Um we are most confident we can grow in areas where we're motivated. So um you know kind of seeing those synergies there some correlation there at least slightly uh including that organizational influence and then also driving action with uh stakeholders and partners uh and lowest is areas where we do feel least confident um again particularly in the quant area. So that was identified as the biggest gap on our team, but maybe on other teams you would be stronger in quant and a gap might be in qualitative areas. And then um we kind of have been thinking as well about future applications. So kind of ripping off of that identifying skills on the team. U we felt that you could also use that to identify skill masters and potentially set up kind of a mentoring opportunity where people can collectively know um you know oh I really want to go to Lorenzo for for this um area or oh I really want to go to Jenny uh for this area and again setting up those opportunities for learning uh from uh folks that have really honed their skills and are seen as experts uh on the team. Another area is to help set goals and measure progress against those goals. So if you are you know conducting your own analysis on yourself and you're kind of reviewing that with your manager is that you know something that can be used to help to say I really want to grow in this area and what are different um you know opportunities or uh different um you know ways that I can start to grow and build on that success criteria for that specific um job. Then transition it over to key learnings. uh what worked. So what we uh kind of took away from when we were kind of going through and using this as an opportunity to say, you know, yeah, what did we think was um you know, really particularly um successful about the way that um you know, we worked through it initially and then also um as we created, you know, specific or additional followup from this. Um we really felt that involving the team up front helped create a sense of shared ownership, but not starting from scratch. So we would have definitely had a longer we would we wouldn't have had time in the full site just to to come and even you know us as a core team kind of spun our wheels a little bit to kind of get you know that high level um you know theoretical to a direct application. So again kudos to Jenny for really taking that on and and giving us something to work from that really definitely helped and identifying potential use cases up front. we actually saw the most momentum on things where we had identified well gosh how could we use this before we actually created and finalized the framework. So that really gave us a springboard to say okay now that we have this framework let's immediately you know have Lorenzo take on you know the quality piece and Lauren you take on this piece and Jenny you take on this piece and that really again allowed us to um kind of build that momentum um throughout the whole process. Uh and the last one is just finding ways to incorporate with other um frameworks or or well-known things um that are comfortable and and known with the team that just helped us kind of better understand how to incorporate it. So when Lorenzo was talking about um the question bank, we really wanted to try and code the questions not only by um the outcome framework uh for product research but then also um different things like we have focus skills and broad skills. And so we could say, well, this question not only addresses, you know, this framing piece of it, it also addresses this core skill. So again, you can kind of squint and see how um there are some similarities across other frameworks. And if we move to the next slide here, as always, there's um opportunity for improvement. Um with anything that we um that we do, we always like to look and reflect back and say, how could we have done this better? And one of the things that we identified was really holding the core team accountable. And um this is really about being deliberate and persistent with milestones with deliverables. It's a little process heavy, but it's something that we we identified that we didn't quite have when we um started to implement this. But moving forward, we're definitely going to look at, you know, just having persistence with um with our goals and you know, where we're at within the process as mentioned um you know, in the um activation part is, you know, continuing to evolve and really having a plan to come up with um additional uses. We've explored some, but in the future, what we're looking to is really to get back with the team and start to explore other ways that we might activate the framework itself. Last but not least is um ensuring that we are maintaining and growing our shared knowledge. Um, understandably within our company as well as others, we have reorgs, we have new team members joining, we have team members who may not be familiar with this process and really just coming about process in a way to make sure that we're maintaining and continuing to implement and activate um, our framework. All right. And one other thing we're just really really excited about is some of the other opportunities um for how we could uh you know use this framework with some of the other research disciplines at Zillow. And so a couple different ways uh we we hope to use this is just to better educate our colleagues on what we do and what we don't do. Um, you know, it goes a long way being able to articulate and and talk about our process and um, even then applying some of the tools that we use to do that process to help people um, you know, get us involved earlier, get us involved more often in a lot of the in many of the initiatives that that we work with partners on. And then on the flip side, we could use this to better understand what other related disciplines do, market research, um you know, data science, um you know, behavioral sciences. Uh we have this this foundation, this framework that we that we can talk about our process. So we can use that to then um you know, better understand others processes and how those could fit together um or not. And that that leads into this last one here is just helping Zillow um as an organization better understand how to leverage each of the research related disciplines. Um you know given there's so many of us which is fantastic but you know how do we work together? How do we work in isolation? Um you just being able to facilitate operational excellence at a larger scale is really really exciting. um you know to open up more opportunities for us to work together to tackle questions that might even be similar that just require you know slightly different approach or a mixed method approach um you know I'm hopeful uh that we can really um activate on this piece specifically. All right. So, just some final thoughts here uh before before we wrap up is that you know the exercise that we went through to build this map to build this framework and ultimately just build a shared understanding of what we do what our team does what the UX research discipline at Zillow does is super super valuable in of itself. Um and you know a couple reasons is that it can just break down misconceptions about what we do. Um it can it can you know give you space to have those convers conversations illuminate similarities even though you know at the time um our our discipline was across a whole bunch of different lines of businesses doing a lot of the same stuff tackling a lot of the same questions. Um we have a lot of similarities and you know doing this type of activity type of exercise really illuminated that. And then lastly, um, you know, all those applications that we talked about, this really sets a foundation to do those things, to have that shared language and these these, you know, units of analysis that we can then layer all these things onto. So, we're really excited about how durable this type of this type of work is, this framework is, and all the things that it sets us up to do moving forward. And so, you know, there's no one way to build it. You know, true to the outcome driven approach, jobs to be done. How you build it totally up to you. Um but we really encourage you um to you know do the work to to build that understanding and then you know put it to use in whatever way um is valuable to you. All right. Well, thank you so much. Uh thanks for you know having us here today. Um you know listening to us um you know asking questions and if you have any more really happy to discuss and answer them with us again. We do have that first one from uh Susan and she was asking I think this was kind of um upfront when we were talking about the off-site and the creation of this but question um to you Lorenzo and Jenny and I'll give a stab at it as well is you know how have we approached collaboration and workshopping um as a team during um this kind of working from home experience and what have you seen maybe work and and um maybe need to be adapted um as we shift to a different different world. Yeah. So, I can uh I can answer that one or at least provide some uh perspective on that. Um yeah, I mean the work the work doesn't stop. Um you know, we've continued to have to um push forward through these uh you know, these trying times and um we have uh we've definitely done a few different collaborative efforts um that come to mind. And um those mean I' I've set in on a few different design sprints which are typically held in person and uh you know initially when when covid hit we we saw these as challenge and you know particularly being difficult as these are typically held within a a single room we typically get in a conference room we'll you know have 20 to 30 people and and literally collaborate for you know a week straight and um I will say that it's been a work in progess ress and it has been an overall um evolving process. Um some of the initial ones we had were you know trying to reflect exactly what we're we were doing in a room virtually. we saw and identified some things that worked, some things that didn't work. But I can say is, you know, this last sprint um fairly successful in some of the processes and methods that we've we took from those learning activities. Um you know, specifically being the usage of collaborative tools that allow us to, you know, affinitize and and post notes as we would in person. One tool we use um you know for that specifically is Mural which has been fairly effective um and you know somewhat of a learning curve for people to start using that and adapt to it but super super helpful. I think another thing that's been particularly helpful is really um dividing up what would have been you know complete collaboration and that's you know everyone in a room all together for every one of those days for you know 9 to5. We've had particular success dividing it up into shorter time frames. And so, you know, we'll have three to four hour sessions together. We'll divide that up with breakout rooms, which um thankfully uh Zoom is very capable of handling. And uh those allow us to really kind of, you know, get with smaller groups rather than have, you know, immersive time with everyone. And then we've also had success dividing up some of the activities to more kind of heads down individual particularly in some of the ideation phases where you really just need to take some time away you know really get your thoughts together and kind of focus and those are just a few of the things that I think have been fairly successful that I've seen Jenny more. Yeah, from my perspective, I think some of the um the early things were really just a learning process and kind of saying what's working, what's not working. And one of the things that um we we kind of understood is sometimes there are applications like Mural that work really well when you've got designers who are well versed in that versus if you've got a lot of product managers or people coming in. It's like just use Google Docs. Just create a space where you ask questions and people just plunk in together. We also have created some uh Google slides where you have some um of the you know slides that's actually kind of fixed and you can't move. And then I've had a designer and she just copied what looked like postits and she just copied 60 one over the other. So everyone was dragging and dropping and just it looked like magic but it she just copied and pasted 60 you know things over each other with text boxes. You just add it and then organize. So, I would say um there's another question here about what te techniques work even better now that everyone's in a virtual environment. I would say um we had a lot of folks in our Seattle office, but then we also have a lot of counterparts in San Francisco and increasingly in different areas. And so when if if you were not in that room, it was really hard to share a voice and you're kind of trying to have this one camera. You're kind of saying, "Okay, we're whiteboarding over here," but you can't really see it. And so I would say um it's kind of equalized that playing field in terms of everyone's remote and everyone's coming from that same either advantage or disadvantage depending upon your perspective and then also not that just the loudest voice in the room um can kind of control things. So I feel like that you know sharing of perspectives is a little bit more democratized. Jenny, do you have anything else you wanted to add? Nope, totally nailed it. Couldn't agree more. So, I'm not sure if we have any other questions. We do have a couple minutes left, but I will we'll hand it back over to you, Christa, if you want to take it from here. Thank you so much. Um, so I think let's see, I saw another popup. We might have just gotten one more question. Oh, no, that was a So, the first um question that we received, they just responded to say thank you. So, on behalf of those who asked questions, thank you. On behalf of Accelerant Research, thank you to Jenny, Lorenzo, and Lauren. This was a great presentation. Um, I really enjoyed seeing the colorfulness of the post-it notes and that image that you shared. Um, so we really appreciate your time. We appreciate you breaking down each step throughout the framework and the process. And I know that I have some great ideas that I can take back and share amongst our team. I'm sure the other participants do as well. So again, thanks. We really appreciate it. And um I hope everyone who was able to join for this presentation will hang on. We have another presenter joining and another team member from Accelerant Research who's going to introduce our next speaker as well. Thanks so much, Jenny, Lauren, and Lorenzo.