Using Card Sorts to Unlock Cognitive Models

Hello everyone and welcome to the top of the hour. Thank you for sticking with us today. My name is Moren Valentine and I have been with Accelerate Research for about a year and a half as a quantitative specialist after finishing my PhD at Cornell in 2018 in animal science and excited to be in the world of insights and market research. So, thank you for sticking with us and for joining us for this October Accelerate Research virtual insights conference. just to go over some of our little ground rules. Please just remember we're at the mercy of tech. Everybody's joining in from their respective locations. So, please be respectful, have patience with us, and use this as an opportunity to ask questions, network, engage with people. We have some interesting people to speak with today. So, take advantage of that opportunity. And today we are going to be hearing from Katherine Matzo. She is a senior product researcher at Lowe's Home Improvement. And I know I'm very interested to hear what she has to say about using card sorts to unlock cognitive models. Katherine, are you with us? There you are. Hello. Welcome. Hi. Great. Looking forward to getting started. Um yeah. Okay. Well, I will stop sharing and you can start sharing. Excellent. You should be able to go now. Mhm. All right. Can you see my screen? There it is. Yeah. Awesome. I'm gonna try to do this uh with full screen. Hopefully that won't mess everything up, but if it does, Meen, you'll just uh interject and let me know. Yep. Very good. Since I can't really see anything anymore other than my screen. Perfect. It looks good from my side. All right. Awesome. Um, so what I want to talk about today is a methodology that if you're in the UX space, you're probably already familiar with card sorts. It's something tactical that has a really specific purpose within UX. Um, but I want to show you how to get a little bit more out of it. And I don't expect anybody to leave here today after an hour knowing exactly how to do this technique that I'm talking about. But I do hope you'll leave wanting to dig deeper into that toolkit that you already have. Um, and while we can't be fully interactive here, I I am excited about the previous speakers and and attempting to be a little bit more interactive. So, we'll see how this goes. Um, there are a couple moments in here where I'll be asking you guys to jot some things down on paper um, and play around, play with me conceptually as we go through this uh, sort of to kind of help it stick with you a little bit more and give you an example that you can hold on to um, in a way that you might not be able to in in sort of more abstraction. I'm also going to say that I I'm not going to share anything from the research that I uh, recently did using card swords. um can't but I am really excited that it was a technique that suited the situation. It suited the problem that our stakeholders were concerned with. Um I was able to use this cognitive model as a type of evergreen research artifact uh that's generative but it's also both qualitative and quantitative. So you get sort of the strength of the quant metric or the quant measure. And that measure is really um sort of a spatial measure in terms of how uh how tight the agreement is within these spaces of what you're looking at as as the the why behind um the patterns that you're seeing. Right? So um I'm really excited to be able to share this technique with you. Um, and there is also way more than I can reasonably cover in an hour. So, I'm going to actually append a bibliography to the slide deck that goes out for sharing. Um, that way hopefully you'll have some reading materials to get a little bit more in depth. I've also included my um contact information on LinkedIn at the end. So, always feel free to reach out to me. Um, I think this is, you know, card sorts are like the bulk of it, but there are a couple steps um that I want to outline for you. and uh and really give you the confidence to say, "Hey, I could do this, too." Um, all right. So, I'm going to get started. And I want to start with an experience that I know everyone has had so that we're sort of starting from a level playing field. I don't want to lose anybody straight out of the gates. I want you to think about your favorite retail website. And by favorite, I don't mean the one that is um the one where you do the most shopping because I think that's probably common across all of us. Uh I want you to think more about the retail website that feels easy or familiar the first time you use it. And if you've got pen and paper out or a stack of stickies as some of you might, um, feel free to jot some notes down or you could open up another window and jot some notes on a fresh workspace on your computer. You could even for this one pull up the website, right? And and take a look at it again. So the question really is how does it feel to find what you are looking for on that website with little effort? And I want you to think about more than just it feels good. How does it make you feel about the people or the organization behind the website? So, what I want to suggest is that easy to navigate websites are about a lot more than just buttons, headers, drop- down menus. And I'm going to illustrate that um sort of with sharing my favorite retail website. Um I'd wager that the reason your favorite retail website is your favorite, the reason you love it, aside from what they sell, is how they make you feel about being on their site. I suspect that some of you might have thought they get me or they're like me. And if you look at REI's menu, and I'm thinking about this menu right here at the top in the white, not this. I'm going to call this a submen. Um, that's above the above the search bar. But as you look at this main menu here for their retail website, what you've got are adventures from left to right. You get to the kids, the women, the men, bargains, brand searches. You get to that on the far right, but right at the heart of their search or at the I'm sorry, at the heart of their um menu is adventure. Uh, and even if you look at the secondary menu up top, once you get past the retail pages, you've got REI Adventures, you've got classes, you got expert advice, you've got this really provocative, uncommon path, right? This is a retail website that gets their users. Um, it's not just hiking, climbing, and water sports, right? It's it's about all of those things and defining where you fit within uh within the the field of of shoppers, right? What are you shopping for? You're not just shopping for stuff. You're shopping for adventure. And certainly, there are a lot of industry standards that they're following here. Best practices. You've got that search bar top front and center. you're searching for something really specific because you are a brand conscious or or a a you know, you have a specific thing in mind, but it's still, you know, this this experience that you have of seeing things that resemble how you think about what you're going to do with the things that you buy. So, I don't want to assume how they got here, but most people for their uh information architecture, they use a technique called card sorts. And so, I'm going to review it really briefly for people who maybe don't have um any background with this particular technique. But you start with a stack of cards. And this could be in our digital world, like Jamie was talking about. This could be a bunch of stickies on a whiteboard. Um, you get a stack of stickies or a stack of cards and you ask the participant to sort those into groups that are meaningful to them. Uh, this can be either an open card sort like the one pictured here which um this is from this is just an image I extracted from the interaction design foundation website and it is a useful article that I extracted it from and so this will be one of the things that I share in our bibliography at the end. And like I said, I have no idea if REI actually did card sorts. And for today, I'm going to say it doesn't really matter. Um, card sorts are not necessarily the only way to arrive at intuitive information architecture, but they are a staple of user experience research, right? And in an open card sort, you wouldn't really give a whole lot of guidelines on how to name or how to group things. you would let that happen organically and then ask people to name or label those groups that result. Um what's interesting here is that you know what you're looking for really is agreement over a certain number of users. You're going to try to get at that intuitive organizational structure that makes sense to most of the users who are part of your target audience. This is an interesting example because you look at it and you think, "Okay, yeah, that makes sense." It seems intuitive as soon as you see it. But what about if you moved frogs over into the vehicle pile, right? And instead of vehicles, it was things that move from point A to point B. So, you probably wouldn't label the the um the second group that has just grass and leaves in it as things that are green. You might give that a different label. So that's an example of an open card sort. You can also do a closed card sort where you kind of give people the headers. I suspect that whatever REI did there was something open about that process. So they really got at what are the categories that are meaningful to our users, the people who are buying our stuff. Um, I'm going to go back for a second and just say at its core, card sorting is really sort of generally speaking, it's about taking a set of concepts, themes, ideas, asking users to group them, and ultimately what the researcher is looking for is agreement. Right? So, I I just wanted to say that again because I'm not sure I said that as clearly as I wanted to. I have a little script written, but I'm not doing a good job following it. Um, so that point about agreement is going to come up as um really important in this counter example that I want to share. And this is just a random photo that I grabbed off of um a Google search because um I want to talk about a counter example that came up um in one of the places where I've worked. Uh this is a situation where we were creating an internal port internal portal to consolidate forms from around different websites within the organization and the designer I was working with in fact this wasn't even my project the designer reached out to me because he was trying to figure out how to structure the website and he was really excited to try card sorting but he had never done it before. So we talked a little bit about how to set it up and he executed just reached out to a bunch of associates uh you know employees at the organization to come up with this intuitive information architecture and I didn't really think much about it. I didn't know enough about what was being consolidated and so he reached out to me again. Um well so so uh briefly he did an open sort because this was really exploratory. These were a lot of um forms and and places that hadn't sort of been available in the same place before. So he really wanted to get a sense of what made sense for people. But he also created a discard pile for items that people were unsure about. And for me, this seemed like a really good decision in a lot of respects, but in the end, it contributed to the problem that bubbled up as results started coming in. So, when you get when you get um card sorting outputs from programs that exist out there in the you know UX sphere, there's a number of different programs that will allow you to generate uh card sorting outputs. Most of them will have the two on the on the left, the similarity matrix and the dendagram. Most of them won't have the 3D modeling, but the tool we had access to for this project uh did have that 3D modeling. So, he had these three um three artifacts, these outputs from the analysis that was happening on the back end that he really had a hard time making sense of. Uh, anybody who's done card sorts before will look at these and say there's, you know, there's some agreement, but it's not like really really tight agreement um on any of these visuals that you see in front of you. Um, and we can talk a little bit more about that, nerd out about that in the Q&A if there are some of those folks in the room in the virtual room. So while he was looking at these outputs, trying to make sense of it, and trying to come up with what with the architecture for this website, he was like, "Oh my god, I don't know how to read these. Either that or I'm not getting the kind of agreement that I need to have to do my job." Um, and so, uh, let's just say for now that about half of the cards that he had as part of this card sort appeared to be outliers from the data that we were looking at. Um, so and this guy, I mean, he's one of my most trusted colleagues. He's a great guy. He was feeling like he had completely failed, that somehow he had done a horrible job at this. But as I dug deeper uh and talked to him about how to read these artifacts, I got overwhelmed with excitement because the reason this card sort failed is the exact same reason that REI's menu is such a great match for their target customers. The user base for this portal included everyone across the organization from customer support to dev teams to management. So this is a lot very much unlike REI's tribe of outdoor enthusiasts who might opt into different adventures but really value getting outside. So in other words, my design, my dear designer's sample was filled with specialists and card sorts are not great for creating an information architecture with specialists who have a range of specialties. Card sorts work when there's agreement about how things are related. And what this does for UX is it helps us increase intuitive navigation on our websites. It helps reduce friction and churn on the website. And all of this should, if you're thinking about a retail example, this should reduce attrition, people abandoning your site or cart abandonment. Um, it should increase delight, sales, and return customers. Right? the easier it is to get what I want and put it in my cart and buy it, the easier it is for me to want to come back, the more likely it is that I'm going to want to come back. There are certainly other factors that make people come back, but if people are leaving before they even get to what they want to shop for, that's a problem of information architecture. Um and so instead of this wonderful population of users of of uh folks who are interested in outdoor adventure, right? You have a group of people who are um you know customer support development management all over the place. If this were a if this exercise that my designer had done were really a tight group with an agreement among users, he might have gotten there if it was just UI designers, UX architects, content strategists, researchers, there might be a better sense of agreement among that group of users about tools that they use. Um, and what I want to suggest is that this element of agreement is really what makes card sorts work for other types of research. And so you've got uh you're going to hear me use the term cultural domain a lot for the rest of this talk, but cultural domains are about agreement among a group of members. And what this indicates when you have agreement from a card sort it indicates that there's common knowledge, shared concepts or even shared values. And what that does is it allows us to uncover cultural meaning. This is what makes cultural domains through card sorting or cognitive models through card sorting an evergreen type of artifact. So it takes a little bit more time to do some analysis on the back end. It doesn't take any more time than typical UX research to do the actual interviews. Um there's a little bit more work involved in getting to those models, but what you're creating is an evergreen model. Cultural domain analysis also allows us to distinguish between the shared cultural knowledge and specialized or expert knowledge. So you can identify um people who might be experts in a particular area. And that would take probably even more time to dig into but it is possible. Um, so I'm going to talk a little bit more about what I mean by cognitive models and cultural domain analysis. I'm going to look in the chat real quick because it looks like there is a question from Oops. If I can get my mouse to work, how do you reconcile several user groups for taxonomy decisions? Um, that's a long and complicated answer. Um, I think the the resolution that this designer got to, I believe, was um to take a bunch of questions back to his stakeholders that I fed to him from the from the data analysis and ask them to make the decision themselves. basically say, are you really the primary users of this site or can you uh allow us to create a hierarchy where there's um different kinds of drop-own menus or or whatever whatever they they they decided on. Um and I said your plan B if they are unwilling to have a conversation about that is to do some test and learn. um actually track people in terms of who's getting lost on the site, where are they getting lost, are people getting to the form that they want to get to within however many clicks the business has decided is the metric that's going to be used. So, um I think the last I heard this this particular internal portal um was still sort of in a test and learn mode. So, um let's see. So, cognitive models, there's so many qualitative ways to map cognitive models. Um, this is just one of them and it's one that I think has very particular use cases. Um, but I also, as I said, think that it's really helpful as an evergreen artifact about a particular target audience. And this could be an internal audience if you've got associates that you work with. It could be a a segment if you're working in marketing. Um, it could be, uh, you know, based on personas or or some other subgrouping of people who use your service andor products. Um, and the term you're going to hear me use a lot, like I said, is cultural domain analysis because what you're looking at with a cognitive model is really a shared cultural understanding. Um, and the model that I'm talking about is not just taking a bunch of qualitative data and making sense of it visually. It's really about quantifying qualitative data. So, a really good um a really good analogy to kind of attach to this cultural domain analysis is the idea of a folk taxonomy or an ethnobbotanical taxonomy. So anybody who's, you know, had to memorize the kingdom film order, clearly I didn't memorize it because I can't all go all the way to the end. Um, anybody who's, you know, memorized that kind of lenan structure for bi biology, for botany, um, is going to understand how researchers who are going into another part of the world might try to create a sort of a scientific taxonomy, a western scientific taxonomy around the plants they're finding. But there's a lot of information that gets lost when we do that. And so cultural domain analysis is actually one of the ways that ethnobbatonists make sense of the plants and animals that they're identifying. Um even today there are, you know, we're still finding new species of plants and animals. And so making sense of how people in different cultural contexts use or don't use these um animals and plants uh as part of their daily living, that's that's that field of ethnobbotany, right? And so it's kind of a taxonomy uh based on cultural or folk knowledge. So that's a really good way to sort of wrap your head around this. But it doesn't just have to be about biology. Um it can be about a lot of different things. And now it feels like my There we go. So when I say cultural domain analysis, very generally speaking, I'm talking about a shared cultural understanding of a domain of knowledge. Right? That's a really broad definition of what we're talking about and we'll get into an example in just a second. That domain of knowledge doesn't have to be anything as sophisticated as an ethnob botanical taxonomy. It can be really really simple. Um it can be like list all the vegetables you know and then really understanding what vegetables are you know commonly understood commonly shared uh sort of to a particular cultural group. the more specialized the domain, the more specialized the respondents should be. So if you're talking about uses of plants, so if you went from just name plants to how do you use plants to heal people, right? So again, thinking about that ethnobbotany example, if you want to talk about medicinal uses of plants in the Amazon, you're going to talk to people who are healers uh within whatever cultural tradition you're studying. Those are specialized respondents. Um similar to the specialized respondents we saw in this uh portal that my my beloved colleague was trying to uh incorporate into a very generic website. The cultural domain analysis is also a quantified mapping. And I think I've said this already a couple of times, but it's it's taking qualitative data, the verbal responses you get from people, the information that you call out of these interviews, and it's quantifying that qualitative data through a mapping exercise that shows relationships between things or ideas that have meaning to that particular group. And we'll get into a lot of what that looks like in a minute, but I want to take a little ever so brief trip uh through history. So for cognitive anthropology, which is where this technique um originated from, and in fact, that's really where uh card sorting comes from, too. It's a simplification of a more complex methodology. So, for cognitive anthropology, we could go all the way back to the 40s, look at Ruth Benedict, patterns of culture, but I'm not going to do that, at least not for today. Um, but I'm going to start with a classic. And so, this table that's on the left hand side of your screen is a taxonomy about this domain, this cultural domain of the trustee that was uh in James Bradley's book, You Owe Yourself the Drunk. And so this is a James Bradley did work in the 1960s with a group that he called urban urban nomads, quoteunquote urban nomads. The common vernacular of the day was to refer to these men as tramps or skidro bums. Um, today I would argue that the way that we talk about our homeless neighbors and particular home particularly homeless neighbors with addiction problems with much more compassion and understanding is in part because of the kind of ethnographic work that James Bradley did. So this notion of trustee relates to uh these urban nomads who were imprisoned because of their drinking or because of activities that they did to enable their drinking. And trustees were given a lot more leeway and flexibility uh within that prison space. And the domain of trustee was a domain of like who what is this cultural phenomenon that is existing as an informal category within um the prison system. So you see that he's got sort of higher level labels and then uh information very very detailed information um as subpoints of each of those categories. This is not something he used cognitive domain or cultural domain analysis to get to. This is hours and hours of interviewing, shadowing, um, repeated iterative interviews with the same people, um, looking at people's prison records, right? And so, and then sharing this information back with his uh, participants to get their feedback on did I get it right? And this is certainly a technique that we could use, but this is 100% qualitative. Um, and again, he did his work in the 60s, was published in the 70s. By the 1980s, there were a group of folks, again, in this in this field of ethnobbotney that I've been sort of referencing, um, as well as medical anthropologists, um, who were digging into how people outside the US continue to practice quote unquote traditional medicine. And they wanted to understand how those groups, those individuals perceive of common diseases, diseases that have sort of um you know a scientific understanding that's developed in the west. These researchers wanted to understand how do the women who were primary caretakers within their households understand contagious versus non-contagious diseases? How were diseases classified? What might be a disease that would be classified as risky? Um and the image to the right is not actually from that. It's from a different article. It was just a clearer image. But what you what they did was develop um and Romney Weller and Bachelder are the the authors of this research. They proposed a mathematical model to actually measure what they called cultural consensus around domains of knowledge. So that's another thing that we can cognitive model, cultural consensus model, cultural domain analysis, all of those really mean the same thing. But this is a multi-dimensional scale of the proximity in the responses. So your core items here are those core cultural concepts. Um th this is the knowledge that anybody who is a member of that society can reasonably expected reasonably be expected to know and understand. Um, so the image on the right is probably enough to make some folks heads explode, but all it really is is a spatial relationship based on mathematical proximity. This is how you get to quantification of qualitative data. Uh, I'm going to walk you through some of how we get there, but not all of it. Um, I don't want you to worry about what the math fairies are doing behind the scenes to get to this information. Uh, again, I'll be adding a bibliography including the article that this is pulled from. There's also some other ways to get there, but I'm going to I'm going to start by saying the first thing you want to do if you're going to do a cognitive model is not start with somebody else's list of terms. You want to start with ideas concepts um, names of phenomenon that are meaningful to the group that you're studying. So, this could be if you're thinking about a retail audience, maybe you're focused on millennials and how do we get to understand how that category, how that group shops. So, you might talk to a bunch of millennials, do a longer structured interview, potentially over multiple interviews. Um, and you would want to have about 20 users total. Uh, you might need more than that, but 20 users is a good place to start. Uh, and then you take that list from each person. You want it sequenced in the order that they listed the items. So, you ideally are getting them to write it down for you on a piece of paper or in my case, I opened up a mural board, threw a bunch of stickies up there, and I said, "Just fill them out as you go. Feel free to talk about what you're writing if you'd like to. We'll go back and I'll clarify anything that I don't understand when we're done." So, um, once you get all of those lists, you have to do something with them. You can't just smush them all together, spin it around, and magically, I mean, you can. There actually is software that you can spin it around magically um play with it but it's actually a DOSS based it's actually DOSS based software so unless you um are on a PC you might have a difficult time using um that magical that magical software. Um, but really what you want to do is generate a master list from all of the different lists. And you want to get to about 20 to 40 items that are common. Now, that doesn't mean that all 20 or 40 items have to be on every single list. Um, what you want is a master list that pulls together those 20 plus items based on how frequently they're listed. So that can be a simple count and you might get to you know this showed up on at least five of the lists for the 20 users and anything that didn't make that cutoff falls off your master list. Um but if you can you also want to look at the rank order of those responses. So if something shows up five times and is ranked number one every single time versus something that shows up on all 20 lists but is usually towards the bottom, those may actually have equivalent weights. Um what we did for the project that I just finished was we did it based solely on frequency. Um partly because our master our our um our big pool of stickies was enormous. There were thousands of responses that we had to call through. So there were a lot of outliers. In a perfect world, I probably would have added 10 more people to the mix. Um it seems counterintuitive to if you have too many categories to add more people. But if you can add more people or or look at the characteristics of the people and say, "Are these people all really part of the same group or do I actually have people who represent different cultural groups?" Right? Um so there's different ways that you can sort of evaluate your list as you're going through, but let's imagine that you have a a population that you reach out to. You get your 20 users. You structure in some free listing into a longer interview, a longer structured interview. you have some frequency numbers that you're like you feel really like there's enough people who are saying this that it belongs on my master list. Boom. I'm ready to go and I'm going to give you guys an example um something that we are going to play with and this is if you have that pen and paper available this might be a place where you can start to freelist. So, we are going to take a second and um list all the foods you consider southern foods in an interview context. I'd be able to see, are you writing stuff down? Do you have writer's block? Are you do you have questions that I could respond to? and it could be a little bit more interactive. I'm assuming that the category of southern foods is a little bit easier for people to wrap their head around in an environment like this. I will also say that there's a little um there's a method to the madness of choosing southern foods, which is that I used to do this. I I used to have um college freshmen read a book that used this technique of uh cultural domain analysis. It was a book about ethnome medicine and so it was that classic example that I've kind of been referencing throughout. But I wanted to give them a chance to play with this exercise to kind of feel the methodology in their body a little bit, attach it to something that was commonly shared across all of them. I'm in Charlotte, so talking about southern food is kind of a normal thing. And so I'll uh when we get closer to the end I'll share a some of the insights that I gained from doing this exercise with hundreds and over three years I did this exercise with probably over 500 students. So um it's interesting very very interesting. All right so your free listing outputs um and keep feel free to keep adding to your list of southern foods. things are going to come to you as I'm talking. Um maybe if you didn't notice the photos on the page and you don't have those things listed. Um that's I mean I feel like I'm cheating by putting those pictures up there, but uh I digress. Um so so there's like I said there's really two ways to deal with these free listing outputs. There is this multi-dimensional scaling which I've already talked about. You've got your core items. They're mathematically you know proximal to each other in the center of your diagram. This is great. Um, this is a lot of work to input it into the software. I am going to try to find out if there is a uh an updated version of this software. The last time I checked a couple years ago, there was not. And like I said, the study that we did, um, this really simple frequency uh the the frequency uh ranking uh was really helpful. Right. So you've got the he has the average rank here as well. Um so if you've got like you know if you want to have it rank ordered and you've got frequency of 13 you make the deci decision between which is in first place and which is in second place based on where they're typically ranked within the list. Um and oops so this is this is essentially the technique that we used. So, it's a simple count. Uh, tie breakers happen through looking at the rank and it'll get you easily to that top 20. And again, you know, these are these are exercises that you can do to get started. You can maybe play around with your colleagues talking about animals, talking about southern food, talking about um vegetables, whatever topic you want to play with to kind of get your feet wet. um with this technique of freelisting you don't necessarily have to do a card sort with it but we are going to do a card sort with ours with the southern foods. So this part phase two of cognitive domain analysis is exactly like a card sort and we've we've already talked a ton about card sorting. So this should feel like repetition. This should feel like okay I think I get this now. Um, you're going to ask your respondents to cluster the items on their list and and let's just pretend for the sake of argument today that whatever list you came up with of southern foods is actually the master list. So, I want you to cluster those items based on how they relate to one another. And I think one of the reasons I chose Southern Foods as a sort of example for my college students was because I moved to the south and there is this thing on menus in restaurants down here called a veggie plate. And it was astonishing to me that you could have a veggie plate that didn't have any vegetables on it. You could have mac and cheese, French fries, and black eyed peas. Um, none of I mean I guess a potato is a vegetable, but is it really? I mean, like there's there's some And beans. Yeah, they're a vegetable, but are they really? Like it's not a tomato. It's not a cucumber. It's not a collared green. So, this was this was something that made me think, huh, I wonder if people how people are going to deal with these things that end up on that veggie plate eligible list of items on a menu. So, so hopefully you guys are sorting whatever list you came up with. I'll come back to this left-hand side of the slide and say again, you can go back to the same people you asked to freelist and ask them to do the card sort or you can go to completely different respondents. I think it's kind of neat to do a mix um especially if uh these can be recorded interviews. So, I'll tell you what we did. So, in this land of virtual research, we did virtual stickies on whiteboards with our respondents for the freelisting exercise. Some of those interviews were just really fantastic interviews where people were able to kind of think out loud as they wrote the list. And I thought, "This will be really neat." We didn't have a lot of time by the time we got to the card sorting. And so what we ended up doing was doing a remote unmodderated exercise with a card sort built into it. And that card sort that was built into it, I thought it would be really nice to have some people do this exercise. and talk out loud as they're putting things into categories and naming those categories. It would be really nice to have some people that we know are going to be able to think out loud. And so we recruited some of the same respondents. Um but we also brought in a different group because the different group of respondents should be able to work with the master list you've generated if they are part of the same cultural group. It's all about agreement, right? So you should be able to land on on a list as your master list that's going to make sense to the new people that you bring in to do the sorting exercise. So you can intentionally do half and half. You can intentionally do all one or all the other. Um but you don't necessarily have to worry as much about those kinds of questions. And again for this one you want to have sort of a minimum of 20 participants because you want to have agreement on those clusters. don't necessarily have to have agreement on the name that people assign to the category, but it is interesting when that agreement does come up and they start to use some of the same language. Um, and again, because this is exploratory, you're trying to get into the minds of people on a question that you're like, we don't understand how they think about X. It's good to start with an open sort. If an open sort doesn't work, maybe you shift and do the whole iterate, iterate, iterate, iterate, do the whole exercise again with another 20 participants, but do it as a closed sort. Um maybe pull out some of the category names and then just see how things land. Maybe you go back to Scratch and um add some people to that freelisting group and make sure that the list that you came up with is truly the master list. So there's a lot of points along the way where you might want to go back and iterate. Um we had a time deadline. Our stuff was close enough. So we uh continued to push forward and it really in the end the the product that we uh created the the evergreen uh artifact that we created is really solid and um had a lot of resonance with our stakeholders. Um because what it comes down to is you know what you're asking with the grouping and the naming of the category is what is it that makes these things alike? What is the common ground for these items? So I want to go back to this question of is it a vegetable? So if you pile if you've grouped your your um your c your southern foods into piles and you've named those piles um I want you to look at what you terms you used and and Just take one second. So in the three years that I did this exercise with students, remember it was probably 500 or more students who did these exercises with me, there was one category that always showed up, always, always, always. That was fried foods. So, if you think back to the slide for the free listing exercise, notice that these are all fried foods. There was a hint in the photographs. Um, and there's a hint in this one as well. The category that almost never, maybe once or twice, this category showed up among 500 students. And if I had them grouped, they had them when they did the sorting exercise, they did it in a group of three to five students. So approximately a hundred card sorts vegetables showed up maybe twice because all the vegetables were either fried or they were part of another category like holiday foods, grandma's Sunday dinner. Um so that was a really um eye-opening moment for me as I did these exercises that I had the first class I did this with there was absolutely no nobody in the entire group of 100 students so what 20 card swords nobody had vegetables and I was like um so even if you think you know a category the lesson in that example is that even if you think you know a category you're still going to learn stuff you didn't know. Um, and when you present this kind of information to stakeholders, there is going to be something because your stakeholders, if they're in an industry that is selling a product, working with a particular clientele, um, focused on a particular type of experience, there is a good chance that they know a lot about their users, about their segments, and there's a good chance that they can't really put their finger on what it is that they know in a concrete create tangible way that this technique helps you get to. Um, and I'm doing a quick little time check. So, some of these slides that you're going to see next are kind of going through. You're going to you're going to see some stuff that you've seen before, and then I'm going to have a couple wrap-up slides. So, these are the same card sorting outputs that I showed you earlier. Um, again, the similarity matrix is in pretty much every card sorting software that I've seen. The denog is in most and the 3D modeling is probably the least common. But in situations like the one we found ourselves in where we're like ah I mean like are these really the same grouping? So this dendagram and these these kind of go together. They're completely anonymized here. Um but one of the things that I want you to notice is uh with denogs 70% and this is like one of those math fairy things that they told me and I just believe it because it's true. um that 70% line, that uh uh vertical line is sort of where you want to look at your groups. What are the groups that you're identifying here? Um and this is where you have 70% agreement. This is this is good. But what you'll see here is there's a lot of colors. So there's a lot of different groupings, and that's maybe too many to make sense of things. Um, and you'll also notice that there's some lines that come all the way out. If you look at these over in, and this is a similarity matrix from a different project, um, that I pulled off the web. Um, but if you look at the similarity matrix, that also helps give you a sense of like how much agreement there is between two individual um, two individual nodes or two individual cards, right? And so when you've got a 100% agreement, those things have to be together in your final model. Um it's really helpful to be able to look at all three of these things or at least two of the types of um outputs. There's one thing that I want to um yeah so I'm going to talk about each of them individually because there's just one one highlight that I want to share from each of them. So the similarity matrix is probably the best data source, but it is pretty difficult to understand and it's really hard to think about how to cluster things based on these numbers. Um, this one is a great reference point when you've got decisions that you need to make as a researcher based on the dendagram or the 3D modeling. So where are we going to draw the line between this cluster and that cluster? The dendagram is easier to read. I've already said that you want that vertical line at 70% agreement or higher, but you may need to drop it back to 60 or 50% to get a larger number of groupings. Um, this is a situation where you may want to follow on with additional iterations, but in our world, there's rarely time to do that. As I mentioned in my example, um, it's a little easier to get to a a larger cluster, a larger grouping in this 3D model. And this is from optimal sort which is a spec very specific tool that's great at doing card sorts. There is a free version. Um we did this in the paid version so we could actually see this 3D model. And so with this one it actually this is exactly like that multi-dimensional scaling except for it is in instead of being in two dimensions it's in three. And so you're able to spin each of these around in an interactive way and you're able to say oh how close is this? So, this dot looks like it's really far away from these dots up here. I don't know if you can see my mouse. Um, this dot is really far away from these two dots up here in this view, but as you turn it, it may actually be closer than you think. So, it's really fun to play with this 3D modeling. Um, but you within optimal sort, they have sort of a slider that allows you to say, "Let me see what this looks like if there were five groups or six groups. Let me see what it would look like with three groups." And surprisingly where you think the cutoff is is not always where the cutoff is. And so it's also helpful to look at the actual data points it creates a very cluttered graphic. But if you're moving between something like a 3D model and a similarity matrix or a 3D model and a dendagram, it really does help to look at all three of them across and say, you know, where where am I going to use my best judgment to land on the final set of categories? And what I'll say is like a taxonomy, even if you're going to do a different kind of visual representation, like a taxonomy, you can have subsets within that. So I want you to take your eye back to this dendagram in the middle. And if you look at this pair at the very bottom, the pair, the two pink lines at the very bottom on the left of that dendagram image, those are tightly tightly clustered. That is the best and strongest agreement in this entire chart other than right up here. This orange cluster is generally speaking is very very tight. Like tighter than anything else on here. Um but then this one you're like hm that's like 90% agreement or 95% agreement. That's pretty awesome. I think I'm going to make this cluster of two sort of a subset or or somehow link them more closely together than the rest of the stuff that's in this bigger group because these two seem to go hand inand for our uh target audience. Likewise, these two up here I'm going to have go hand in hand for our target audience. Um and then you might have some that are close and you want to kind of play. So, the second um this this second group of pink items that's closer to the top, you've got a pair right here at the bottom of that that pink cluster in the middle and you've got a sort of bluish purple um you've got also some tight agreement there as well compar comparatively speaking. Right. So, would you want to look at those two as subsets of the larger clusters? Um that's where you're like, "Huh, I don't know. I can't tell much from this dendagram. That's those are the those are the particular use cases where you're gonna want to go back to the similarity matrix and look at is the agreement comparable to what I'm seeing over here. If I if I cluster these two together, am I going to have to do that with um say this larger group of pink ones at the bottom? Right? So there's there's some there's art and science to coming up with your final drawing. And your cognitive model that you repres the representation that you create from cultural domain analysis is going to look a lot like ones that don't use this qualant technique. These are just random ones I pulled off the internet that sort of match um in spirit that what I created out of the research that we did recently. Um, and I think this one, the McDonald's Golden Arches one is nice because it represents that you've got subcategories within a larger category. Um, so you can do that. You can use color coding. You can give really specific things. But I suspect that this one uh is really based on um, you know, a set of interviews with loyal customers. What do you love about McDonald's? What keeps you coming back? So you can get to something like this without going to all the effort that I've I've been talking about today. You can definitely get to a cognitive model for something really specific that people have at at the top of their mind. Right? Same thing with this uh the one on the right is from uh the European Association of Remote Sensing Companies. It looks a lot like the artifact that I created, but it's based on um you know a taxon a taxonomic model of how different users integrate this tool into their process. This is something that has to be top of mind because it has to do with operations. It's people are able to talk about it in a structured way. Most of what you're going after with cultural domain analysis are questions that are not top of mind because it's not a decision that I've made about how I'm going to interact with your brand as it is with this McDonald's example, right? And it's not about how I manage my operations or how I manage my process, which I have to figure out in order to be effective as a as a business, right? So the kinds of questions you're going to ask with a cultural domain analysis are sort of uh sort of down a level or up a level which however you want to think about it. Um they're abstracted from the things that we can necessarily consciously talk about um in a relational way. So what else can you do with card sorts? There's a lot that you can do. Um and I'll briefly share a couple ideas. Um, back in the day, this is probably 10 years ago, I did some work with for Pepsi for one of the um, market research companies I worked with, and we looked at occasionbased consumption. And this was really just a sort and rank exercise. All you did was sort and rank. And it was what are the occasions, you know, what's the baseline? Why do you love this? These were diehard Pepsi drinkers. Um, and fizz, fizz, fizz, fizz. Fizz is everything except at Christmas time when you have to feed a lot of people. I am not buying something based on fizz level for my whole family. I'm buying a 2 liter bottle because I got to buy a lot of stuff and it's going to be really expensive. Another occasion uh where you saw some differences in how people thought about it were the morning commute. I'm going to be sipping at this soda all day long. I'm not going to buy a can. and I'm going to buy something that can be closed in my car and it preserves that fizz fizz fizz that is everything to me. I might have a can of soda with lunch because I'm drink going to drink it all at the lunch table. Um, but if I'm sort of on my morning commute, I want like, you know, a one liter or a half liter bottle that I can close and preserve that fizz and prevent spills. So that was research that we did with Pepsi. Literally taking people through all of these these characteristics of soda, of carbonated beverages, things that pe draw people to that product and then having them walk through occasionbased. Pull in the things that matter to you in this situation and then rank them as to how important they are. Um you could do the same thing with situations or scenarios where you combine a mapping exercise with sort and rank and that's kind of what we did with our study. Um so you have a baseline of how the customer segment understands a domain of knowledge and then the sort and rank are really what concepts are most important in a given scenario. So, if you think about a retail environment, interaction with a website versus interaction with an associate, um the store layout and how that affects your experience, uh you know, omni channel, if you're moving to that buy online, pick up and store experience. Um, and all of these are things that I don't work on, so that's not what we are sorting and ranking, but those are things that you could do if you had a group that you really wanted to understand the cultural experiential piece of how they interact with your brand. you could get that baseline map and then say like what are the what are the things that matter in particular situations? How do things change in order of importance? What are the things that are important no matter what the situation is? Um and again with Pepsi that was fizz fizz. Um, you could also use sorts in that sort of sort and rank uh if you were doing a competitive analysis. So, you know what's important to customers and maybe you get that through a co a cognitive map. Maybe you get it through some other type of research where you've been able to extract information from those existing data sets and say these are the things that people have said over and over are important to them. We're going to create this list of 20 based on past research, but then we want to look at, you know, maybe a a competitive analysis. How do So, if these are the things that matter, how do we rate against our competition? Who's the closest match to customer expectations? Um, that could be another way that you might use that. So the sort and rank are for really tactical questions that you can drill down into from this this more abstracted cognitive map. Um and then again uh if you have if you work with a group of experts if you work with health care providers and I'm thinking of you know the first talk from today if you're working with healthcare providers or financial planners and you really want to understand uh so again financial planners thinking maybe to to some of the folks that Jamie works with um h how do they think about the world that they operate in not from like what my what my employee manual says I should be doing but how I can be thinking about sort of my relationships with customers for example. Um so in conclusion so it's really cultural domains are really about agreement. Um and this is an evergreen artifact but it can lead to actionable insights right this helps us understand key drivers. It helps us quantify qualitative data. It provides strategic opportunities like maybe increasing loyalty. Maybe your certain rank questions are about loyalty, uncovering unmet needs. Oh my gosh, this is something that everybody says is important, but there's nobody in our space who's who has a product that addresses this. Um, prioritization based on key customers. Um, so if you have a customer base that really values a particular area of a relationship and you're not focused on that or or there's a decision that has to be made about product X and product Y and product X is the one that addresses that customer need, that's the one that gets prioritized. So it gives you it it it allows a lot of this qualitative why to really drive um to really drive some strategic decision- making. So that's all I had if there are questions. Marine didn't see I think you already addressed the one about how do you reconcile several user groups? Yeah, iteration iteration iteration. Yeah, that's just Yeah, I can imagine. I don't see any other questions. Um, do you have a final slide with your contact information that maybe people can reach out to you if they do have any followup? I do. You can find me on LinkedIn and like I said, uh, the slide deck that I provided, Bill and Marine, uh, it will have a bibliography. There's a lot of really good information out there. Um, yeah. Perfect. I really enjoyed it. Thank you so much for taking the time to speak with us all today. And this is our final talk for the day, everybody. So, thank you all for joining us and we hope to see you during our next virtual insights conference. Thank you.