Treating Employees Like Customers: Internal and External Listening Strategies at Indeed

hi everyone good afternoon my name is Krista Reed and I'm the director of operations here at accelerate research very very excited to introduce our next speakers for our virtual insights conference if you've been with us throughout um over the course of the many virtual insights conferences that we've done welcome back if this is your first time with us thanks for joining a few rules that we're going to just go over before we get started we are at the mercy of tech so if there's a glitch if there's an issue with the audio just please bear with us we'll be back we'll get it worked out also use this opportunity to network to engage to speak with one another I'm sure each of our speakers will be happy to follow up with you after each of their presentations so take that opportunity to engage with them and also just be kind and courteous in your comments with your questions whatever input that you have we greatly appreciate your feedback though so we'd love to hear from you if you have any comments um so with that I will introduce Laura Laura and Benjamin who are part of the ux research team over at indeed.com and I'll turn my camera off and my microphone off and I'll hand it over to both of you so thank you so much for your your time thanks everyone for listening and enjoy yeah hi everybody thank you so much for having myself and Ben Ben and I have been working at indeed now for five years and we've seen the organization evolve from a 3000 person company all the way up to 15. and with that Evolution has really come a lot of change as you can imagine burning Platforms in terms of organizational design and change product changes and with that Evolution Ben and I have really helped the organization develop internal and external listening strategies so when we talk about internal it's with our employees and then external with our customers actually engaging and using and investing in our products so we're really excited today to be talking to you all about um what we've done what's worked what hasn't best practices tips and tricks along this journey of a really a growth organization um I work Ben and I both work on the voice of customer team which sits in the sales and client success organization um we are both researchers I focus on the qualitative side and Ben focuses on the quantitative side um and we work directly as a conduit to our customers work with them directly and hear from them um we transitioned from a team around two around almost two years ago where we focused on internal listening so we truly have seen both flips of the coin of internal and external listening and and he and I have found a lot of synergies between both of those worlds so Ben do you want to talk a little bit about yourself and then we can dive in yeah sure so as Laura said that we're both uh on the VOC team she does the qual I do more of the quantitative side we have a ton of experience working in both like internal listening strategy or really developing those internal listening strategies sort of from ground up where you know you don't have a ton of communications from from business from internal members to your businesses and how do we survey and how do we develop that sort of strategy and really sort of the the focus of this presentation is is not only making a case for the synergies as Laura said between internal and external customer listening strategies but also like how you should in some regards almost treat your internal customers like your external customers and your external customers kind of like your internal customers and so it's they're different kinds of customers but there's a case to be made for for treating both like customers and wanting to show a little tips and tricks and how to succeed in both yeah great and and please let us know if you have any questions throughout the presentation we'll try and pause and and and and answer anything that you may have again as a reminder there's no smaller larger dumb or highly intellectual question we're here to we're here to help you and and let you know what's worked and hasn't so let's move into our slides and we'll go from there we'll start our journey together foreign so thinking about like what are some of the key differences between internal and sort of our external customers obviously it goes that being said your internal customers are the people that sort of work for you I love this sort of Richard Branson quote it's something that we kind of carried in our hearts when we developed our internal listening strategy and it's you know take care of your employees he'll take care of your business as simple as that so taking care of your another quote that that often comes up is you know take care of your people and the profits will follow um oftentimes you know thinking of your employees as customers kind of requires a slightly different mindset than thinking of your employees simply as employees or simply as resources again they're your employees are sort of integral partners that help you deliver your products and your services to your external customers so if you want to think of them almost as a middle customer then that's probably a good way to think of them you have to get them bought in you have to get them sort of energetic about the things that they're doing and then they pass that energy along to the customers and again the profits sort of follow external companies just for definition's sake can't leave them hanging as we know they're customers who get in the data end users of your products not directly related to your organization okay and so here we're sort of making the case for for treating your employees as customers um in all our experience treating your employees as as customers results in some of the best performance strategies to get the most out of your employees uh in some respects what we've seen is it helps humanize the business a little bit where you know we oftentimes put a love not unrightly so we put a lot of attention on the needs of the external customers but focusing on some of the internal customers too um shows that the shows the employees that your business can pivot the needs to the needs of your internal customers as well and so how did Laura and I go from from pivoting you know a strategy where the employees were treated a lot like employees to where now we're sort of treated like an internal customer base and that is to employ some of the same listening strategies that you would use when you're building some of your your external listening strategies and and this is a really like difficult shift in mindset um especially in in Tech firms where you're trying to make money you're trying to optimize your products create new ones innovate um really it's it's all about the customer it's about the bottom line it's about what's the new feature that's coming out and it's and there's often a lot of Chaos in Tech firms um you know trying to organize around the products and and optimizing that go to market strategy so really shifting this um this culture of like well we got to look internally as well um we've seen a journey of it Ben and I it's it's been really difficult I think one of the key things on this slide um is you know something that we had to do was not just look at the HR department right like HR is HR and that's it um it's it's really about creating a stakeholder group outside of HR to really get on board with this concept and getting those leaders to really buy into this Pro this concept so it takes a lot of networking a lot of effort and it's not just an HR tool an HR listening channel it's everybody's and we all Own It collectively in different parts of the organization including those teams who actually are innovating and developing developing those products and and bringing them to Market and selling them to on the front end basically um so how do we sort of transform this shift in mindset as I had just mentioned I mean I think we really had to go back to the basic building blocks of organizational design and with that really comes that North Star and that starts with the mission statement and then values and what have you um and so in our journey when we inherited this a little over five years ago um in order to listen to our internal employees we only had five questions that our senior leadership team would ask biannually um that and there was a lot of disorganization of like other mini surveys going on in certain teams but there were zero cohesion around um what we're going to do with this high-level data um doesn't even mean anything how do we operationalize it um it was it was there was really nothing so we had to really put our heads together and say um how do we how do we actually give people a voice how do we give people a true Channel and a platform where they know that something's going to happen so we put our heads together and kind of built this Mission and it's based on action I know we talk about action a lot when it comes to listening in any Forum but um so it's the action piece we'll talk a little bit more about it's complicated but um the first building block was something needs to happen with this data um and with that concept of something needs to happen we have to really investigate also what are we measuring um what is important to indeed what is our culture um what drives um engagement what drives um our DNA what makes us unique we had to really dig deep into that and figure out how do we measure against those um those topics and then what we had to do was we have to think about like this has to be practically operationalized somehow um and to really to to create that meaningful change across all different organizational levels and again to switch that mindset that this is not just owned by HR this is owned by all so again it was really building a network of stakeholders getting that buy-in from leadership to say this data is important this is an important listening Channel and here's your role in in driving the action forward once we get those insights um we also had to um really push our leadership to say you know high level insights is not going to get you anywhere we need to better understand and investigate and get curious about um our employees internally and um with that came a set of of more detailed um questions to really understand the drivers and the why behind things behind high-level kind of insights and we also introduced the concept of focus groups one-on-one interviews to really get deep into the rich insights of of what is going on in our organization globally as we grew um so for us it also became it's not about the known okay yes we have to have consistent measurement strategies that we that we measure year on year to see the health of an organization but it's also figuring out covert Pockets where we're hearing grumblings of issues potentially that we can get ahead of with action and the right questions and the right measurement strategy um if we have you know our quantitative data maybe for certain um insights it does make sense to go into focus groups and one-on-one interviews so we get that full picture and we can give our leadership and our action groups the right set of insights actually operationalize um what we've learned what we found coming to the external side of the house on the team that we're in now is that the missions are very similar um it is all about action and it is about making sure that our data goes toward um um changing things for the better so you know Ben and I were laughing when we looked and we're comparing these mission statements they're very much the same um and you know we were excited about that so here are the internal and external channels so as we went on this journey Ben and I have comparing kind of listening strategies what we also found here yes the mission statements were the same but also many of many of the ways that we measure internal and external ways of listening is is very similar to and we want to give you all some ideas around what we do here at indeed that we find Innovative and interesting and unique and it may align with what you all are doing as well and hopefully we can give you some ideas as well um on the left hand side we'll talk about the internal um and you know from this as mentioned we went from only measuring five really high level questions to this robust list and it took a lot of time and influence to get where we are today but I can talk you through that Journey um in terms of listening to our employees um this is a multi-pronged approach um we debunked the fact that it's your traditional engagement survey or now you know the buzzword is pull surveys um or even daily surveys I know some organizations do like Amazon does one question a day um we we felt that it has to be in multiple different channels um to give people that platform and voice depending on how comfortable they feel and the type of measurements we need to drive organizational change um so yes we do have an engagement survey that is more detailed we also have quarterly pulse surveys depending on some lower scoring item questions and topics and also some hot topics for our for our leaders in SLT we have ongoing ad hoc surveys one of those examples is you know life this hybrid life that we're leading post pandemic how do people feel um what are some changes we can look there and and some grumblings if there are any we also have your standard HR processes around hiring onboarding and exit surveys and and we really do we're trying to you know what's a I think a challenge but um it's really important for us to knit all of these data sets together where nothing is in Silo right we're not just looking at the engagement survey and one snap point in time but we're also weaving in okay so what's happening with their exit surveys as well or onboarding as well to make that experience best for new hires so we have a really a robust teams here who build this story this employee story from the minute you walk in the door at indeed to the moment that you leave you have that Journey here and that's why this first bucket here is really important for us to measure that we also offer focus groups depending on areas of concern that come out of these four main pillars of quantitative data we also have office hours where we you know we really are there for to be on the ground and and have those safe places where people can have a voice and we get more detailed informations information this is also a really good place where we get solutions to things where we test Solutions where from the quantitative data we may have some hypotheses yes but we don't want to design things for our employees internally without having them have a seat at the table and be part of that design factor that is also very similar to what our product teams do to go to market external strategies for our products so we take that same kind of design methodology internally as well for programming that we roll out and that is the key for so many um for a lot of success I think in our programs and again is getting buy-in from stakeholders as well outside of HR across the organization to make sure we're designing together as one solution um we also have a slack Channel I don't know how many of you use slack but here we we did something quite daring um that our CEO endorsed here and it's a slack Channel where it's dare to share so you can talk about anything and everything um that comes to mind Hot Topics around Career Development um emerge from this salary hybrid working Arrangements fear of the pandemic a lot of anxiety is there and and it's it's been quite successful it's a really good listening tool um you know we get real live conversations and really good um qualitative data and uh from this I think I think this is a tough one because um psychological psychological safety is is fraught and can be really challenging or in organizations um so this is quite unique to indeed and that culture was really dictated from our senior leadership team and CEO to encourage to have a voice and that any voice is okay so I think that really has to be a top-down effort and part of the fabric of the DNA of the of the firm um our CEO again really leads the charge here he is bi-weekly syncs with the entire organization it's a free forum where we can ask as many questions as we want and again people feel safe to do so it's pretty amazing um what people ask quite intimate things we also ask him personal questions what do you like about your records you know he's a big guitar player so it really comes from the top is is what I what we have seen here um and Ben do you want to talk a little bit more about some of the similarities to the external listening channels and what we've set up here internally sure so as Laura's kind of saying for when we sort of transitioned over from like the engagement side to more of the external side a lot of the same strategies that we employed for that sort of internal listening we also employ for our external listening strategy too so similarly you'll see on the right side where we have that sort of our engagement surveys we also we obviously have large MPS style surveys quarterly pulse surveys we have product surveys um you know multi-channel ad hoc product surveys like csat ask them we're we're sort of closing the loop so similar to and we'll get we'll sort of touch on this as we kind of go on we're talking a little bit more about some of the action planning that we're using but we're using some of these surveys to probe for customer pain points kind of like what we'd be probing for sort of employee pain points and then eventually using asking or eventually using action planning to start closing the loop with those customers to figure out like what's going on what can we do to help you similar to what we would do with with our employees and we'll give you a example of something that we did um focus groups we have something kind of unique uh indeed where we have specific customer communities so these communities uh are made up of our of our external stakeholders big big users of our product and so we can always go to them to have sort of like these focus groups but then again it's a two-way channel right or two-way street I should say where you have you know the cut we're going to the customers asking uh their opinions about products and and services that we offer but then they could also again sort of come to us and so one of the things that we we do have not unlike other organizations but we do have you know customer feedback forms which is essentially an ongoing suggestion box so what do you like about our products what do you find the what are some of the pain points about using our products um you know what can we do to help make your experience uh of using indeeds products and services all the more better so in sort of closing a loop again we have like the bi-weekly syncs with the CEO we also have sort of broadcast interview between clients and Senior leaders and what that does is allows so similarly to us as employees getting a little Forum to go ask questions about things going on in the organization now of our senior leadership now our customers have the same opportunity to do exactly the same thing you know can we what's coming in products and services what are you what's new on the table can I get you know different sort of strategies now our customers have the opportunity to to do the same thing and I think what we're sort of touching on here is like we have this difference between you know external internal external channels but at the end of the day we're using all of these sort of multimodal approaches to close the loop with our not only our internal stakeholders but also our external stakeholders and I think that's a big we'll talk about action planning but I think that's a a big step like you do all of this work in order to measure in order to you know if you have pain points in order to help make the experience better but then what are you doing on the side to to closing the loop you know data is only so good in a vacuum you got to put it into action at some point right I can jump to the next slide yep yeah go ahead go ahead go ahead okay um so yes Ben said we have the data now what do we do um so it just you know just in terms of what we did with um you know our internal listening groups um just want to give an example about about that so action planning with internal listening channels is like I mean we've all heard about it and it's it's it's a must do but in practice it's really hard to do it truly is um it it takes um a really finding Champions across the organization and I think it it has to come from a top down um in our leadership and and leaders and you know SLT VPS down to directors down to managers really encouraging that change um you know people get their dashboards they look at a report and they say uh I now would right and what we found that with a lot of internal listening channels a lot of that data is sensitive it's complicated it's emotional it has to do with your day-to-day life here it has to do with very sensitive topics like diversity inclusion and belonging fields and it's not so easy to to kind of take action versus like I'd like to change a button from you know the color purple to Blue right so it's it it takes um a lot of effort and time and effort but again what we did here at India need was we took a multi-prong kind of years-long approach to this so the first piece was really educating our HR business partners our HR Partners on what is actually action planning what does engagement mean why is it important to your organizations why should you then in turn be helping your leadership that you support so that took a lot of training and effort it took around a year I would say plus we then in that year built Champions and stakeholders across the organization in different seniorship senior leadership levels we then kind of put the message out there too why is it important to look at this Ben and I um felt that the pandemic was actually um a good thing for um for for our space and internal listening channels um it really kind of propelled to the Forefront even more so um the pandemic really helped solidify our plan and and get those champions in place where listening had to happen as we were working across the world in such a fraught distraught way um and so what happened here is we ended up creating um like tiger teams for different topics and and focus areas that came out of all of these listening channels we spoke about beforehand and we elected leadership teams to those small tiger teams and we posted as an opportunity also for people to have Career Development so for people who wanted to lead get exposure and different ways of working um you know really amazing things actually came from people for people's careers um being involved in these tiger teams um and then what happened was the tiger teams would report out on on progress figure out is this is this a short term or long-term win we manage expectations to everybody that action takes time it's not going to happen overnight potentially sure they're easy wins you can do a different milk in your in your refrigerator right for coffee but for really complicated things like you know diversity inclusion and belonging Career Development Performance Management these things take time and so it's making sure that we're evolving on that scale and looking at it as a long-term product but it's really um getting those organized tiger teams and getting senior leadership buy-in which has made action um really more organized and impactful here um Ben do you want to talk about the other yeah so we obviously turn the data into action plans but one of the things that we think about when we're talking about sort of that internal versus external listening use your external listening to sort of help you know you productize some of that data and make changes to your goods and services well when we did it with internal listening it was kind of the same thing the the shift in the mindset is that sort of the products that you're you're selling are more products to your employees and they're a little a little more tangible like in things in terms of things like Career Development or in terms of you know next steps and and what you want to do with your job Etc so by creating these sort of tiger teams and creating these sort of pillars that that we did it was essentially like creating whole new product strategies that we have to turn around sell to the organization as like this is something that you should be doing like a product that you should be selling to your customers and then you know sort of take it from there so that was that's that's something that we we certainly did with the internal side so we formed Partnerships within the organization we created our pillars or sort of our products though that we were selling now sort of selling and then um sort of went from there but then also again sort of to go back to that closed loop right so we have these products and services that are out on both sides but we obviously are going to end up with some sort of pain points going forward so one of the things that you always have to do is you always have to touch base and and re-measure so did the the changes in the products I.E like the services you're providing for your employees or the products and services you're providing for your your external clients produce the change in the behavior you're wanted are we seeing and in case the internal side it was are we seeing our retention rates go up are we seeing our satisfaction rates go up are people feeling satisfied in their career developments you know one of the things that our CEO says is you know find your next job that indeed and so making Career Development sort of that that pillar are we seeing more people finding their next job and indeed but then also on the external side again you still have to do the closed loop you still have to bring it back home did the changes that you made to your product eventually produce the the the changes in the behavior that you want our customers more satisfied are they are we seeing Less customer churned are we seeing more Revenue come in in the case the internal side it's kind of interesting you know talk about revenue on the external side but then on the internal side our Revenue numbers were always basically like what are the engagement scores like that was our our Roi what are our retention scores that was that was our return on investment so on both sides you have again the listening and then the looking the product changes and then the ROI on both the internal and then the external side yes so we want to dive into a case study around um you know just to just to operationalize all of all of what we've been discussing um so in turn in terms of the internal listening side of the house um you know to bring kind of home the internal and external um theme Here is so our mission statement here at indeed is we help people get jobs and you know we were so focused on that infusing that into all of our products um and you know interestingly it really was only a few years ago where we really brought this same mission statement internally as well and and it's essentially we have a phrase that our SLT members call like drink your own champagne um and and kind of productize it internally as well and live that mission here as well if we're saving the world for so many others we should also be looking and doing that internally so there's been a big push um you know over the past I would say three years around this topic internally and it came to the burning platform due to multiple reasons first of what I just mentioned um second is you know we've been measuring the question I have opportunities to learn and grow at indeed since 2017 and that started slipping um and we did find that it was a key driver of many different factors of our culture and engagement here um so it was a key metric to really look at so on the Quant side that really checked out with our story as we as we unfolded this story um and then also throughout all of our listening channels our dare to share on slack on all of the multiple um you know CEO bi-weekly meetings a career came up all of the time and so it really became a saturated theme for us to to focus on and and definitely became a burning platform and quite clear um we also found that became alarming over time as we analyzed the data is that in the exit hiring and onboarding surveys development was really the number one reason why individuals were coming to indeed because of that mission statement externally that we really wear very proudly and it really was the number one reason why folks and employees were leaving indeed um and so when the labor market became really tight um especially during the pandemic um this just again became magnified and a really important retention tool for indeed um so what we did was um I mean this became a full effort across the organization this again was not just an HR sponsored um go ahead yes HR was involved but um there were training teams that got involved from other parts of the organization Leaders with different types of mindsets ways of thinking um who came to the table and and really developed strong action plans um you know careers in Tech are really Divergent we have sales and Cs on the front line which have a different skill set versus those who are behind the computer and engineers and product developers ux um you know user experience researchers like Ben and I and it's really different ways of thinking about what does a career mean here it can be complicated um one of the most beautiful things indeed did is um they take a chance on people who have similar skill sets internally just as we do EX in our external world and give those people those platforms where you may not be in the perfect role or Department um but they do take a chance on you becoming potentially an engineer um and and how we do that is we really built and operationalized a very strong internal Mobility team um kind of career development coaching mentoring um the ability to invest in yourself with trainings internally of really really good training teams here as well as invest externally but I think the key there was making sure that there was a strong internal Mobility team that's not often the case in organizations um where that happens and so all of this came to fruition over the past couple of years and they've been in baby steps but it's really made a big impact and those score have have gotten higher and and people feel um feel like they can actually find their next career at indeed I mean even Ben and I are products of that right we came from internal listening and they took a chance on us to prove our skill sets um and and even showcase Even in our presentation today that there's so much a crossover between internal and external products and and go to market strategies and ways of listening and influencing decisions um so we're really proud of of this organ of of this Career Development Journey here and our contribution to that um and you know it's again you know we we thought of this as a product right this is just as it's go to market externally it's go to market internally we iterate we brought focus groups and people to the table to make sure we were designing it correctly and we're always have a focus on a lens on well how do we innovate how do we make this better um so it's always tweaking um and we'd like to give you another case study around what we've done externally here too yeah exactly so sort of talking about what we've done sort of internally in some of the multimodal listening channels that we use for you know internally growing that that learn and grow sort of pillar one of the the second case studies obviously like the external listening right in this case it was a change in in payment structure and so what ended up happening here was that a company we were we were working with decided how to change some of the ways the customers pay for their their services it resulted in an upcharge a customers uh based on you know perceived increase in some of the quality services again um using that multimodal listening channels where we have open channels of feedback we have our large surveys we have focus groups before sort of the changes even fully implemented you can already start to see maybe some bubblings underneath the surface of things that may or may not be going on um because when actually back up here a little bit one of the things that that helps us certainly is that it using those multimodal listening channels certainly allows you to to anticipate problems that are coming down the road before they really happen and so you can have your feelers out there for for months in advance because your your ears are always turned on you're always listening to what your customers are saying you're not necessarily placed into your reactive situation where oh no we made this change now our customers are upset we can sort of preempt some of this that being said after after some of the the preemption of some of the dissatisfactions were bubbling to the surface of this company decided to feel the large survey to sort of confirm so again it's that large sort of like satisfaction survey hey we're hearing these things are these things really real um you can get a lot of good data off of surveys but I think one of the things again sort of that to you the answer thinking about quantitative surveys is it always answers the what not necessarily the why and so turning to customer communities specifically we always know that through focus groups like we got a good idea of the what through our quantitative surveys but now we can we can sort of probe a little deeper for the why and you know that you can find that through focus groups a little bit better than you can from from just an open text on on your survey and so some of the actions that were taken from this sort of external listening was that obviously we have all these channels open we know the multitude of pain points from from customers and because of of the way in which the listening the listening strategy was set up we can bubble up things with Incredible speed it doesn't take you a billing cycle and drop in Revenue to figure out like hey stuff is going on um this company because of the the the listening strategy and some of the data that was provided back they're making substantial changes to the way the pricing model works and um a large portion of that was due to the feedback heard across different channels so again kind of summing up in in both of these you have sort of a ground up sort of building and external an internal listening and resulting in a sort of productization in terms of like learning and growing and finding your next career but this is your more typical sort of external listening strategy but one of the things I want you to keep in mind is that both employ the very very similar strategies so it's not like you need some sort of secret sauce on the internal side to figure out like what your employees need from you as a business again you don't need a secret sauce on the external side to figure out what your customers need from your products and services yeah and all these multiple listening channels again what Ben had said it's like we can anticipate strategically versus be reactive and and create Panic um through these like varying listening channels that we've developed internally and externally um and I think the future for indeed as we look forward both for internal and external is how do we more strategically create those listening touch points throughout um you know building a product from from inception from ideation Inception design and then go to market what are those ideal touch points with the what with Quant as we layer it on or with the why and with the qual um and how does that balance and that shift and pull um really work as we map that out um for our products internally and externally what is ideal and when and how and it's easier said than done it's not easy to operationalize something like that and with that um I think it's it's not about working in silos like you do have to really have you know a well-oiled machine in terms of having the right stakeholders and in the room to make sure you know things are rolled out properly so with that do you does anyone have any questions or um want to elaborate around what you're doing internally um does it does it resonate or not anything that we can help with so I'm not seeing any questions in the chat or through the YouTube chat but one question that I do have maybe um some folks are starting to gather their thoughts and typing now but um I'd like to know more about how you engaged internal employees and Leadership so if there's anything that you can share tips and tricks or just some of the initiatives to get some of that buy-in internally with employees and then escalating and taking things up to the next level with leadership as well that would be really helpful for us yeah yeah so one of the things that that we did to sort of get internal buy-in is you could always attach you know whatever your let's say engagement scores to like a particular outcome so if your company has let's say retention metrics that are that are pretty important um and you want to retain like let's say your your top talented individuals obviously then tying back sort of the fundamental sort of Engagement scores back to like retaining some of your top employees that's that's pretty standard but then that helps you generate some of the buy-in from from leadership and as you start to go in and say well hey your your folks are leaving because of you know let's say managerial issues and then you say well what specifically is about manager on a five question survey you're not necessarily going to to figure out like what is about managers what is about career development that are causing these people to leave and so then that's when you say okay now it's time for us to dig further we're gonna have to launch a little bit bigger of a survey we're gonna have to gather more data we're going to have to start building sort of more of the programs to go get the questions answered that you want and so tying your programs back to any sort of you know looking at key root causes of things that are going on within your organization is it has been in our experience a pretty easy way to generate buy-in yeah and it's to build it's it's not just about reporting and saying here are the numbers it's it's about building a story around to what Ben had just said answering the questions of why and um I I think that was really important because that why had never been explained to indeed five years ago when we started they didn't have that data and it was us you know getting curious investigating getting to know our our organization our people um what ticks what doesn't um on a macro level and then on a micro level and even Local 11 level internationally what are those nuances and what can we do about it so um I think just by building the story really helped and and again just building you know this goes without saying it's quite obvious like building relationships from the top down but also having you know there were some Grassroots efforts and indeed too and not like ignoring those voices too so it's kind of like bringing your Grassroots to the to the senior leadership and and you're kind of that middle man essentially um as well you have to play that role thank you um so I have seen two new questions come through the first is apologies if I missed it do you measure Dei metrics and if so have you seen a change over time with programs implementation yes yes no go ahead go ahead no you go then I'm good no no go ahead go ahead go ahead no no I'll talk about it just strategically and then Ben talk about a bit more quantitatively but yes so d e and I has been a huge effort for us um so with that you know again we productized um this is why I think it's a really good example so thank you for the question um we did yes we did measure it on on surveys um we we did you know deeper dives into into the eni metrics and of course belonging psychological safety um Etc but but with that the data just wasn't enough so what indeed did a couple years ago was really invest in in the infrastructure of building that team out so it's a very well-oiled machine now with dedicated members to certain business groups um we're looking at Deni not just from a survey perspective but on on like macro level big systemic processes so they're looking at Deni in their performance processes at what is talent development how does that factor into compensation how does that factor into building our products externally to folks so Deni just isn't a survey metric and belonging is one tiny piece of that huge Network that we have and making sure that we have Equitable hiring practices in our products that we are that are go to market that our customers are investing in is is like critical it's at the heart and soul of everything that we design and that philosophy philosophy of design is mapped into everything internally training all those other things that I had mentioned hasn't moved the needle I'll pass that on to Ben yeah so as Laura's saying have we seen a change in in those metrics over time with the program implementation yes we have seen an increase in Dei scores as a result of of the change in that program uh implementation one of the things that we always do is anytime we go to measure sort of Dei scores across our organization you always have to make sure your whatever you're measuring is psychometrically sound and you're getting the right data that you you need and so once we did that we were able to help the build assorted programs around the the things that they that they wanted to do and so we were very intentional with the way we structured our our surveys to measure the outcomes that they wanted to measure and so as a result of of the way that the programs were implemented we did start to see an increase in the high scores over time as Laura sort of mentioned earlier each part of the organization gets their own sort of look into what's going on in their organization they can compare themselves to other parts of the organization and then it creates a situation where yes yeah programs implemented but then also leaders can reach out to other leaders and to see what they're doing for for Dei within their own org so it makes a little bit more collaborative versus you know working on it just in a silo thank you the next question is how closely do you work with the indeed insights team um so indeed Insight great question and difficult to answer because how do I okay so indeed is a data-driven company so insights are their insights Team all over the place and that is one of the struggles for indeed is how do we make those insights less siled and how do we create like more of a strategic storyline for certain initiatives um so we our particular team our arm we are um we are an insights team so Ben and I run a research panel here which I think is quite unique um organizations like credit cards have it um I know Amex has a really you know strong research panel um LinkedIn has as a research panel too and what what do I what am I saying with research panel so it's basically a self-elected group of customers who say we are willing to um give you give you feedback and information around research topics that Ben and I are investigating so Ben and I get um mandates from our senior leadership teams and product teams here internally and a variety of different questions so we work very closely with product marketing on go to market strategies on even marketing language we've worked very closely with um design features of products we've worked philosophically on high-level discussions like what do what do our customers think about Chachi BT and that disruption in their hiring process end-to-end hiring process so Ben and I work in a variety of different types of topics and when we get a question or a hypothesis what we do then is we get to the root issues of what are we trying to um gain um and then we build a discussion guide we try to figure out our methodology is it Quant is it qual is it a combination um unique ways of gathering the data I mean I can go on and on about this and then we go to our indeed insiders research panel and we say to folks are you interested in XYZ topic and talking to us we then have put out a poll or we have one-on-one or focus group discussions and we build a story with that data and where that data goes is it it goes to sway a product decision it goes to influence um a market um a go to market marketing strategy and and wording of things so we um we are an insights team among many here um but Ben and I work on like kind of pretty high level strategic type of uh discussions um and do you do you want to say how we partner with other insights teams Ben and how we kind of bring it home yeah so one of the thing is that in terms of insights team with indeed we're always very collaborative with one another so we might have access to the panel through indeed insiders but then other insights teams might have questions that they would like our insiders to answer and so we're always pretty good about partnering with other insights team to create sort of a holistic look because we might have you know more more product focused researchers on like the very mundane portions of the product that might want to know you know if I implement this button here how is that gonna affect um you know perceptions of of this thing that I'm trying to do and so um partnering across organization allows us to again sort of create that different listening channels where you're tackling the same problem from from multiple angles so while we generally work on some of the high level stuff we do partner with teams who work on some of the the more you know focused things about our products and services I hope that answers your question yes sorry it's a little great thank you I am not seeing any further questions on our YouTube channel or in our chat so um Laura Ben thank you so much for your time for presenting this was very interesting um engaging and very helpful I'm sure our audience really appreciated it and um if it's okay I'm sure also people will reach out to you through Linkedin or other sources to make that connection with you so thanks again absolutely please reach out we'd love to help thank you so much everyone thank you thank you bye