Best Practices for Using Crowdsourcing Platforms in Survey Research: The Case of Amazon Mechanical Turk

um welcome back everyone uh we are all set for our next presentation in the july 29th accelerated research virtual insights conference uh next up we have dr chihan chobonolu i hope i i hope i got that pretty close absolutely no problem with a really uh compelling presentation that i'm excited to hear about and that's outsourcing uh for market and academic research um that's about it i'm i'm not gonna stand in the way for too long i'm going to step out of the way and let the presentation get started i'll circle back for uh for q a at the end um but otherwise the floor is yours thank you thank you bill so much i i appreciate you inviting me here uh it's a privilege it's an honor and you know i came across to this conference series that you are organizing by google search and i myself enjoyed and learned uh quite a bit from the previous speakers so then we got mad and and i'm happy that i'm also part of it so as well too so thank you so i'm gonna go ahead and get started uh with my my presentation today uh is two piece one is i'm gonna talk a little bit about the theory part of the using crowd sourcing let me share my screen with you so that i can uh show this to you i don't think i have a sound but i'm gonna still so hopefully you can see my presentation now that yeah perfect so this is best the way that we title it best practice for using cloud sourcing uh platforms for um survey research mainly social sciences the case of amazon mechanical turk as we have kind of like made this the title and i will tell you what that means you know one of the things that that we do as the social scientist is collect data obviously right um survey research we do use a lot of it and the you know i got into academia in 1996 i am 50 years old at that time we used to collect our survey data on paper uh we would you know type some of you uh who may be listening this presentation remember this if you are in my age or even older that we didn't have the internet we didn't have these mobile phones we didn't have mechanical turk amazon any of them no google no facebook imagine those days that there was even hardly cell phones so we were collecting these surveys on a piece of paper we would mail them we would wait for one week two weeks for the mail to arrive and then for the respondents to respond and then it will take one or two weeks for them to you would usually put a self-addressed and self-stamped envelope inside again if you've done it this way you you know exactly what i'm talking about and then you would just um then you know code this into a statistical software like spss or sas and then you would just um analyze the data that's how we did it then some people use facts and then of course 1996 1997 2000 let's say the internet became very common to collect data how did we do it before before crowdsourcing concept is that we actually email people we invited them would you please answer my survey we are doing a research about this of course you go through the irb internal review board approvals in your university when you do research and then after that you invite people or you now the social media you put on social media you hope for people to come of course they are um convenience sample in all of them but then this idea of uber you know what did uber do uber actually introduced the consumers the the passengers with the taxi drivers it's a mid midway right same thing with airbnb airbnb introduced the travelers with the people who have extra room or a house to to rent so same idea uh crowdsourcing is actually bringing the people who would take surveys and the researcher who would like to would like to survey takers to take the survey to meet them in the middle so that's what crowdsourcing is actually all about so um i um am currently let me just tell a little bit about myself so you understand where i'm coming from i am uh mckibben endowed chair professor in the school of hospitality and tourism management at the university of south florida our school is in the moma college of business and i also am the director of m3 center m3 center is a center for research for hospital technology and innovation as you can see here and we do have a lot of different services um that we have so please feel free to go visit our website m3 center dot org and also when you go into the m3 center you will come across with three journals open access journals that we publish which i invite all of you to submit articles uh journal of global business insights the first one and journal of global education and research and journal of global hospitality and tourism so again they are open access 100 free of charge so okay let's get to crowdsourcing as i introduced you a little bit earlier it is uh if you were to define it it is uh defined as the um collection of information opinions or other types of input from a large number of people typically via the internet which may or may not receive financial compensation so there's different ways go fund me i'm sure that you know that it's the people who would like to get something uh money donations can get like this or um what was the name of that sign up um that you get signatures i forgot now the name but there is a variety of different different things the use of internet services for hosting research activities and for creating opportunities for a large population of participants is the type of crowd sourcing that we are interested here again this is for the researchers um to be able to do that and um oh by the way if you have any questions or any comments feel free to put them in the chat box and q a i'm happy to entertain them while i'm doing this presentation or afterwards no problem at all so the what is crowdsourcing research uh for example you can have online reviews forums wikipedia qualtrics mechanical turk craft flower prolific academic i think this is a new one so these are all different crowd sourcing tools so to speak in terms of uh research mechanical turk is what we are going to focus today as a case study you know it's not because it's the best one it's because it's the one that i'm familiar i use mechanical turk on a monthly basis i will explain to you why i am using it and what in what conditions and then we will uh obviously there is qualtics for example which is also a survey software which i use by the way when i'm doing mechanical turk as a crowd sourcing for data collection uh or other forums as well to crowdflower i didn't use the other ones very well i didn't use qualtics simply because of the cost uh qualtics is quite um expensive as compared to mechanical terms there is ups and downs this presentation is not about uh comparing qualtics to mechanical turk that could be the maybe topic of another discussion but but crowdsourcing what can it do for you as a researcher how could this help you okay what are the advantages some of the advantages of crowdsourcing you uh it offers you a large participation poll how can you actually find people from all around america from 50 different states uh if you would not use this one right it's difficult you can do it social media snowball uh recruitment you know you ask people they ask their friends and etc so you can try to get as much as possible but um but but it's not easy of course it's a streamlined process usually it's much quicker participant recruitment study design data collection is all one and i'm going to show that to you in the second part of my presentation because we will do an example actually integrated participant compensation system so you can actually pay to these people directly from mechanical turks so it is almost um very easy i i'm gonna actually show you it can be easy and of course like anything else it can be easier to detect possible sampling biases in this one and i will show you some of the best practices as a matter of fact we titled this presentation um after an article that myself and mohitin chowsholu one of my uh former students now my colleague of course and dr goes that took tahan we have written an editorial article in the journal of global business insights which i will give you the link a little bit later uh feel free to go and look at this kind of like what i am telling you here is also listed on that article even in more detail and um usually it's much less costly so it's affordable uh again affordable is a relative term like for some people uh 50 cents per person may be cheap for a researcher in america but it can be very expensive for another person in another part of the world so i'm not gonna claim that it's cheap but again at least for american standards it's it's not too bad um and it's a huge market i mean this is really intelligent uh whoever come up with this idea um global crowdsourcing market is expected uh to uh 154 million dollars by 2027 so there is a lot of people are using it not just the researchers by the way it's not when i go into the mechanical turk i'm going to show you different kind of crowd sourcing ideas that you can do not just collect data in other words it's not only to give them a link for them to fill out a survey there's many other things that you can use this market researchers use this let's say that you want to produce create another new item to the market you want to test the colors which color is more appealing to the consumers than the other one you can use uh crowdsourcing you know like mechanical turk or qualtics or any other tools so there is many many uses of it mechanical turk is actually a startup i really don't know too much detail about the company rather than that it was actually a startup but then it was bought out by amazon uh for you know probably whoever come up with uh made good money there good for them right so it's a crowdsourcing marketplace that makes easier for people like me researchers or even businesses to outsource their processes and jobs to a distributed workforce who can perform these tests virtually actually you know it's very interesting that amazon mechanical turk is even used by accountants you know they scan these receipts and then these people whom they uh hire from amazon mechanical turk actually look at that receipt and they enter that into an accounting software so there is even activities like this one so it's not just like think about only research but there is also other parts of it it's reliable and cost-effective tool that's some of the advantages of mechanical turk and providing representative data for research in behavioral sciences this one let me just qualify what i mean here representative data in the sense of general public right mechanical turk is available in select countries not all around the world uh i i i think that people from all around the world can sign up to take the surveys but for you to give the surveys see there is a recruiter and there is a also a survey taker right there are two parts here mechanical terp combines them together so that's not available in every country uh and we'll talk about that as well too um in a second when we get there or uh between 2012 and 2017 more than 40 percent of the studies published in the journal of consumer research which is a top journal by the way use crowdsourcing websites for their data collection so why i am saying this is that this one mechanical turk or crowdsourcing if you do it uh with precautions like you take the actions to be able to make sure that the data that you collect a fits your study if you are going to do a study among ceos of fortune five of cup fortune 500 companies and use mechanical turk i can guarantee you that no journal will ever accept your article because obviously mechanical turk is not a place where ceos of fortune 500 companies are going to go and do it for 50 cents 30 cents uh i'm not saying that this is uh not serious money but you know there are people who are doing this as a job i mean they are going and taking these surveys on an ongoing basis making pretty you know decent sellers especially people who are in developing countries it can really add up so but yet if if you take some precautions which i'm going to explain here that you can really have this paper accepted by a quite high level of journals and of course there are some disadvantages possibility of unsatisfactory data quality what i mean uh by this is that if you were to uh do it without taking any precautions that people may just collect i mean uh reply to your survey without really their knowledge so if you're looking for accountants mechanical turret may not be the best place i'll tell you what is the best uh kind of demographic of survey takers uh is general public if you are going to do a study for general public american public united kingdom public or world public this is great what i mean by this general consumers so if you're doing a study about travel this is perfect if you are doing a study about shoppers online shoppers mechanical turk is perfect if you are going to test an advertisement let's say youtube advertisement you've got six seconds of um advertisement versus 15 seconds with the option of skip you know like i'm making this up by the way and you want to find out which one is more appealing to the consumers this is perfect but if you have a very specialized sample it may not be as perfect lack of monitoring and data quality issues is another disadvantage however even though i make this i list this as a disadvantage but there is steps that you can take to make it better so uh and we'll talk about that in a second issues of trust and quality of the research findings is an issue for some journals but again if you were to take some of these precautions you may get rid of these trust and the quality issues and of course these people are experienced survey takers so you need to again take some precautions to be able to make sure that they are indeed are um reading your questions so you need to build in your questions in such a way that that that you ensure them there cheaters speeders fake respondents are all part of the real world uh they are here and most likely if you were to do a survey collect data with mechanical turk or other crowd sourcing tools chances are very high that you are going to get some of these people and you hopefully will build your survey in such a way that you're going to get rid of them you want to kind of like clear them out filter them out so that your data is pure and nice as as accurate as reliable as trustable as possible so some of the best practices uh entered worker as we have said uh respondent selection so you need to just like i've mentioned earlier that you need to actually uh select the right sample for crowd sourcing so whoever that is again i will repeat one more time once i you know i i didn't tell you in the beginning but i am serving as the editor of three different journals one of them is journal of hospitality and tourism technology which is published by emerald and it's an ssci journal index in ssci and scopus with a 4.2 impact factor we do get crowdsourcing articles what i mean is that we get articles that have used crowdsourcing as a data collection method so obviously i i'm doing this presentation that means that i probably accept those articles which is no problem however i look at the um the the sample so if the sample is very specific and then they use mechanical turk that might be a problem right so if you are looking for people who have been to a small island in indonesia chances are high not high that you are going to find them in mechanical turk right because it's from across the way uh but if you are looking for people who visited a country let's say a turkey that might be a place where you can do it but again you need to do some actions you need to ask some filter questions to be able to identify if these people are really the truth or not so one of them could be potentially is to first ask have you ever been to turkey for example i'll just give you an example if they say yes and then ask them special questions that will uh get into the details what is the city that you have uh visited or cities that could be one uh what was the date and what was the top one activity even though this may not be part of your survey may you may not be interested in this but yet if you ask those questions and if somebody is a cheater they will probably write um unmeaningful things there which you can then filter out later uh in terms of survey validity we are going to talk to you i'm going to show you a couple of examples actually you know what why don't i show you what i do uh in actually i collect data from american travelers on a monthly basis uh which i call american traveler confidence survey so each month i do this since july uh 2020 it's because of covit so i collect data and i use uh mechanical turk for it and i will actually show you what i do and how i do it in in few seconds researchers should consider their student population study population as we have said and also you can use selection tool in the software so if you want to get people from america in mechanical turk there is a way that you can say i want only american people to order people in america to answer this you can select a location once you select a location you can even look at the ip addresses because not everybody is honest they may say that they live in america they may have an american email like gmail for example but yet they may live in other countries so then you look at your ip address there is a tool which i use the ip address then if there is outside of the united states we just simply delete them out from there so also um consider the demographics of workers on mechanical turk which must reflect the study target population we advise that you use multiple screening tools that identify quality respondents and avoid problematic response patterns and include screening questions in different places and i will show them in my study how i do it i uh some of the best practices i have i ask the same question two different times in due to two different parts of the survey to ensure that people actually do uh tell me the truth that gives me and for example just to give an example instead of asking how experienced are you with accounting practice let's say you're doing something about accounting you know this question the first one that i'm i'm talking about here let me see if i can actually uh outline this one so instead of asking about this particular question here that a better question is which of the following is a component of an income statement so in other words if these people are ready in accounting they will understand that this question that you are you are asking them so this is one example survey validity conduct a pilot survey from mechanical threat workers to identify fix any any potential data quality and programming problems before you actually collect the entire multiple ready check tools into the survey be there be aware of internet bots because these people you know the cheaters are also intelligent that they are using internet bot which is a robot uh it's software it's automated task just like randomly select the survey results so there are again something that you can do in quality survey when you design the software that will detect the internet bots which is butt detector you can do that and that will also eliminate that problem you can also do time limit options for example if somebody were to if your survey takes about 10 minutes on average and if somebody replied that survey in two minutes or one minute you can easily find it out and then get rid of them again that's one of the other things that you do to increase the validity in your survey and obviously you can collect data in different days and times during the week to collect more diverse and representative data and once you collect the data definitely you need to clean because there are some people who do not read the questions and you can um you can get rid of them especially the ones that you built like for example one one of my favorite is i ask people in the beginning of the survey is how old they are i ask them their age so they write 30 35 40 42 whatever it is and obviously if they were to write anything uh younger than 18 that means that that's not good because they are also rules is that you have to be over 18 and you can also put that in your survey that you have to over 18 to take the survey so if anybody is less than you'd definitely delete them as well and i asked them their age then i asked them also at the end of the survey what year were they born so if somebody if we are in 2021 so if somebody said that they are 20 years old so they need to say that 2021 so they need to be born 2001 right if they are say that they are 20 years old so then i compare that that age with that one so if they said 20 years old but they put 1970 as their birthday so that means that they um lied somewhere they gave incorrect information some point of time so you will actually get rid of that that particular and i always say the rule of thumb is that collect 30 more data than needed if you need 300 uh data you need to collect about 400 because you are going to clean it up usually about 30 percent goes into the waste basket and calculating a response rate for the samples using mechanical turret is not possible because you really don't know it's not like you have a population uh or sample of one thousand people and then you got two hundred survey spec so you say twenty percent however it's possible to calculate active response rate it can be calculated as the row response numbers deducted from all screening and relative check question results so let's say that you got 1000 and you're going to see that actually i usually get about 1200 but i end up about 1950 clean data here is the article that i mentioned to you that myself uh dr cho choli and dr han has written this is an editorial it's not a peer-reviewed article but you can actually find it i will give you even easier website than this one you can either scan this qr code to go there or you can go to m3center.org m3center.org and actually access to this data uh this article as well from there and so let's see it in action so i'm gonna uh show you how this actually uh works so let me stop share my screen and i'm gonna actually show you again another of my screen in um in a different way so let's do it uh i'm gonna actually go and click on igniter okay so let me go back to zoom and meanwhile if you have any questions please let me know happy to to entertain them okay so i am here the first thing that i'm going to do if i would like to create a crowd sourcing survey research is to do my my survey right to create my survey you can create your survey in different ways qualtrics is one way you can do it in surveymonkey google forms you name it my university has a site license for qualtrics for that reason i uh use qualtics actually i used to use qualtics even when it was came in the beginning i think it was like 10 uh 15 years ago even more maybe i remember as um so i am going to okay here is um qualtics right so this is this research that i'm going on right now there is quite a bit of them so let me just show you traveler um confidence index survey that i'm using right as you see right now i've got about 9 000 responses and i've done it different versions so here's my questions right this is my irb my university requires us to go through a approval process for our surveys and um and then we asked the people where do they uh stay uh reside in which state to be able to understand look at here this is one of the valid questions that i mentioned earlier what is your age remember this question what is your age i asked them if you look again if you know politics obviously i'm not going to go in qualities to teach you how to create a politics survey here uh although that's another great topic for those of you may not know but i'm forcing this response to the people i want them and i want them to give a number between 18 to 100. so i don't accept anything less than 18 that means that if they are they are not going to take it so more than 100 so if they were to let's say fake a number okay that's just for the sake of example that somebody is really um really 20 years old they put here 25 and then if you go to the end of the survey okay and here i ask this question that i asked earlier again one more time let me find that question what is your birth year you see this is exactly the same question that i asked earlier but in two different formats this is the birth year so if somebody said 20 they need to put 2001 here i compare this data with each other if they are one to two years apart from each other i accept them because some people may be born in december some people may be born in january for them maybe they are 20 years old or they were born in 2001 or 2000 or 2002 right depending upon the the months one or two years um you know variance from these two numbers is not a problem right they can do that and also what else we do again this is from there what else i do is that i also add for example another uh attention check question what is 2 plus 2 please choose 5 from the choices below again i am checking to see if these people are reading the questions in its entirety and you can also put other questions similar to this one in there such as this one what's the color of banana choose red one more time that you are doing this because you are paying money to these people that you want them to actually uh do it so there is um other ones as well but i'm not going to go over them with you but if you were to design a new survey there is one thing that i want to show you which is actually you need to create this survey so let me just create a test survey here for mechanical turk i just click on create project there is also you need to connect the survey in other words when the people the survey takers mechanical turk take your survey they need to get some kind of a proof that they have done it which is actually the code that you are going to display here i do not suggest that you display a static code i recommend that you show them a dynamic code it's a code that changes on a ongoing basis so that you can actually really check because again these people are very clever if they recognize that you are using a static code they can open many different accounts in mechanical turk and simply add that code without taking the survey hence getting the money from you and how much are you going to pay to these people i usually um make my payments based on how long they are going to spend and based on the hourly rate in america so if you were to let's say i think it's eight dollars uh 50 cents in america i'm not sure but i usually calculate from 10 an hour so if they're gonna work for 10 hours so you calculate and you pay accordingly but this particular um this particular let's say that you are writing the question you are just telling people that welcome uh to this survey and you are asking them that please answer the questions below carefully of course if your university requires you to put that language that you've seen earlier as my university does you can put that carefully and you start writing your questions so whatever that your questions are multiple choice are you uh for example a online consumer or do you shop anything online let's let's say that you are looking for you are looking for online shoppers okay so you will say yes here no and then there's yes or no question right you can easily uh come here and i can uh actually um create a you know this this qualities have changed uh the way that they do it so i'm gonna put a skip logic here if i wanted to if people said yes uh no is selected i oh no no not skip actually i should say is it skip sorry okay you know let me do one more question here i will do um how often do you purchase something online let's assume okay this is just a makeup questions just to build a survey for you so that you can understand somebody um yeah you know trivial coding for exist for a bot this may not be like somebody john has written a comment that um age check and year um that's not easy to do for a bot right bot can put there a random number but they will not be able to read that that's the birth year and the age that i'm asking also i didn't show you in the previous article but i'm gonna on the survey but i'm gonna show you here let's say once a week again i really don't um write these questions um knowingly right uh once a month this may not be the best scale but whatever that is so if somebody were to do not qualify for your choice then you can so skip this question is how often do you purchase something if they say yes um you know hold on a second i want to send these people oh display logic okay okay here we go welcome do you shop online if they say no then it's selected then this flow no no sorry about this you know quality exchanges formatting earlier i am trying to remember how do you send people who say i think it's the escape logic so this is the is selected not really anyway don't worry about that that's really not the point here uh so somebody again asked for a but you can also add a question here which is called um captcha verification this is something that robots are not gonna say please confirm that you are not a robot when you put this question actually let me show you what it will look like so you can see i just click on publish and i come here and i'm going to copy this link just to show you here it is how it looks so this is the survey we just created you shop online and you see please confirm that you're not a robot so it shows you these kind of things you need to um check all the traffic lights one so if that's correct then you can confirm that you're not a robot right so this is something that you do but this is the easy part of it you all know how to create a survey then we need to connect this to mechanical turk so let's do that so i'm gonna go to mechanicalturk the first thing that you need to do for mechanicalturk is actually to create a um to create an account as a requester you can also be a worker if that's interesting to you you know it's a very good learning experience to see other surveys other researchers what they are looking if you're a worker there's a you know approval process for it so you need to do it but we are more interested in the requester so i'm going to click on requester i am already in there but if you are not you need to create a new amazon account so you need to do this and after you create this amazon account that you will be able to actually log in into um into mechanical terms so i'm going to just do this jihan at usf.edu is one i use just for this to show you here it is it looks like this this is a brand new page mechanical turk this is how it looks you see i was telling you earlier that you can actually do many different things in mechanical turk by using this crowd sourcing population so what we are doing here we are trying to create a project for the survey takers to be interested and take the survey so you can do sentiment analysis you can ask them things intent detection this is about language uh you can do data collection website classification you can ask them to go to your website ask them to do uh classification or collect data or data collect go and ask them to collect for example information from hotel websites or businesses whatever that might be there is a lot of tasks right these are i think they call it hit human intelligent tasks so something that humans will do image contains image tagging i am really not too familiar with different types of these research the one that i'm the most familiar is surveilling so you can actually create a project so i come here i click on create a new project so here i'm gonna click create a new project so i will click in here let me look at my timing i want to make sure that so let's do one quickly okay so we'll say um test survey this is they don't see this you see it so answer a survey this is what they are gonna see this person when i say date that's the survey takers so all these uh crowd out there who are willing to take surveys for a fee right you need to give them a reward you need to give them an incentive which is usually the money here which comes to them as an amazon gift card so answer a survey about your travel behavior imagine again i'm just making this up right so i am uh doing this about travel your travel behavior so you put the keywords here if they are interested um in this maybe you can put travel you can put hotel restaurant if your question will have questions about that okay here's the important part how much do you want to give to these people for the survey that they are going to take for you again remember how long it will take the survey is going to impact how much you're going to pay if a survey is going to take 10 uh to 12 minutes usually something about 40 cents 50 cents is reliable don't overpay uh don't underpay right so this is something that you can test over the time that's why it's important to do a pilot first right just candidly take your survey or ask others to take the survey how long does it take if it takes four minutes maybe paying 15 cents might be enough right um or 20 cents whatever that might be let's say that i'm gonna pay 40 cents how many response do you want if this is the first time that you're doing it it might be a good idea to do a a simple survey for 50 people but if this is something that you're going to collect the data for full study and you're pretty comfortable about it that you may want to put here 500 if you're looking for 400 data clean data for whatever that is so if you are doing a general american public you need about 1 000 um data so i put 1200 how long are you going to give them one hour i usually do not touch this default here is the important one worker requirements this particular one if you want their masters so if you want really qualified people the some of the survey takers become masters after they go through um different surveys and they get satisfactory uh uh points from the survey givers right the requester in this as they call it so obviously if you do masters it's going to cost you more money i usually do not do this because i clean my data instead i go and put the location let's say that i am interested in only people from crochet if this is what you want obviously how many people are a member of mechanical terraform grocery is another question mark right probably not a very good idea especially if you do the survey in english not only that they need to be on the mechanical turk but they also need to speak so so these are the ones that i'm telling you that you need to be careful so for me uh i am pretty comfortable in in america so you can even target a state if you wanted to do so or you can just do united states or united kingdom as a country so here this is good for me and also i'm going to add one more thing that which is hit approval rate so the people who are going to take your survey you need to approve them and i'm going to show them to you in a second if this is the case if you need to improve them if you don't approve them these people are going to get percentage so you want people uh who have been approved more 90 percent of the time sometimes people are not approved for good reasons like not because they cheat sometimes they miss the code or something so that's why i put usually 90 which will take this means it's going to take longer for you to collect data especially if you don't pay too much money uh but it will make it better so i go to the next step survey uh instructions this is what people are going to read when they click on your survey so you will write here what is the requirements travel behavior so you can just say in order to to take this survey you must have traveled somewhere in the last 12 months you can write anything and you can say that please note that i usually put this that there are multiple validity attention check questions in the survey i write like this if you do not reply them correctly you will not get that and then of course you need to put your link there do you remember my link here i'm gonna go in here so here's my data uh no no i clicked on the wrong place actually i clicked on distributions so i'm gonna get my survey uh link here this is my survey link i'm gonna copy this go back here so i'm going to put this in here this is the link that you need to give and they're going to put the code here after they take this survey they're going to take the link here and how they're going to take the link i'm going to show you in a second because we need to give them a code obviously i'm not going to do this one but then you just simply click on finish and then after you click on finish you can just say test survey and you will be able to publish this match if you publish you need to have money in it in this account i don't think i have any money so i'm not going to actually publish this i just want to show you what it's going to look like see this is the cost i i said i need 1200 people i'm going to pay them 40 cents each person so 480 is going to go to the survey takers and of course amazon mechanical truck takes a commission they have actually a fee so they take like 16 cents for each of them and there's more details about pricing if you're interested you can look at uh in these places for us it's not matter right now so of course you need to put a credit card here for you to be able to make the payment for see if i pay somebody 40 cents and 1200 it's going to cost me 672 obviously this is quite a large number and you may not want to maybe uh shorter survey so you may pay less and uh there it's going to be so when people take it so let me just sign out here and i will show you uh i'm going to log in as jhan.org which is my other account that i usually use to collect data so here is my confidence if i go to click on manage you will see all the data that i have been collecting so this is the travel confidence 1003 people have collected this completed the survey and i can actually see the results here right now i don't see it but i can actually easily see here as people take it uh that i will be able to uh do it as a matter of fact uh we don't have too much time but now let me go back to politics let me show you how this code is is actually processed so so remember we need to give them a code right when these people take as you remember right now this is a very simple survey if anybody wants to take this survey they are not going to see any code they will come here another robot traffic lights i will click on here i'm going to verify this verify nothing so how will these people tell you that they've done your survey and get their money so to do so there is one way there is many different ways right you can actually put here a question right one question you can say here is your code for mechanical work and you can actually do um for example go into excel i'm going to go into excel i can actually create a random numbers so here is i think i click too many times i can click make some random numbers random numbers like this some of you may know this code okay here we go rand between random number between for example i usually put ten thousand and ninety nine thousand nine nine something like this in between so you can do it this way here is i'm gonna generate some numbers okay this is just to show you okay you obviously you need more numbers than this one but you can just generate some random numbers go back to your survey and copy paste these numbers here as you can see there is many if i were to publish this right now and it's going to actually show it all numbers together but i don't want to do that how i can do this is let me show you back again i'm going to randomly show one of them so i can randomize choice randomization i can say only one of them just randomly one of them so you can put here 300 400 codes and it's going to show only one of them and let me show you how this works and i want to put maybe add a page break so that people will not see this code before they answer the survey let's see how this works what i just told you how it works this is one way there is also another way which i'm going to show you one tool that you can learn so i come here shop online once a week i'm not a robot uh fire hydrants so i need to click on all of these to verify myself and then my code should come as you see here people can get this code and put it in in there so there is other ways that you can do it the way that i do it i usually put some kind of a name in front of my code like this one i usually say uh i use concanakin8 i don't know if you know uh this particular um tool looks like i'm gonna have to wrap up very quickly so you can actually say cc and then i can put dash here sorry i can put dash comma i can put this number that i take and then i can put one more code here like this so this way you can make it like this i'm gonna put also a minus here as you can see i can make my code like this so it's much easier for you to understand the number random number still changes as you see but in the mechanical turk you can identify that this code is your code without checking that if this is really code or not right so once you do this uh you can actually come back here and click on the review results as people take it and then as you see this code the codes are gonna um actually show here as people take it then you can just simply click on approve them if somebody didn't take your survey and they put a gibberish code that doesn't look like this okay then you can just simply um get rid of them you can just reject them one more tool for you i would like to show you if you are interested even more i did a very quick job if you go to mooc.academiacentral.org which is the mooc platform that m3 center pub does here there is a course about this it's free course just like this one very similar to this one a little bit more detail so it's about two two and a half hours in total so if somebody any of you is more interested to learn about crowdsourcing feel free to go here register for free and enroll in this class you will be able to see much more you can even take a certificate if you are interested so this is in a nutshell what uh qualitic uh mechanical turf crowd sourcing is uh i'm gonna stop here we have about five minutes to see if if anybody have any questions um whatsoever i'm happy to take them uh if there's any questions if not um that feel free to experiment with this it's really good very effective works for me for more than about three four years that i'm collecting i publish papers in some of the top top journals by using this by taking you know of course when you collect the data you need to uh clean it up like this age one that i told you is a good one the people that you asked them to uh answer this way but they didn't answer so you clean them up you get get rid of them so the end data is usually quite reliable bill back to you ah that's fantastic um yeah i think we're running up against the top of the hour so so not a lot of time for for q a um we are having some good sidebar conversations in in the chat uh both in zoom and on youtube uh you know some debate about survey platforms and sharing of best practices um you know as you mentioned you know some of those quality controls that are really important not just when you know tapping into crowdsourcing but even you know expensive purchased sample um for jobs as well um the recommendation for any survey that's being executed and any source that you're tapping into um put those quality checks in place make sure that you know you're scrutinizing the day that you're receiving because yeah i mean the truth is there are bots and manipulators out there trying to to gain the system exactly but no this is great you know i think liz sort of provided us a book in her initial presentation this morning um just talking about overviews and platforms and how to you know apply some of this in an organization jihan this is a great you know very you know in-depth uh kind of exploration of one particular way to to go about sourcing and you know putting some of these best practices in place um absolutely awesome you know like i said not much time for q a so we're going to go ahead and rap anyone does have additional questions you know feel free to log them in in the chat in the q a um we'll make sure that they find their way to to jihan and and you know we'll get those answered for you but this is great really thank really appreciate course bill i also shared my uh contact information here my email my website m3 center website and the mooc if anybody uh who may be watching this live or from the recording later that they want to learn more feel free to go to mooc academy central.org and enroll in that course for crowdsourcing and also our article that you can download from m3center.org outstanding thank you so much thanks for inviting me and thanks for the good work that you're doing