In this design specialization we've talked about crowdsourcing and social computing, but what is crowdwork actually like, what is it like on the ground? Today we're here with the real experts to give you a window into crowdwork. WIth us today we have Lily Arani, Lily is an Assistant Professor of Communications, at UC San Diego. And joining us by Skype is Rochelle LaPlante. Rochelle is a worker and advocate on Mechanical Turk and other crowdsourcing platforms. And they're gonna share with you what crowd sourcing is really like. >> I thank Scott, thank you Rochelle for joining us today. So I thought we could start by talking about how you got started on Amazon Mechanical Turk. >> I got started in about 2007, and I heard about it from a friend who was working at Amazon at the time. And just kind of mentioned it as something that was a project that he'd heard about at work. So I checked it and got interested in doing some of the tasks in my free time just to make a little extra Spending money here and there. And as I got more into it and time went on and my situation kind of changed I got into it more full time and more seriously, and kind of have developed it into more or less a full time income for me. As well as the work I do helping other workers, helping requesters and helping everyone who is involved to learn more about the system and make the best use of it as they can. >> Great. And can you talk a little bit about what your involvement in the Turk community is? >> In addition to being a worker, I am also administrator and moderator on one of the major forums, mturkgrind. I am also moderator for the Reddit community which is /mturk. I also just gonna help out whenever questions come up, if a requester has questions about setting up a hit. I might get an email from them about that, just kind of helping out wherever I can and on whatever comes up in the Mturk community. >> Fantastic, so you've talked to a lot of different workers, you've been in the community for a long time and you're part of the communities that make Mechanical Turk work behind the scenes and behind the platform, that most people think of when they think of Mechanical Turk. >> Yes, and I also keep up on a lot of the new research and studies and other things that come out, on Twitter. A lot of the discussion happens on Twitter, so I'm pretty active there on Twitter too. By talking about Mechanical Turk and the issues about crowdwork there. >> Would you like to tell us your Twitter handle? >> It's @Rochelle, R-O-C-H-E-L-L-E. >> There you have it. So you can go to one source of knowledge on Mechanical Turk, where you can find online after your course is over. So could you give us some background on who's doing crowdwork? Why they're doing crowdwork, and where they come from? >> According to the most recent research that we have, about 80% of Mechanical Turk workers are from the United States. The remaining 20% are mostly from India. There's a few from other scattered countries around the world, but The majority are from the United States. And people get into crowdwork for all kinds of reasons. That might be because someone's laid off and they're doing this work in between finding a new job. They might be a caregiver for someone at home. So, in their spare time in between being a caregiver, they can do some Mechanical Turk work. Other people have situations where they've been unable to find jobs in their community because of discrimination, because of disabilities, because of their past background. So it's kind of a wide conglomeration of people who come to Mechanical Turk for different reasons. And it really fits into their lifestyle and their situation at the moment. >> What do you find motivates you and other Turkers you've spoken with? Especially ones that stay in Turk for a long time? >> I think the ones who are more serious about it and who stick around and who do make a job out of it are there primarily to earn money. I'd say that the majority of people I talk to are there, who are working are there for earning money to pay for rent, for food, for bills. There's a lot of people who are there to pay for healthcare related expenses. Loans, debt, student debt, all of those kinds of things is a major motivating factor. >> Great, that's really helpful. So what are some examples of tasks that you've done on Mechanical Turk in your time working in the system? There's all different kinds of tasks and it changes every day. There's everything from audio transcription, there is tagging photos. There's sentiment, which is looking at a phrase or a sentence and deciding if it's positive or negative towards a company, or toward a specific idea. There is writing projects where someone needs some text written to describe a product, maybe. There's editing HITs for, you can edit some written text. And there's also a lot of academic surveys and studies where university academics will post surveys and studies that you can complete to help them with their research. >> Okay. So how do you decides what tasks to do when you sit down to work? >> There's some requesters who post frequently, and so I definitely have a list of my favorite requesters that I go and look for to see if they have any work up, or any work available. I also look at the price that someone is paying and see if it's worth my time. If it's a job that I think is worth my time for the money that they're willing to pay. I also look at things I'm interested in. Find jobs that are particularly interesting to me that sound like an interesting topic and will gravitate toward those. I think, also, the last thing is, and probably the most important is, that I look at the requester's Turkopticon rating. Which is a rating system that workers use to review past requesters, write about which ones are safe to work for and which ones may not be so safe to work for. So, I look at those ratings to make sure that whoever I'm choosing to work for has a good history. Or, at least is brand new with no history and has potential to be a good requester. >> Mm-hm, since you mentioned Turkopticon, how would you advise requesters to be good requesters with TurkOpticon, and with workers? >> I think the requesters who get the best Turkopticon reviews are the ones who pay fairly, and who communicate with workers. If a worker contacts them with a question, they're responsive. And that they pay fairly and on time. And they don't take an excessive amount of time to pay. They pay rather quickly and fairly. And if they approve work that's fairly done. And don't reject work without having a valid reason for rejecting that work. >> Does having a requester communicate on Turkopticon make a difference in the comments, having them come respond to reviews, for example. >> Yeah, I think as long as the requester is communicative in some manner, whether it's commenting on Turkopticon in the reviews or emailing someone. If you email the requester with a question, and you get a response. No matter which method the requester uses to communicate, I think having some kind of method of open communication, where you're getting your questions answered is really helpful. >> Okay. So as you're mentioning the importance of open communication between requesters and workers, what kind of advice would you offer to people to be good communicators with workers more generally? What kinds of tools, platforms, techniques, and strategies, would you recommend? >> I think that what's really successful is when a requester comes to one of the Mechanical Turk forums. There's a couple different forums that workers are on and most of the forums have a section for requesters, where the requester can come and create a post and introduce themselves and then actively interact with the workers there who have questions. That's probably a lot more of an accessible way for requesters to answer questions there, rather than dealing with incoming email and answering the same questions over and over. >> If it's all in one section of a forum, that's really helpful. So I think that's a really good tool to use for investors to improve communication with workers. >> So concretely, if I were an employer and I was wondering do I just show up at the forum and make a general post and say, hey, where do I go? Or Is there a way to find out the right place to communicate as a new requester? >> Yeah, yeah. That's usually how it works. I know on the forum that I'm administer of on MTurk Grind, when you first sign up and you come, you see right there there's a section for requesters and you click on that and there's an introduction to new requesters. And it gives you instructions on how to sign up, how to get a little flair badge on your name that shows that you're a requester and how to create a post as you introduced yourself and introduced your work. >> Can you talk about the range of forums that are out there, just name some of them? So people have a sense of variety of spaces that workers have set up to help each other out, communicate with requestors, make mechanical Turk work more generally. >> Sure, yeah, there's lots of different forums and they all have their own little personality and different groups of people who are there and different sets of rules and things like that. So you kinda have to go to the different forms and see what works for you and what helps you most. The one that I'm most familiar with is MTurk Grind and I also mentioned the Reddit community. MTurk Forum is another and Turker Nation is the oldest forum. So those are probably the major ones that are the most active. >> Great, thanks. So you've been working on Mechanical Turk and done a variety of tasks over the years, what kinds of credentials do these platforms offer for tracking workers as they've been building up experience? >> Built into the forum, into the Mechanical Turk platform, there's some qualifications that are just part of the system where once you reach a 1,000 HITs completed, or 10,000 HITs completed, you get to reach those levels and so that opens up more work for you to do. There's also a Masters Qualification on Amazon, which is granted to Workers who Amazon has supposedly said are the best workers. And so if you have that Master's qualification, you're also able to access different set of hits. And then in addition, each requester who comes up can setup their own qualifications. If you're a new requester to the community, you can come and decide that you wanna setup a test and all of the workers can take the test and those who pass the test are then granted access to your work, so that's free to any requester to set up their own qualification test like that or qualification system to get workers that they trust. >> So those are the kinds of qualifications about what workers are doing on specific tasks, but could you talk about some of the other skills that workers have behind the scenes that help Mechanical Turk, actually be a viable place to do a lot of data processing and power interactive technologies? >> Sure, yeah. In addition to those climate qualifications, there's a lot of things that workers do that are unpaid and that you just do on your own time to help yourself be more productive on the floor, on the platform. And some of those things are maybe using different browser extensions to get work done more efficiently or quickly. So, it's a learning how to use those kind of extensions. Some workers use browser scripts, like Turkopticon is one of them that you can have on your browser to help you view hits that are review differently. Other workers use something called Hit Scraper, which is a different way to organize the hits that are available and you can see them differently, it's a different layout. So there's several different scripts and browser extensions that workers use to differently view the hits and to organize the hits and to accept the hits, so a of it is learning and taking the time to learn how to use those tools and setting up the system to work most efficiently for us. A lot of Turkers who work full time or work seriously have two or three different computer monitors open. Maybe one for active hits they're working on. One for a forum that they're monitoring, the one for Turkopticon reviews. The one for a chat group they're in. So setting up your system is something that is kind of unpaid work behind the scenes that helps you really improve your efficiency. >> Wow. [LAUGH] So as you were describing this two, three monitor setup, I was imagining stockbrokers of the Bloomberg terminals or computer programmers, hacking in a dark room. >> It doesn't look much different. [LAUGH] >> It doesn't look much different. Who builds the scripts and who keeps the forums going? >> It's all the community members. It's all the Turkers who want to contribute and want to help out others. Everyone kind of has different skills, so some people are great at writing scripts and helping keeping those updated. Some people are really good at helping on the forum, monitoring and moderating the forum. Everyone has different skills and different interests as far as what keeps the community going and what helps the community. So everyone pitches in differently. >> We've been talking about a lot of the skills, talents and energy happening on the worker side that is not at all obvious to requesters. Are there other things happening on the worker side that you think requesters, future employers should know about? It'll help them be better requesters, use Turk in a way that is great for workers and for the projects that employers are forwarding? >> I think the best thing to do is to communicate with the workers and then also make sure you test out your hits that you're posting. Test out the work that you post. You may have an idea what you're posting and think it's all perfectly worded and clearly understood, but then when workers get it they interpret it differently, they don't understand the directions. The pay doesn't seem to match up with the amount of work required. So I think testing out a few projects or a few hits at a time before you kind of jump in 100% is a good idea, having a couple workers look over what you're testing out might be a good idea. There's Mechanical Turk Sandbox, where you can go and test out your hits and see how they look before you actually publish them. So using those kinds of tools and those kind of ideas is a really good idea and you could also post on the forums and say, hey, I'm a new requester and I have this HIT that's available. What do you all think? Do you have any suggestions for me before I post this? And so you can get the workers side, the workers input of what workers think about looking at it and what the questions are that are gonna come up to kind of prevent a situation, where you posted a task and then there's some misunderstood instructions or something and you don't get the results that you want. >> That's fantastic. So just to make it really concrete for people, can you tell us some stories about what happened when well meaning requesters didn't do this? >> Yeah, there's been situations where there was one not long ago, where a requester want an email address is found from a list of blogs that he had and so he posted the list of all these blogs that workers went through, looking for the email addresses. And in the instructions, he said, if you find an email, paste it in and submit. There was no instructions what to do, if the blog did not have an email address on it or if the blog only had a contact form. So for those jobs, some requesters or some workers had emailed him saying, what do we do about this and received no response. So for the ones that weren't found, we typed in not found. Email not found, there's no email available and all of those hits were rejected. So workers were pretty upset about that, because we had searched very well. We'd done all the work and the email just was not available on the blog website. So workers felt like that we should be paid for that work, because it was time and effort that we put in and we followed the instructions and did what was instructed in the In the HIT but the requester thought that because there was no email found that it was a failed HIT and it was rejected. So that's some, a situation where it could have been prevented where if the requester had put into the instructions that typing in not found was acceptable, or if you don't find this don't submit anything, or some kind of clarification what to do if those email addresses were just simply not able to be found. That would have really helped, and also responding to all those initial emails asking what should we do, would have really helped. >> That brings up a really important problem I see in Mechanical Turk. So one big problem I see in Mechanical Turk is when the requester puts out work and they get work back that doesn't look like what they thought good work would look like. They can assume that it is a worker who is just slacking off, or someone who's trying to spam just to get money, or somebody doesn't have the skills to interpret their task. So what would you recommend to a requester to avoid that pitfall of misjudging the source of the error? What should they do when they see work that doesn't look like good work to them? >> I think monitoring the work as it's coming in is important when they get their first results coming in, maybe publishing a very small batch to see if, kind of test out the waters and see what the results are gonna be looking like. And then if they start getting a lot of results that are rolling in that just aren't what they want they should close the batch and make it so the workers can't accept anymore rather than letting it continue on and then having a completed batch that they then have to feel like they need to either approve or reject the HITs. But I think testing out the HIT ahead of time is really critical to prevent that kind of situation. >> Mm-hm. Okay, let's talk about pay. Amazon Mechanical Turk let's people set whatever pay rate they want. How would you advise requestors to figure out how much something ought to be paid? And what do the pay rates mean for the workers who are powering these systems behind the scenes? >> I think that's a really complex question. And people have I have a lot of different answers to that. People who are living in high expense areas have different answers than people who are living in lower expense areas. And the workers who are in India who have a different cost of living than someone who lives in Los Angeles or New York, they're gonna have a wide variety of answers for that. But, if a requester is trying to decide how to set pay, I think what would be good is for them to try it out with a couple friends or a couple colleagues. Have them do the test, see how long it takes them. If it takes them five minutes then calculate that out for minimum wage and how much would that pay if that test was set based on minimum wage. Minimum wage may be federal minimum wage, it may be minimum wage for where the requester is located. That's kind of up to the requester to decide, but I think paying at least minimum wage for United States workers is what's considered fair, and more if it's going to be including workers that are in higher cost of living areas. I think another thing that requesters need to remember is that Mechanical Turk income is taxed for workers. We have to pay federal tax on it and state tax for all of your income. You have to report it just as income. So, if I'm making $10 on a HIT, I'm not going to actually pocket $10. I will then have to pay taxes at the end of the year on that income too. So, that's also something to take into consideration. >> So, taxes are a really interesting point that I hadn't thought about before. Turk workers pay taxes as independent contractors, right? So they basically- >> Right. >> Have to pay their own payroll tax as well as income tax. So it's actually not even analogous to someone who has a full-time job if they think, oh, how much tax am I paying out of my pocket? They actually need to think, how much tax am I paying out of my pocket and how much is my company paying behind the scenes to even get it what your final take home would be? >> Right, yeah. It's quite a large percentage at the end of the year for turkers who have to pay taxes and then have to pay health care costs in addition to that. So those kinds of things do add up and do cut into the amount that you are supposedly earning when you look at your earnings number. >> Wow. Okay, so in addition to pay, are there other factors that employers should take into account when they're thinking about how much they ought to pay workers who are doing their tasks? >> Yes. I think it's important to remember that Mechanical Turk workers are independent contractors. And so we have to pay all of our own taxes, as well as health care. So if you're making $10 on a HIT, you have to then pay part of that toward taxes and toward your health care expenses. So I think it's important for requesters to remember that when they're setting their prices, they also need to know that, that's not how much the worker is going to end up pocketing after taxes and healthcare expenses are taken. >> Okay, so can you tell us a little bit about the experience of working on other platforms? Cuz you work on platforms other than Amazon Mechanical Turk. >> Right. I also work on UserTesting, which is a website that recruits workers to go and review websites or apps for businesses, to kind of test it out, and see what works and what doesn't work. Maybe it will be a company that's trying to make a new website. And you have to go check out their new website design and say what you like and what you don't like about it. So that's a site I use quite a bit. There's other sites out there like Try My UI which is a similar kind of user testing site. There is also Crowdflower, which is sort of similar to Mechanical Turk, where workers can go and review tasks and accept them to do. >> Mm-hm. >> And then what's a little bit different is sites like Upwork, formerly Elance and Odesk, where it's bidding based, and you go and you find a task you want to do, and you place bids to complete that task. Which is a different layout or different design that Mechanical Turk and Crowdflower where you get to browse the tasks and then just choose which one you want and start it right away. >> Can you talk a little bit about the similarities and differences that affect how good of an experience it is for you as worker in these systems? >> Yeah, I think for me personally I really like the set-up of Mechanical Turk. The micro-work set-up because I don't have to sell myself or try and convince anyone based on my bidding that they should hire me. I'm just able to do the task, kind of something that sounds interesting to me and start right away. It's definitely a time saver than bidding on a task and waiting around to see if you're going to be accepted or not. On Mechanical Turk you don't have a worker profile that you need to set up with a list of which you completed to kind of make a resume or something like that, like you do on sites like Upwork where people with the work then look at your profile to see if they want to hire you or not. To me it's a little more streamlined, it's a lot more faster and simpler on Mechanical Turk and sites like that where you get to choose your own work. >> Is there a particular task you've done that was really interesting or memorable. I mean, this might help people who are thinking about tasks that they can do with Mechanical Turk workers. Think about how to make that work experience more interesting for everyone involved. >> I think that there's been a couple. It kinda depends on what type of tasks you're asking about. There was an artist who wanted to collect audio recordings of workers saying om. >> [LAUGH] >> So they collected, I believe it was, hundreds of recordings, and then laid them all on top of each other. And it's a YouTube video that you can check out, where it's all of these workers saying om, at the same time. It's really interesting to hear all the different voices all put together that way. That was really memorable. >> Mm-hm. >> Other ones have been research studies that are really on important topics that you didn't hear about in the news. And you think, oh gosh I remember that study, and I remember doing that, and now it's on news. So that makes you feel really good, that you had helped in the kind of research, and you were part of that research. And then other projects that maybe are a little more fun. There was one that was up about a year ago where you got to go through and take people's food and write about what they were eating. So it was a picture of a hamburger and you type out hamburger. And there was someone's spaghetti dinner and you typed out spaghetti. And it was really interesting to see what people around the world ate. And then to try and come up with descriptive words for labeling what their food was. That was a really popular one amongst the workers who had a lot of fun with checking out what people were eating and what everyone's food looks like around the world. >> That seems like it would be really hard. [LAUGH] >> Yes, it was. It was challenging but it was also really educational to learn about what other people in different places eat. >> Mm-hm. So you you mentioned seeing academic researches based on experimental work that Turkers contributed to coming out in the news and how excited that is. Does how interesting the task is affect how much you want to be paid for it? >> I think having an interesting task is a benefit. It's kind of a perk at the end of it. But it doesn't really effect the pay because the majority of the workers that I know, the ones that are there to make money enjoy having a task that's interesting. But having an interesting task doesn't pay the bills. So the pay ultimately does matter but could be made a little bit more pleasant if the task is well designed or is an interesting topic. >> That is fantastic, thank you so much. >> Yeah- >> Is there anything that you wanted to say that we didn't ask about? >> I don't think so. >> Okay. >> That's pretty good. >> Thanks. Have a great day. >> Thank you. >> [LAUGH]
