You may have heard that 50+ million Facebook user accounts were harvested by Cambridge Analytica (CA). Or maybe you heard it was Global Science Research (GSR), Dr. Aleksandr Kogan, or that Nix guy.
You may also have heard it was an inside job facilitated by Facebook staff, a data breach from lax security, Peter Thiel’s data-mining company Palantir, or the daughter of Eric Schmidt, Google’s former CEO and Burning Man buddy.
What few people were discussing was Amazon’s role in facilitating the data access. Specifically, their microtask site called Mechanical Turk.
Few people know about the online community that chronicles what happened, which anyone can access. It tells the story from the perspective of Amazon’s microtask workers, who granted access to their Facebook accounts.
Within this community, you’ll hear from the people who participated and granted access to GSR. Some enjoyed the process. Others expressed disgust. But what’s bizarre is that over the course of a few years, 16% of participants consistently warned others that this project was exposing people’s identity, which violates Amazon’s privacy practices.
In this article, I’ll explain the platforms used, share my perspective on the issues, and talk about psychometric profiling, for evil or good.
Overview of the data mining process
Here’s my understanding of how Dr. Kogan collected user profiles from Facebook:
- Step 1. Target Mechanical Turk micro-task workers (Turkers) with Facebook accounts
- Step 2. Ask the Turkers to complete a psychometric survey
- Step 3. Ask the Turkers to grant access to their personal Facebook account
- Step 4. Have Turkers reveal their identity
- Step 5. Download data from the Turkers’ Facebook profile, and their friends’ data too
- Step 6. Withhold paying Turkers until they grant access to their Facebook profile
- Step 7. Pay compliant Turkers, or invite non-compliant Turkers to complete the task
- Step 8. Repeat about 200,000 times
Next, I’ll discuss Mechanical Turk and describe how researchers use it.
What is Mechanical Turk?
Amazon’s Mechanical Turk (MTurk) is a crowdsourcing platform where people pay virtual workers to perform micro-tasks.
The name sounds ridiculous, but it’s quite witty. The original Mechanical Turk was a chess-playing machine built in the 18th century that dazzled people. It was a total hoax, with a person hidden inside the machine controlling each chess move. In essence, the original Mechanical Turk was a bogus machine powered by human intelligence.
Amazon’s version works in a similar way, where workers perform micro-tasks that are difficult for computers to perform. These are typically tasks that require human judgment, such as detecting sarcasm in a tweet or classifying photos.
The people who request work are “Requesters,” while those who do the work are “Turkers.” Each microtask is called a Human Intelligence Task, otherwise known as a HIT.
MTurk offers an API for on-demand HITs. With Amazon’s massive pool of Turkers, Requesters are guaranteed a steady supply of workers eager to perform HITs at the right price.
How academics use MTurk
Several years ago, scholars started using Mechanical Turk for research, including psychological questionnaires and experimental designs. They found it was a great place to recruit participants.
You might be thinking: isn’t a population of micro-taskers sort of biased? That’s what I thought when my friend Don Steiny first told me about it. But Mechanical Turk may be the world’s largest supplier of participants for scientific studies.
Numerous studies evaluated whether MTurk was a credible source for participant recruitment, examining Turkers’ demographics, psychographics, and comparison to US census data. The verdict was that Turkers offer a reasonably representative sample of the US population at a fair price. It gave researchers access to panels previously only available through massive polling companies, and in the early days, it was dirt cheap.
There are some major pitfalls, though. MTurk is not a survey panel group; it’s a microtask portal full of workers who aim to make as much money as fast as they can. From my experience, about 20% of respondents only skim your survey questions, and some completely fake it. To use it well, you need to get good at setting traps to catch cheaters. If you don’t, your data will be junk.
Using MTurk for research is a bit like shopping in the most dangerous part of town. You’ll find the best deals if you’re street-smart and can get out.
Perhaps the most relevant point here is that MTurk lets you recruit people with a Facebook account. My guess is that GSR used this feature to target Turkers with a Facebook account.
Turkopticon: the paper trail the media missed
Amazon manages MTurk like absentee landlords, and the interface feels like a property that hasn’t been updated in 20 years. They keep things working and intervene to fix problems, but don’t appear to value resolving Turker disputes. Since there’s no real oversight, Turkers can be subject to all sorts of abuse with few options for escalating or resolving them.
This situation got bad enough that Turkers set up Turkopticon, a plugin and community where Turkers could comment on Requesters and HITs. It’s a place where people can share what they’ve experienced, good or bad, with others.
Turkopticon is independent of Amazon, so Amazon staff don’t necessarily know what goes on there. Though I’d be shocked if they didn’t monitor it.
The blame game and risk management
It’s easy to use the Turkopticon data to point the finger at Dr. Kogan. Here’s my perspective on where the finger actually needs to be pointed.
Dr. Aleksandr Kogan set up a company, GSR, which carried out the data mining work and sold that data to CA. In the fallout, many of the actors turned on each other. CA blamed Dr. Kogan and GSR for violating Facebook policies. Dr. Kogan argued that CA was intimately involved in the process and was just scapegoating him.
I worked with some amazing people during my seven years in the United Nations. But one unethical manager once advised me to always bring in contractors for risky projects. If things work out, you can steal their credit. If they fail, you can blame them. Either way, your job is safe.
I totally disagree with the ethics here, but know that many corporate arrangements have exactly this type of risk mitigation in mind.
The Turkopticon data only refers to Dr. Kogan and GSR, but I have a hard time believing that CA and its parent company had no idea what was going on. How could staff at corporations specializing in data services have no clue about common data collection norms and terms on Amazon and Facebook?
My point is this: don’t use this evidence to single out Dr. Kogan alone. The Turkopticon evidence places the smoking gun in his hand, but I have a hard time believing he pulled the trigger on his own. CA was happy to contract GSR for data extraction services as a separate legal entity, which mitigated their corporate risk. What made the deal sweeter was that they were in a position to scapegoat GSR for any breach and keep the data too. One side said they didn’t know the data was unethically obtained. The other said they didn’t know it would be used for unethical purposes.
Apart from Amazon’s breach, there was nothing illegal about the data collection process. It was legal and within the terms of use. Facebook had no problem with GSR or CA working independently. But when GSR handed the Facebook data to CA, that’s when Facebook intervened.
Mechanical Turk’s requirement to protect worker identity
MTurk has several rules, but a key one is the requirement that you never ask a Turker to reveal their personal identity. When I first heard that MTurk was used to collect the Facebook data, it didn’t make sense. The minute a Turker logs into Facebook, you’ve linked their anonymous Turker ID to a real person.
Even if the study had been carried out by Cambridge University, this would have still been a problem. You’re not supposed to reveal Turker identities under any circumstances. As far as I can tell, the only actual violation was against Amazon’s rules.
Analysis of Turkopticon
I found two accounts in Turkopticon linked to the Facebook collection process. One is from Dr. Kogan’s personal account with 104 Turker responses. The other is for GSR with 49 Turker responses. Together, there are 153 comments running from 5 May 2013 to 15 December 2015.
To access these accounts, register at turkopticon.ucsd.edu, then search for “Aleksandr Kogan” and “Global Science Research.”
For this analysis, I merged the data from both accounts. Turkers ranked items on a 5-point scale, which I averaged and converted to percentages. Keep in mind that this is a synthesis of two accounts, so some responses may be for work Dr. Kogan was performing independently. People also tend to comment on these sites when they have bad experiences, so the picture may skew negative, and disputes settled by email won’t appear here.
Overall impressions
Overall, Turkers held an okay but not great impression of Dr. Kogan’s and GSR’s HITs. Both profiles are full of complaints about not being paid, a lengthy process, and being asked to redo surveys. Some people reported good experiences. It sounds like GSR processed work manually at key times, so some people were paid immediately while others had to wait.
GSR received a 48% approval rating for responding and a 60% rating for approving work, which was the biggest complaint. Fairness in approving or rejecting work came in at 85%. The workers rated pay fairness at 71%, which is okay for the platform.
Here’s what some Turkers had to say:
Survey on wellbeing for $2. Can’t recall how long it took but I think it was under 30 minutes. Approved next day.
Jun 12 2014 | hs | Account: Global Science Research
I remembered this guy and almost didn’t click on this latest HIT, because he was so disorganized in a prior communication. Now he wants to download my personal FB info? Get lost.
Feb 25 2014 | Giraffe | Account: Aleksandr Kogan
HIT took 4 mins and paid .50, approved/paid 8 days later.
Dec 11 2013 | Thomas | Account: Aleksandr Kogan
Paid within 48 hours, pay could definitely be better but overall it was a great HIT.
Jun 25 2013 | sf1…@t… | Account: Aleksandr Kogan
This guy says that he has no records that I completed his survey. I did everything as it said in the HIT, even pasted the confirmation code. Then he says he couldn’t find it. I see he does the same to everyone that works on his HITs. He is a professor at a University? God only knows what he is teaching his students. For me he’s just another SCAMMER that rejects our work so he doesn’t have to pay us.
Jun 22 2013 | msor…@b… | Account: Aleksandr Kogan
Quality of scientific instruments
One Turker felt that Dr. Kogan’s tools weren’t up to scratch and took potshots at his questionnaire. MTurk has a large population of people who do many surveys, and after a few years some Turkers develop a keen eye for high-quality questionnaires.
Below is a fair point about an error in the ordinal ranking of scale categories. This would cause some bias, but probably manageable if it wasn’t too widespread.
The survey was carelessly done, which makes me believe it’s all a setup to collect information about a person for really cheap. For example, on a scale from “neutral” to “strongly agree,” the order should be neutral, slightly agree, agree, strongly agree. The survey listed it as neutral, agree, slightly agree, strongly agree. The survey sometimes did this for the “disagree” scale too.
May 12 2014 | AIW | Account: Aleksandr Kogan
Transparent representation
Both accounts used real names, so it appears that GSR and Dr. Kogan were acting transparently. The Turkers were able to track down the study. There were a few references to Cambridge University early on, then nothing, and later to the University of Toronto.
Here are some examples:
I barely got through the survey in time. I seem to be fortunate in that I got paid. Aleksandr Kogan is a professor in the Psychology Dept. at the University of Cambridge.
Jun 21 2013 | Humpty Dumpty | Account: Aleksandr Kogan
University of Cambridge 30-35 minute psychology study investigating personality and the evaluation of media clips. $1.20. HIT pending after 15 days. VIOLATES MTURK TERMS OF SERVICE.
Jun 21 2013 | capecoralhobo | Account: Aleksandr Kogan
“University of Toronto study.” 27 minutes for $1.88, approved the same day and paid by the next day.
Nov 03 2015 | kimadagem | Account: Global Science Research
The breach of trust
Dr. Kogan made this promise to safeguard Turkers’ identity: “Please note that we take several precautions to ensure all of your data stays anonymous and safe, and it will only be used for research purposes.”
This is the standard commitment used in academic research, and the promise on which the breach of trust hinges. Turkopticon has one reference to it, dated 3 Mar 2014. After that date, I didn’t see any other references. I assume they removed it and put it on the app, which I haven’t yet seen.
It makes no sense to use this ethics statement in MTurk, because MTurk’s policy already forbids revealing Turker identity. There’s nothing left to protect.
Here’s the reference:
Violates MTurk Terms of Service. This requester explicitly states: “provide our app access to your Facebook so we can download some of your data — some demographic data, your likes, your friends list, whether your friends know one another, and some of your private messages. Please note that we take several precautions to ensure all of your data stays anonymous and safe, and it will only be used for research purposes. You will receive $.50 for participating.”
Mar 03 2014 | joysinger | VIOLATES MTURK TERMS OF SERVICE | Account: Aleksandr Kogan
Chaotic process
There are numerous complaints of messy management, with lots of lost payments, rejected work, and requests to redo surveys. Overall, the speed of communication was rated 48%. In some cases Turkers received fair treatment, but it sounds like GSR was overwhelmed.
“Thank you for being interested in our research! However, when we compared the MTurk ID entered at the end of the survey to the records on MTurk itself, we did not find your record of participation. There might be some technical problems, or you may have entered your MTurk ID with typos. If you still wish to get your Amazon Mechanical Turk credit, please redo the survey as soon as possible.”
Asking me to do the survey twice, so I assume he’s trying to pad his results. I copied and pasted my MTurk ID, so I’m pretty sure it was fine. Others have gotten this message as well.
Feb 11 2014 | soulo…@y… | Account: Aleksandr Kogan
Using a stress-inducing dark pattern
There’s evidence that the team employed dark patterns early on to pressure Turkers into giving up access to Facebook.
Dark patterns are interactive design patterns that use psychology in a manipulative way. Early iterations of the survey combined bait-and-switch with loss aversion. The Turker would start a survey for money, complete most of it, and then midway through be told they’d have to grant access to their Facebook profile. At that point, they had to either lose their time and money, or give up their privacy.
For those who cared about their Facebook privacy, this placed them in a loss-averse dilemma with no good option. This may explain much of the moral outrage in the comments.
Evidence suggests the GSR team later replaced this approach with an incentive system, offering bonuses to anyone who opted to provide Facebook access.
Here’s an example of the earlier bait-and-switch:
Survey on personality, wellbeing, demographics, and Facebook. $2. Without warning in the description, the HIT asked me to violate MTurk TOS by giving them access to my Facebook account. Since I had to return the HIT, I was not paid for my time.
Jun 09 2014 | DisplayName | Account: Aleksandr Kogan
Violating Amazon’s MTurk anonymity principle
Turkopticon has a button that lets Turkers add a badge to their comment reading “VIOLATES MTURK TERMS OF SERVICE.” Overall, 16% of all Turkers clicked it, flagging violations of Amazon’s privacy practices. In total, there were 25 complaints over 2.5 years, running from 21 June 2013 to 15 December 2015.
In MTurk, it’s a serious violation to do anything that forces Turkers to reveal their identity. Numerous Turkers pointed this out, flagged it, complained, argued, and sent warnings to others.
This is clear evidence that Amazon’s privacy requirements were violated for several years. Whether Amazon or GSR knew about it, both can always play the plausible deniability card. I’d estimate around 200,000 Turkers were involved in the Facebook data collection, and about 16% complained in Turkopticon. Given that Turkers normally send emails to discuss problems, I have a hard time believing nobody at Amazon or GSR was aware of the violation.
Here is a discussion thread that broke out:
VIOLATES MTURK POLICIES. Asked to access my Facebook and download their software. I HOPE AMT bans all Turkers that have reviewed this HIT by admitting they too have violated the policies by participating.
May 10 2014 | ASU ALL DAY!
Banning all people who decided to take the survey. That’s a bit extreme don’t you think?
May 10 2014 | alpine
If you WILLINGLY and KNOWINGLY participate in a crime in this country, will they arrest you? A POLICY is a POLICY, and if Turkers participate in violations, they RUIN the legitimate experiences for those who DO FOLLOW THE RULES.
May 13 2014 | ASU ALL DAY! | Account: Aleksandr Kogan
Nothing on Facebook is private anyway. If you want privacy, don’t create a Facebook account. Good HIT. Approved next day.
Jun 09 2014 | AK
Privacy and moral disgust
In 2015, the Guardian published an investigative article about GSR using MTurk to drive Facebook data mining for political advertising. Amazon leadership said they had no idea what was happening and stopped GSR once they learned about it. But what I do know is that Amazon set up a system where Turkers can be abused with little power to stop it. Because of this, the Facebook situation happened.
Since Dr. Kogan’s data mining was perfectly legitimate on Facebook until he handed the data to a third party, Amazon was the only company in blatant violation of policies from day one.
Don’t ask, don’t tell
Despite all the finger-pointing about privacy practices, my experience in data-driven technology is that it operates on a “don’t ask, don’t tell” basis. People and companies do it all the time but don’t like to talk about it.
Profiling is no longer sneaky spycraft. It’s widespread and institutionalized. Full Contact gives you 500 free profiles every month via their API. Or if you have over $100,000 per year, established companies like Oracle and Adobe will sell you people’s cookie IDs so you can send them programmatic ads.
Media are spying on you to survive
What I find ironic are some of the most moralistic articles on protecting privacy, published on news sites loaded with identity trackers. Many are specifically tied to programmatic advertising platforms that offer high degrees of targeting. Just by visiting these news sites, there’s a good chance you’ll be added to a programmatic advertising list of cookie IDs, and if you authenticate, you may be added to a more detailed list that’s more valuable to sell. Get the Ghostery plugin if you want to see for yourself.
I don’t think this is inherently a bad thing. It helps pay their bills. I’d rather help maintain what’s left of our independent media than demand free content with no revenue model behind it.
Facebook harvesting your contacts for growth
I don’t believe companies like Facebook would have existed without data mining massive volumes of contact lists in their early years. A leaked internal memo focused on ruthless growth, even with questionable techniques.
I consider Facebook’s Messenger app one of the darker design patterns out there. I suspect they selected the name “Messenger” to create confusion with the native messaging app. I see it as a trojan horse that slowly moves toward its goal: your contact list.
Google scraping, to take over the planet
The world’s biggest data miner is Google. Their PageRank algorithm works because it extracts judgment from other people’s data. Google’s crawler doesn’t walk around your site. It extracts data from your site and stores it in a database.
Normally, Google can scrape you, but you cannot scrape Google. I met the head of an advertising intelligence company and asked her how her company even existed, given that they were using data scraped from Google. She told me that Google tolerates them because they add value to Google’s products, even though they technically violate its terms.
Free-flowing data
Tim Berners-Lee, who invented the World Wide Web, considered HTML a second-rate language. His original vision was to make the markup language more like XML, so that all web data could be exchanged freely in a semantic web.
That never fully worked out, but RSS, APIs, mashup services, and powerful scraping tools have created numerous innovations that can only exist in an environment of open data. Many big tech companies owe their existence to this ecosystem. But once in power, they try to close those data doors to others.
Data acquisition operates a bit like the French civil law system: if no rule explicitly forbids it, you’re probably safe legally. But if people feel moral disgust, you’ve crossed an ethical line. Ideally, you want to pass both tests.
I suspect the data access issue would have been ignored if GSR hadn’t left the Facebook ecosystem and used the data for politics. For those who feel visceral disgust at Donald Trump’s election, it’s difficult to accept that their data helped get him elected. For those who support Trump, they may feel equally troubled by the privacy breach, but won’t feel that their data was used against their own interests.
Perhaps this is why there’s less talk about broken laws and more about broken trust.