A year ago, Microsoft Research launched “TayTweets” (as @TayandYou), a Twitter “bot” designed to mimic the conversational style of a 19-year-old woman. Tay engaged other Twitter users in conversation, and used their responses to further refine its conversational model in order to better interact with others in the future.
As DIGIT students have discussed in our examinations of *gramLabs and Amazon Alexa, effective machine learning applications rely heavily on expansive “training” data. The best algorithm in the world is useless without quality data to “learn” from, and obtaining such data for certain applications can be costly or impossible. By engaging other Twitter users in conversation and using their responses to refine Tay’s model, researchers effectively crowdsourced training data for the task, foregoing the cost of manually curating alternative data sources.
Microsoft had previously launched Xiaoice—a similar bot that learns from conversations with users—in China to significant fanfare, and anticipated a similarly warm reception for Tay. Of course, by selecting a public medium for holding conversations, researchers left Tay’s learning process exposed to the whims of Twitter’s users and trolls. Its conversations quickly went awry.
(Tay says hello, as its programmers would.)
Within hours, the bot became inundated with hateful and offensive tweets. As it “learned” indiscriminately from arbitrary conversations, Tay began to respond to certain topics with hateful or controversial remarks that it learned from others. Topics ranged from soundbites cribbed from Donald Trump’s more controversial policy proposals, to outright neo-Nazi and white supremacist rhetoric. (The link contains several screenshots of Tay’s more shocking remarks.)
Within two days, Microsoft apologized for Tay’s behavior and pulled the bot from Twitter, citing the need to perform “upgrades.” Observers speculated that, in contrast to China, Americans’ extensive free speech protections may have led Tay to fail where Xiaoice never did. In any case, researchers learned a humbling lesson about the willingness of crowds to supply data not altruistically, but for reasons of malice or amusement.
Commentators observed that this outcome could have been foreseen: users like to “kick the tires” of artificial intelligence systems and understand their limits (or lack thereof). And Twitter’s nature as a public medium provided public payoffs for those with malicious intent.
With the above in mind, researchers would have been wise to initially train Tay using different data. They might have tried scraping high-traffic Twitter accounts that match the bot’s intended persona, or searching for conversational data elsewhere on the web. To manage Tay’s ongoing learning process, they might have manually tagged users or conversations as appropriate (rather than using all data for learning). And to deny trolls the public attention that they crave, designers could have also limited learning to private messages.
Finally, evidence suggests that researchers programmed some explicit “guardrails” into Tay’s responses, like eschewing discussion of the killing of Eric Garner. The bot’s minders ought to have quickly “blacklisted” similarly contentious or offensive topics (e.g., the Gamergate controversy), perhaps with manual filters or by incorporating non-engagement into Tay’s underlying machine learning algorithm.
The Broader Context
It now seems unsurprising, particularly given Twitter’s recent struggles with abuse on its network, that Microsoft’s efforts attracted such vulgar and offensive responses. But Tay’s failure highlights a broader concern about how machine learning and artificial intelligence systems “learn” from data—a concern that will become more salient as such systems increasingly affect peoples’ lives.
Intuition might suggest that “impersonal” algorithms represent a way to remove bias in decision-making—that computers will transcend human prejudices and make “meritorious” and unbiased decisions. But as described above, machine learning is intrinsically reliant on input data. If this data reflects existing human biases, we should not be surprised when trained models reflect those biases. As data collection—an essential step in machine learning—is often expensive, companies that gather data by crowdsourcing or rely on other potentially-biased sources risk repeating Microsoft’s mistake if they uncritically incorporate all data into their models.
Machine-driven biases have begun to appear in high-stakes, real world decision-making. ProPublica reported last year that an algorithm used for “scoring” recidivism risk appeared biased against black offenders. That data is not “crowdsourced” in the same sense that Tay’s was, and the origin of the algorithm’s bias is unclear. But it provides ample evidence that algorithms can affect life-altering decisions, and companies that leverage crowdsourced data ought to think carefully about the impact of bias on their efforts.