Online Disinformation: Kate Starbird at MIT CSAIL

September 12, 2018 - Jonathan Zong

What are some of the motivations and tactics of disinformation, and how do geopolitical actors use social media and the surrounding information ecosystem to sow doubt and division? Speaking today at MIT CSAIL is Kate Starbird, Assistant Professor in Human Centered Design & Engineering at University of Washington. I was in the audience and live-blogged Kate’s talk as a way to help myself engage with her presentation and as a resource for others who were unable to attend.

Kate opens by scoping her talk to some specific terms within the variety of ways people talk about related issues related to false information online. Her focus: disinformation and information operations. Kate’s lab at UW has focused on online disinformation during crisis events, especially the ways human behavior is mediated by information technology during things like natural disasters or mass shootings. She thinks of communities coming together during these events as a form of online activism.

Kate originally set out to study the idea of the “self-correcting crowd,” which posits that crowds will naturally identify and root out misinformation through public discourse. This work ended up taking another direction. For example, in 2013 the researchers came across tweets propagating conspiracy theories about the Boston Bombing, which included links to InfoWars and other websites they hadn’t seen before. They didn’t want to give these theories too much of their attention, but nonetheless began seeing these claims appearing again and again. People spreading these conspiracy theories typically used terms like “hoax, false flag, crisis actors,” leading to a pattern of conspiracy theories, or “alternative narratives,” after crisis events.

Alternative Narratives, Alternative Media

In late 2016, because of the increasing relevance of this phenomenon, she decided to go back and systematically study conspiracy theorizing around shooting events in 2016. Her research methods integrate qualitative and quantitative approaches, and often starts with high-level data visualization to identify parts of the data to dive into more closely.

For the 2016 shooting study, she looked for tweets including keywords related to shooting events, and then scoped to conspiracy-related keywords like false flag, crisis actor. Kate observed that many of these tweets link out to self-described “alternative media” websites. On the slides, she showed a network visualization which connected websites that were cited together in reference to the same shooting event.

Performing a content analysis of these web domains, she noticed that most alternative media sites contain many different conspiracy theories. Many contained pretty much all of the ones on her list, which included things ranging from flat earth to anti-vax to 9/11 conspiracies. If you go to one of these websites looking for one thing, you might stay for all of the other things and escalate your conspiracy theory consumption.

Corrupted epistemology

The result of this disinformation ecosystem is “corrupted epistemology,” characterized by different competing ways of thinking about knowledge, and dismantling of trust in ways of determining truth. (Note: I was reminded at this point of this article by danah boyd)

Kate gives an anecdote of a conversation she had with a friend of a friend. This person was worried about negative effects of GMOs, and found Alex Jones’ website while searching for more information. By exploring within that website, they were then convinced by anti-vax, 9/11 inside job, and flat earth theories. A lot of these sites also sell unregulated nutritional supplements. Entering this information space with genuine questions is a gateway to acquiring many other beliefs and lifestyles based on misinformation.

Many of these sites also converge around ideas like “anti-globalism,” which can mean different things to people on the (flawed) left / right political spectrum. To the left, it could mean anti-globalization, whereas to the right it could suggest anti-immigration. This gives the appearance of people from different political orientations agreeing on an issue.

These websites are convincing because they co-opt the concept of—and rhetoric around—critical thinking. Many think that “if only we can teach critical thinking,” many of the problems around so-called fake news will be solved. It’s not so simple—the crisis now is not about competing facts, but competing narratives. There are competing notions of how we know something is true. Kate points out that many of these alternative media sites use the rhetoric of free and critical thinking. They position themselves as critical of the claims of “mainstream” media that are based on authority/expertise, and often call for readers to “do their own research.”

When Kate’s lab started out, they thought that disinformation meant intentional misinformation. However, she references the history of the term originating as a loan translation from a word describing Soviet influence operations (dezinformatsiya). She points to a definition of disinformation proposed by Pomerantsev and Weiss: “the purpose of disinformation is not to convince, but to confuse.” Its goal is to create “muddled thinking” and loss of trust, thereby disintegrating society and weakening response to authoritarian threats.

Information operations

Kate’s lab has turned its focus to information operations during conflict. A recent project examines the conversation around the White Helmets, a humanitarian organization that operates in areas of Syria affected by the civil war. Within the alternative media ecosystem, many websites promote the belief that the White Helmets are a political propaganda construct of the West created by the US/NATO.

Within this ecosystem there is a heterogeneous assemblage of actors with diverse motivations—independent media, state-sponsored media, sincere activist accounts. Their end goal is not actually to convince anyone the White Helmets are bad, but create enough doubt to demotivate the public from taking any action.

Kate created a network graph to show where the same articles appear in the exact same wording on different web domains. There’s a tight Associated Press cluster, where the AP shares articles across outlets. There is a loose alternative media cluster, within which Kate points out three highly influential domains that describe themselves as “independent grassroots media.”

These articles are often copy/pasted word for word across the whole ecosystem, but sometimes there is some remixing to create the appearance of more variety. There are also Russian state-sponsored media that is remixed into other articles, and then amplified by the influential domains. The state-sponsored media is integrated into the ecosystem, but not necessarily coordinated.

All of these different sites talking about the same thing gives the disinformation a sense of validity. Kate realized that the same articles were astroturfed across this ecosystem, and dressed up to target different audiences—e.g. veterans, social activists—across both left- and right-leaning communities. This microtargeting structure brings ideologically distinct, seemingly oppositional groups together around these narratives, and gives the appearance of grassroots cohesion across political ideologies.

In a project examining information operations targeting #BlackLivesMatter, Kate built a retweet graph of BLM-related conversations on Twitter. Around the same time, Twitter released a list of accounts associated with the Internet Research Agency, a Russian state-sponsored influence operation agency. In the retweet graph, both sides of the BLM debate were targeted by the IRA. A common tactic by malicious actors was enacting the worst caricatures of either side, in order to undermine their legitimacy. Perhaps one of the goals was to increase political polarization in the US. Kate also spotted behaviors designed to polarize specifically targeting the conversation on the left.

Many of the actors wrapped up in these information operations are activists who are sincere believers. Malicious actors tap into the narratives of groups and bridge/shape them toward the goals of the information operations.

Kate says that these actors are primarily opportunistic about US politics. They are active not only in pro-Trump/alt-right circles, but also on the left. She pushes back on the rhetoric that disinformation campaigns are all targeted toward the right, or only effective against the right.

Implications for tech design

Kate reiterates that these problems not completely technical. Before technical problems can be properly addressed, we need to better understand some complicated issues. For example, what differentiates an actor who sincerely disagrees on an issue from someone who is trying to maliciously manipulate discourse?

Conducting research on disinformation

Deep qualitative engagement was incredibly disorienting for research team. Kate’s team often felt deep confusion while exploring the alternative media ecosystem, and had a difficult time making sense of conflicting claims. After all, some of these disinformation sites are compelling and specifically designed with rhetoric that resonates with people like them. It’s effective—we need to step back and see larger patterns of information sharing to understand what’s happening.


I was unable to stay for the whole Q&A, but here I summarize some questions and answers from the audience.

Daniela: Is there anything predictable about content ecosystems that would be useful for an ML/algorithms approach to detecting disinformation?
Kate: We don’t want to problematize all alternative/grassroots media—how do we distinguish between malicious and non-malicious actors? We can observe content sharing patterns over time, seeing for example the same sets of sites doing similar things across different conversations. Sites use complex remixing, like translating back and forth on Google Translate, to appear bigger and more diverse than they are. This could plausibly be financially motivated as well as politically—how do we distinguish between the two motivations?

Daniela: What can computer science do to help? Are there advances in computational methods that can contribute to interdisciplinary approaches?
Kate: One important need is for ways of representing what’s happening to help inform people about the information they are seeing. Only by seeing at global scale did the team understand the issue—what other representations can we deliver to people to empower better decisions? (Note: This answer made me hopeful that my PhD will contribute to this conversation.)

Arvind: The co-opting of critical thinking is not a problem we can solve with technology. How should we be changing our teaching methods? We can’t simply make appeals to authority—where do we go?
Kate: Latour’s article “From Matters of Fact to Matters of Concern” suggests critical thinking is taken too far when it is used to deconstruct everything. Critical thinking isn’t bad, but it can be weaponized. On mass scale, people are taking tools of critical thinking to throw everything on the floor—we need to use them to make things better by building common understanding instead.
(Note: I was also reminded at this point of this essay by Régis Debray, specifically the chart at the end.)

Charles: Recommendation systems on the internet suggest content based on what you want to see and hear—this sounds dangerous.
Kate: For example, consider YouTube recommender algorithms. We have the problem of purposeful manipulation, but often recommendations support the gateway effect without anyone actually trying to make it worse. (Note: I was reminded of this essay by Zeynep Tufekci)

Steve: Is it reasonable to conclude that the primary problem is an attack from an adversarial nation?
Kate: This is only one aspect of the problem—anyone can access the tools to perform this manipulation.

Audience member: There are passive vs active approaches to content moderation. E.g. from labeling with warnings that something is not fact checked, to entirely deleting content. What’s the right approach?
Kate: Different people respond differently—some people take the labels as validation that the world is lying to them, which makes them more interested in clicking. Certain tactics only work for some people, and maybe not those who are most susceptible.

Audience member: Can we formalize the definition of disinformation into something we can use in statistical modeling?
Kate: It’s hard to do this when there are so many competing terms for similar things, like fake news, disinformation, information operations, etc.
(Note: This question reminded me of questions around formalizing definitions of fairness for use in machine learning. There are many competing definitions and they are often mutually incompatible. Arvind Narayanan writes, “the proliferation of definitions is to be celebrated, not shunned, and the search for one true definition is not a fruitful direction, as technical considerations cannot adjudicate moral debates.”)