Abuse and Disinformation at Twitter: An Inside Story

Citizens are in the middle of a global disinformation campaign. The attackers know all the legal loopholes, all the social engineering tricks, and all the blind spots they can exploit. Meanwhile, average citizens don’t understand how to protect themselves and just want software to be as simple as possible. It’s a recipe for disaster, which is what we got.

Jon Bell worked on Twitter’s Abuse team as the lead designer in 2016, and is haunted by his team’s inability to take the threat seriously and make significant progress towards addressing it. This talk is a behind-the-scenes look into all the complexity, the surprises, and the lessons learned on the front lines of one of the most challenging social and UX issues of our time.

Kicking off with a story of the weird behaviour of terrorists getting blocked from Twitter – they’d keep coming back and getting banned over and over. Why would you keep doing the same thing? He asked about someone who studies this, and they said he needed to consider the motivation: getting blocked more often was like an achievement, a high score. Once he understood that motivation, he was able to begin designing aginst it.

Jon notes that this talk must include things that are quite uncomfortable, but these are things he’s seen… Jon joined the Twitter ‘abuse team’ in 2016. The team cared deeply about their work, everyone was so motivated they experienced the ‘pickle jar effect’, people became fairly poor at listening to other team members.

The team had to implement some process around capturing the huge list of possible features and projects, so people stopped re-pitching the same stuff over and over. They started doing epic levels of documentation on not just what they were doing, but why they did it and how they did it.

They created a video called “Amy’s first minute” – when someone experiences abuse, you have one minute to show that there is a pathway to safety. If they don’t see that in a minute they’ll leave and they really should leave, you don’t deserve to have them stick around.

User controls:

  • Mute keywords + conversations
  • Notification filters
  • Reporting

So the team was going really well for a while… then… a big re-org happened. Ultimately Jon could not accept what happened through that re-org and the things that were de-prioritised as part of it.

(Meta talk about how easily Jon could paint himself as the hero, but it simply wouldn’t be real or useful.)

Three thought exercises

  1. imagine a world when everyone was paid the same – people would only do a job if they were passionate about it
  2. what if everyone was really good at their job? there is a downside, as it sets people up to let the emotional immune system kick in – so we can write people off
  3. what if everyone was empowered? the problem is turns into an accidental arms race, because everyone has so much power they step on each others toes with their empowered actions

So a team that seems like it should be ideal can have some real issues in practice.

…that was Jon’s epiphany. He was surrounded by passionate, hard-working people who had been told to go make an impact. It was like a bunch of people in a boat, furiously rowing in circles. Teams with a bunch of rockstars trying to be leaders can end up being bad or ineffective teams. Jon acknowledges that he’s the one who left the team, the others were able to deal with the big re-org.

Book recommendations:

  • Animal Farm – it’s ultimately about humility
  • How to Win Friends and Influence People – we need more diplomats
  • The Righteous Mind – how good people can disagree on big things

We need an empathy test, to sanity check whether people who talk about empathy a lot have developed a blind spot.

Here are three tests:

  • Nickelback have sold 50m albums
  • Comic Sans
  • “sportsball”

…we should be able to find empathy for people who like these things, but most people are quick to be dismissive about at least one if not all of these. Beware of letting your personal taste override your empathy.

Five features Jon feels Twitter should consider

  1. Remove replies – this feature got cut in the re-org and at the time Jon couldn’t deal with that. The problems with the feature have been dealt with.
  2. Twitter ombudsman who writes an independent report on big decisions from Twitter. This would also provide precedent and history. It could apply to all tech companies.
  3. Fact checking – when enabled, posts can be flagged with fact check resources
  4. Categorise every tweet on a meta level – which would enable deeper, more powerful filtering and potentially even create some connections between communication silos (“echo chambers”). You could set Twitter to only show you tweets ranked as “thoughtful”; and clickbait might stop making money.
  5. Best news product – provide updates on the tweets you read, including corrections and fact checks. Use all the categorised content to create hubs around stories, showing pro and con analysis; and all the updates.

Last thought – some of this might work, some might not. A lot of design discussion is very black and white – love/hate, good/bad. We need to allow room for people to make thoughtful proposals. Hot Takes™ are seedlings. If they are grown from a grumble to a thoughtful critique they will gain greater value.