(rhythmic synth music) - Hello.
Good morning, thank you all for coming out. Yeah, as John mentioned, this is something that I have been talking about for a while in lots of different ways and I feel like it's a topic that keeps on growing, both in the discussions that I'm hearing in our industry and sort of in how I think about it as well. When I first started working with Eric Meyer we were talking a lot about inclusivity around things like handling situations that are maybe not the most typical use cases but are in fact maybe the most important ones to be designing for, and since then it's sort of like grown, and grown, and grown into this sort of massive thing around ethics, inclusion and harm and, one of the most wonderful things that I realised is I was looking out at the agenda over the next couple of days, is that a lot of people are going to be touching on some similar themes. And I think that that really represents what is kind of a sea-change that is starting to happen in our industry that I'm very excited about. So I'm so excited to be able to open up this conversation and kind of get this going at this particular event because I think you're gonna hear other threads come out of this over the next couple of days. So to get us started I'd like to start with something that's been a piece of design in the news recently in the United States.
It is not a piece of design that is really so much related to technology.
It's actually related to the census.
So the US census is going to happen in 2020, we do it every 10 years.
And for the first time since 1950, the US census is planning to ask respondents about their citizenship status.
And some people are really concerned.
Because, as you might know, immigration in the United States is a little bit of a touchy topic right now. It's always a touchy topic, it is a particularly difficult topic right now. And so there are concerns about asking about immigration status.
So there is an organisation, as part of the census, so one of the departments there, the Centre for Survey Measurement.
What they do is they pre-test.
They go out and look and see if the surveys that the census is going to do are going to work. They went out and met with a bunch of different people, lot of qualitative research.
Had them walk through this process and get feedback. Here's a little bit of what they heard.
One Spanish-speaking respondent said she as uncomfortable registering other household members. When she realised she'd have to provide information on the other people who lived with her, she mentioned she was afraid because of the current political climate and news reports about changes to immigration policy. And they heard other comments like this, like another Spanish-speaking respondent filled out information for herself and three of her family members but then intentionally left out other roomers who lived in the house, which is something you're supposed to provide on the census is all the information about everybody who lives in the residence.
She said it "frightens me, given how the situation is now" and mentioned being worried because of their immigration status.
And there were a whole lot of other examples like this. They would go in, they would talk to people and they would say things like I'm concerned, I've heard about there being this Muslim ban. I've heard about there being a wall.
I've heard about all of these ICE raids, the Immigration and Customs Enforcements going in and finding people and sending them back across the border. And so they don't want to fill this out.
So there was enough concern in the qualitative research that the Centre for Survey Measurement came back with a report saying that they are very concerned about this question. And the reason this is such a concern, right, if people don't fill out this information, is that what you say on the census makes a huge difference in what happens in the United States.
So, it can affect how many members of Congress your state gets, so how many people are in the House of Representatives, that's population dependent.
If you are being under-counted, if there's large population groups that are being under-counted, especially in states where this could make a big difference. So a state like California, for example.
If you actually lose representatives because so many people are not counting that could make a difference in the balance of power. It also changes things like voting district boundaries, where funding goes for infrastructure.
So should we or should we not invest in building a road here? That is gonna be dependent on populations in that area. Where do we build things like schools and libraries? And so what you find is that the seemingly little questions, right, they can add up to really big shifts in who gets resources.
And if there's certain communities that are systematically under-counted or don't count themselves, if there's certain communities that are not showing up then it means those communities are going to be the ones who are going to bear the brunt of that.
They're gonna be the least reflected in our government and they're going to have the least resources allocated to them.
So the people who say that this is a not a big deal, the people who say that they support the idea of asking about immigration status, they'll say, "Well the census data is private. It's confidential." So when you fill out your census information you are not actually supposed to ever be tracked to your specific answers.
It's supposed to be taken in aggregate and your information should be kept confidential. That's true, except if the United States decides that it's under threat.
You see, because back in 1941, just after Pearl Harbour, we passed something called the War Powers Act and what the War Powers Act did is it increased federal power during World War Two and tucked into that particular Act was a clause that allowed the government to take data that had been collected confidentially and to de-anonymize it and use that data.
And that is how data from the US census that was conducted in the 1940 was used to help identify Japanese-Americans, round them up and inter them during the war. I tell this story, because I think it's very easy, when we're going about our day, making all of our little design decisions, to not realise how many consequences each of them can have. Even if we're not designing something that is as impactful to our society as like a government form for a census count, every single time we design something there are consequences.
At the census at least there's an office that is intended to identify what some of those consequences might be.
In our work that's rarely the case.
And I think we often don't spend enough time looking at the implications of our work.
Particularly when we talk about design decisions that might seem really small on their face. This is a story that comes out of Google Maps, I call it the mini cupcake meltdown and I'm gonna tell you what happened here.
So back in October, Google Maps launched an update that went out to a selection of iPhone users and what the update did is it, in addition to providing the normal information you get if you map something, so distance from you, how long it might take you to walk there, transit instructions.
They also started telling you how many calories you might burn if you walked that distance instead of taking another form of transportation. And alongside the number of calories they also told you how many mini cupcakes they thought that would be. Now, this led to many questions.
One of the first questions is, what do you mean by mini cupcake, which I think is, like I don't know what that means. I literally, I don't know what a mini cupcake means. America loves cupcakes and mini items but I still don't know what that means because also American cupcakes are usually very large so I don't know, right.
What's a mini cupcake? Who knows.
Whose mini cupcake, who knows.
But they launched this feature and pretty quickly, they started getting some feedback. I'm gonna share my favourite string of feedback with you. I think it sort of underlines the problem pretty clearly. It is from a journalist named Taylor Lawrence and she started tweeting about, around eight o'clock P.M. the night that she got this update on her phone and she's like whoa, wait a second, this is weird. First reaction right.
Second reaction is oh my God and then WTF.
So she keeps on going, going through this and walking through what is going wrong here. Started around 8:00 P.M. and continuing onward
in her evening and she says like, there's no way to turn the feature off.
Do they realise this could be triggering for people who've had eating disorders? It's kind of shamey feeling.
Then she goes on about the health questions around this, well, what do we mean by number of calories burned? Who would be burning those calories? That's super-dependent on your particular biological makeup. That is super-dependent also on what we mean by mini cupcake.
Also there's lots and lots of different conflicting science about whether calorie counting is helpful and whether calories are the problem and she goes on and on about this.
She kind of walks through all these different things until about 9:00 P.M.
So within the course of an hour she has walked through a whole series of issues that she sees with this product. This is sort of like my synthesis of all these issues, the first one being, not only are you not opting in, nobody asks you if you want this feature, there's no way to turn it off.
You get the mini cupcakes.
You wanna use a map? Too bad, you get mini cupcakes now.
She says it's dangerous, people with eating disorders, shamey feeling, average calorie counts can be inaccurate. Not all calories are equal.
She talks about how cupcakes might not be a useful metric. Like, mm hmm...
And then, and this is my interpretation of some of the things that she says, but pink cupcakes are not like a culturally neutral food. It's a particularly western, particularly feminine and particularly white and middle class food choice to use as your barometer for calorie counting.
And she talks about how it perpetuates diet culture and sort of the toxic-ness of constantly obsessing over diet.
Now, you could agree or disagree with any one of these items.
You could enjoy the calorie counting and the mini cupcake.
Like, at a personal level, you might find that useful and that's fine.
But here's the thing, these are all potential ways that this product could fail. These are people for whom this product could be bad. These are people for whom this product could be actually incredibly harmful.
And it took her about an hour to tweet out a whole bunch of reasons that this could go really wrong.
Within about three hours, they'd actually turned that feature off.
Now, you all work. Design teams or similar.
How long do you think they spent building that product? (audience laughs) How long? So, I know from my personal experience, being in some meetings, that you spend more than three hours deciding like, what the frosting on the cupcake illustration should look like. Right? Like, how many illustrators designed how many cupcakes before they settled on that cupcake? What was the discussion that went into deciding whether the cupcake frosting should be pink or not? Or if it should have sprinkles or not? I love this story because I feel like it's such a perfect encapsulation of all of these little tiny decisions that happen in tech all the time.
The sort of mundane little decisions that really are super hyper focused on thinking that we can delight people and not really thinking on the consequences. Thinking that what we're going to do is keep people clicking and tapping and engaging, and that we know what's best for them.
You said you wanted instructions on how to get from point A to point B, but I know what you actually want to know, is how bad you should feel about what you're eating today. It's a sort of like paternalistic outlook on the world and I think too many of us have been guilty of it for too long.
I think what we've actually done as an industry is really normalised a sort of a chronic underinvestment in inclusion and in preventing harm.
We've chronically under-invested in this and we can see it all over the place.
For example, just last month, a report came out from Amnesty International. Amnesty International has said that "Twitter is failing in its responsibility "to respect women's rights online "by inadequately investigating and responding "to reports of violence and abuse in a transparent manner." If Amnesty International says that you are doing a bad job with women's rights, I don't think you should feel real good about that. And the thing is, Twitter knows.
This is not a surprise to Twitter because, in fact, Twitter's talked about this. In fact, they've talked about it a lot.
They've talked about it for years.
Just that same month, Jack Dorsey, CEO, he said, "We love instant, public global messaging.
"But we didn't fully predict or understand "the real-world negative consequences.
"We acknowledge that now.
"We aren't proud of how people have taken advantage "of our service, or inability to address it fast enough." "We didn't fully predict or understand "real-world negative consequences." Well, okay.
You might think, "Great, they're on the same page. "They're going to do something about this." But here's the thing, back in October, what did Jack Dorsey say? Well, "We see voices being silenced on Twitter every day. "We've been working to counteract this "for the past two years.
"We prioritised this in 2016.
"We updated our policies and increased "the size of our teams.
"It wasn't enough." Oh man. So they've really been trying for two years, and not there yet? Well, actually, if you go back to 2015, at the time Dick Costolo was the CEO, he said, "We suck at dealing with abuse and trolls on the platform "and we've sucked at it for years...
"We lose core user after core user by not addressing "simple trolling issues that they face every day. "I'm ashamed of how poorly we've dealt "with this issue during my tenure.
"It's absurd. There's no excuse for it." If you really wanted to change something, you'd change it. But if you chronically under-invest in something, and then you start throwing some more resources at it, it doesn't mean you've stopped under-investing. And we can see it in all kinds of places.
This, is a lovely screenshot, from a YouTube kid's video.
What it is, is a knock-off Peppa Pig cartoon. And, last fall, James Bridle, he published this look at the underbelly of creepy and violent content that is being targeted to kids on YouTube.
There are tonnes and tonnes of videos, like this one, where instead of a normal Peppa Pig video, where a visit to the dentist would have this happy result, instead, the visit to the dentist morphs into this graphic torture scene.
What James started walking through was so many different examples of these videos that are being shown to kids.
Cartoons about things like torture and suicide, all these sexualized items, all of this stuff that's just sort of creepy and weird and hard to put your finger on why it's so creepy but it's super creepy.
Violence, weapons, live action films where kids are tied up or hurt, where they're vomiting and writhing in pain. And what's happening, is that these videos are being produced and added to YouTube by the thousands.
They're tagged with what he's called a keyword salad, so it's like, they'll take all of these popular tags that popular, traditional children's content has been tagged with, that they know kids like. So, stuff about colours and shapes and counting and whatever, tag it up with all of this keyword salad, and then those videos are being auto-played to Kids based on their similarity to content that the kids have already seen.
So, you give a kid an iPad, they have the YouTube kids app, they're watching a traditional licenced normal Peppa Pig video, the video ends and what happens? Well, fresh content automatically plays for them and that fresh content can easily take them from a licenced channel to an unlicensed one. And, from there they might go down a very, very dark hole. And their parent might not even realise it. So, when we talk about examples like abuse and harassment on Twitter or all of these videos on YouTube, many of which have been taken down.
After this article came out, there was a purge of content and thousands of videos were removed.
I'm sure there's many more still.
But when we talk about this, it's often treated as if it's a conversation about content moderation, right? How do we moderate this content? And, that is a question, that is a question that is worthy of lots of discussion. But what is not discussed nearly enough is how this is a question of product design. When you look at YouTube, you have to look at how that product is designed and what is it optimised for.
Well, it is optimised for you to watch as much content as possible because, well, that's how ad revenue comes in, right? And so you have an algorithm that predicts the thing that you're going to want to watch next.
And, that algorithm, as you can see, has been gamed. It's gamed by people who want to put other kinds of content in front of kids, or who don't really understand the implications of what they're putting in front of kids.
And, because we have this system where that content continues to auto-play and because it's based off of your historical content, if it auto-plays and sends you down this rabbit-hole, it'll think, "Oh gosh, let's give you more "like that stuff down that rabbit-hole." And all of a sudden, what you have is people going deeper and deeper and deeper into these universes.
Those are product design decisions.
Those are product design decisions at a very fundamental level.
And the same at Twitter, right? Twitter is a product that was built by four young dudes in Silicon Valley, who literally when they started it, what they said was they wanted it to be like if you were on the CB Radio network, as like a delivery person, right? So, bike messengers and delivery people, kind of like spouting off chatter and updating people on their locations throughout the day, which was a cool idea to bring that online, and then they didn't really think about how their cool idea might not work for everybody, even though that platform, as it grew, their user base became super diverse and they started getting lots and lots of reports, really early on, probably around 2010 at least, particularly from black women who said that harassment on the platform was really bad. But they wouldn't let go of some of the features of that original vision for so long, that they really under-invested in building product decisions in that would make it easier to report abuse, smooth out that process, help you know what happened to the abuse reports that you submitted, make it easier for you to block people and keep track of what's been going on with the people you're trying to not let into your life. All of these decisions, they spend a very long time ignoring them.
So now they're slowly starting to roll that stuff out, but we're talking 10 years after the fact.
I think we've spent a long time in tech sort of getting very very very enamoured by the idea that we're going to have the hockey stick growth. Upward.
And, when we do that we kind of set aside ethical concerns. We set aside all the stuff about whether we should do this, how is this going to potentially hurt society, because if the valuations are going up, or the money is coming in, nobody really wants to think about the rest of it. And we can see it play out in some really big ways. In fact, we're seeing it play out over the last couple of days.
If you've been watching the news, you know that Mark Zuckerberg has spent the last couple days in congressional hearings, being asked about the Cambridge Analytica scandal. So, for years, what Facebook did was it let third-party developers create these apps on Facebook.
And, up until the middle of 2015 or so, those developers were able to scrape a tonne of personal data about Facebook users.
Not just from the people who used the apps they were designing, but often times from their friends as well. And then, based on some loopholes within Facebook, a lot of people were able to keep that data when they shouldn't have.
And, then 87 million Americans had their data used and potentially used against them to, you know, strategically target and misinform Americans, during our 2016 election.
This is a full-page ad that Facebook has taken out, I think last week, in the New York Times, the Washington Post, the Observer, the Wall Street Journal, a bunch of other publications.
They took them out in like six different U.K. publications, to say like, "Oh, I guess we have responsibility for this." And, in fact, it's finally kind of gotten enough attention. It's finally shaken that company up enough that we're starting to hear a little bit more sort of mea culpas from them.
So this is one from Zuckerberg from last week. He said, "We're an idealistic and optimistic company. "For the first decade, we really focused on all the good "that connecting people brings...
"But it's clear now that we didn't do enough. "We didn't focus enough on preventing abuse "and thinking through how people could use these tools "to do harm as well." This is like the same thing Twitter said.
Right? This is like the same exact thing.
We were focusing on all the cool good stuff we were doing and so we just didn't think about the negatives. I think about these statements all the time, especially in comparison to the kinds of things that they tell investors.
So, this is what Mark Zuckerberg wrote in 2012 when Facebook was about to go public.
What he wrote to investors at the time, was, you know, "As most companies grow, they slow down too much. "They're more afraid of making mistakes "than they are of losing opportunities by moving too slowly. "But we have a saying, 'move fast and break things.' "The idea is that if you never break anything, "you're probably not moving fast enough." Now, officially, this is not longer the tagline of Facebook. In fact, I think somebody asked Mark Zuckerberg about this in the congressional hearings and his response was he didn't know exactly when it fell out of fashion, but now the official tagline is "move fast with stable infrastructure," which is a joke because nobody thinks that's the actual tagline. Nobody says that at Facebook.
Or they don't say that seriously.
This is the cultural DNA.
This is how the company operates, right? Move fast, break things is very much how that company was built.
Now I'll draw attention to a specific item he said here. "If you never break anything "you're probably not moving fast enough." You know, that might be true.
But I think there's a corollary here, and it's the corollary that isn't getting talked about. If you never slow down, you have no idea what you're breaking.
You don't know.
You could be breaking anything, and it could be inconsequential, or it could be like, I don't know, people's access to information, democracy, etc. Who knows? Because you've never paused to think about it. And, instead, what we have is an industry that's built on a lot of what I call wishful thinking. So, like Mark Zuckerberg focused on all the good that Facebook is doing, or we've got Jack Dorsey focused on of this real-time communication, we have all of these examples of the way that we design products and think about the features that we're building, as if things were going to be great.
For example, did you all see Tay a couple years ago? Tay is my favourite.
Tay was and A.I. that was built by Microsoft. And it was a chat-bot.
And, effectively, what Tay was supposed to do was learn from teens on Twitter.
Sorry, it was 2016 that she launched.
So Tay goes onto Twitter.
And she starts having conversations, just learning how teens talk.
Well, the whole point was that the more you talk, the smarter Tay gets.
It didn't work out quite as planned.
Within 24 hours, Tay had gone from "humans are super cool" to full-on Nazi.
Because, what had happened is immediately, immediately...
Trolls had gone in and said, "Aha, I see your chatbot" and they had trained her, not just to learn hateful language but actually to attack people.
So they went and they got Tay to go attack the women who were being targeted by Gamer Gate. So, a bot was going out and attacking people on behalf of Trolls.
So, the researchers at Microsoft, they did not want this to happen.
As you might guess, this was not as intended. And so they apologised for it and they said, "Well, we stress-tested Tay "under a variety of conditions, "specifically to make interacting with Tay "a positive experience." Well, there we are again.
We spend so long on thinking about how we can make things a positive experience, that we just lose sight of so much.
It happens in these little design details too. This is an email that my friend, Dan Hon, got from his scale.
And, it's not actually for Dan.
As you can see, it says Calvin.
"Your hard work will pay off Calvin.
"Don't be discouraged by last week's results. "We believe in you.
"Let's set a weight goal "to inspire you to shed those extra pounds." You may also notice that the average weight of Calvin is 29.2 pounds.
And Calvin has gone up 1.9 pounds in weight. Calvin is a toddler.
Calvin does not have a weight goal.
Calvin is two. He's older than two now.
So, what this scale knew was weight loss.
The only thing that it knew was weight loss. This product had been designed, this little notification, right, this little notification was designed to celebrate weight loss and to reassure people if they were experience weight gain because obviously the only reason that you would ever be concerned about weight is because you want to lose weight, right? And that weight gain is always bad.
They hadn't thought about what would happen in other kinds of circumstances.
And it's pretty funny if you get an email like this that is addressed to your toddler because your toddler can't read and you toddler probably doesn't feel ashamed of their weight yet.
But, it gets less funny in other circumstances. In fact, this is a different notification that they got.
This one is actually directed at Dan's wife. Push notification.
"Congratulations, you've hit a new low weight." She had just had a baby.
She did hit a new low weight, but think about this for a second.
I know people who have things like chronic illnesses, where weight loss is actually the first sign they're getting sick.
I know people who have gone to inpatient treatment for eating disorders and who have spent years of their life trying to learn to not congratulate themselves every time they see a smaller number on the scale. I know people who for whom congratulations, you've hit a new low weight, would be incredibly distressing and hurtful. And, also, just not relevant to a tonne of people's lives. There's a lot of reasons you might weigh yourself. There's a lot of reasons you might be tracking your weight that has nothing to do with trying to lose weight. And the fact that the scale had been programmed with one potential use case makes no sense. And the fact that you congratulate people for something that you don't know is good, again, this is wishful thinking.
Wishful thinking that we actually know what people want or what we know is going to be good for them. And we can see with the example that Eric Meyer had, that John alluded to a little bit at the beginning here, where, when he had just lost his daughter, it was the end of 2014, and he'd just lost his daughter that year.
He went onto Facebook on Christmas eve, and he wanted to see what was going on, his first Christmas without his daughter.
Wanted to see what was going on with family and friends, and instead what he sees is this feature.
So something called Year in Review.
It was the first year that they did it.
They've done it a few times since, they've changed it up a bit.
And, what Year in Review did, was it would take all of your most popular content from the past, so like the most popular videos and photos and things that you had posted all year.
Package them up in a little album for you, insert them into your feed, and say, "Hey, here's what your year looked like." And they would want you to share this album with your friends.
More engagement, more page views, etc.
So, the most popular photo he had posted all year, was of course the photo that he posted right after his daughter's death.
And, he did not want to see this in front of him again. He was not interested in reliving this.
He was not ready.
And he didn't want to share it with anybody. He certainly didn't want to share an album where it was covered with pictures of balloons and streamers and people dancing at a party.
He did not consent to any of this happening with his content.
So, Facebook felt really bad about this.
And even like the Product Manager who worked on this apologised.
I know the Content Strategist who worked on this feature and she felt really awful about it because it flew by her desk at the end of the year, there was a lot going on, the thread kind of got buried and it went out the door. This is what happens, right? But here's the thing.
We can talk a lot about how Facebook learned from that because I think that the people who were involved with this absolutely learned things from it, except that as a company, they've chosen to keep remaking the same mistakes. This is an example from the Fall.
This was in October, and Olivia Solon is a journalist.
So, I'm going to walk you through what's going on here, because there's a lot happening.
Olivia Solon is a journalist.
She writes about technology and it turns out, women who are public online who write about technology sometimes get treated pretty poorly.
She received a graphic rape threat in her inbox. She took a screenshot of it.
She posted that screenshot to her Instagram account because she wanted people to know about the kind of abuse that women, like her, receive. That screenshot was a popular Instagram post of hers in the sense that lots of people commented on it. Facebook owns Instagram.
Facebook wants more people who use Facebook to also use Instagram.
So what Facebook will do is it will take your popular Instagram posts and insert them into your friends' Facebook pages as an advertisement to get them to use Instagram too. So they had taken her screenshot of a graphic rape threat, wrapped it in an advertisement for Instagram and put that in front of her friends.
Enticing, right? And the thing is that this is fundamentally the same kind of mistake that they were making several years ago.
It's the same thing, right? They're taking content, pulling it out of the context in which it was posted, and doing what they want with it.
Turning it into an advertisement meant to increase engagement for their product. And we can see these kinds of like mismatches between your content or a user's content, and sort of product design decisions everywhere. I love this one from Tumblr.
Beep beep, Neo-Nazis are here.
(audience laughs) I know, right.
So, this is the notification that goes out if a topic is starting to like elevate in popularity like the topic of Neo-Nazis has been.
And in fact, I talked to the woman this happened to. She's an Irish novelist named Sally Rooney, and what she was, she was like, "I freaked out, went into all my settings "and said, 'Did I accidentally start following "'the Neo-Nazi tag?' I mean, where is this coming from?" It was coming from nowhere.
She couldn't find anywhere that this was coming from. It turns out, after she'd had a conversation with somebody at Tumblr, that the answer was probably because she had read a Tumblr post about the rise in Neo-Nazism, and that because of that they decided that she cares about the Neo-Nazi topics and that she needed a push notification about Neo-Nazis. Beep Beep.
So, I dug around a little bit into the conversation on Twitter after this happened. And I found a really telling Tweet from a fellow. I'm sure he's lovely at a personal level.
It's not about a personal thing, whose name is Tag Savage, who's the lead writer at Tumbler. He said, "We talked about getting rid of it, "but it performs kind of great." So, here's the thing about that.
Turns out, inclusion might be bad for your business model. Maybe inconvenient to have to think about the impact of your work.
It may be inconvenient if you're Facebook to have to ask some tough questions about how you've been making money and sort of what compromises you've made along the way. It might be inconvenient if you're YouTube to think that like, "Oh gosh, what if we question "all of this auto-playing content? "That would affect our revenue." Yeah. It might be inconvenient to your business model to have to think about ethics.
I saw, actually, a quote from Mark Zuckerberg just this morning, that apparently in testimony there was a question about business model and so I can't remember who asked the question. The Congressperson asked the question, you know, "Are you prepared to look at Facebook's business model?" And his answer was, "I don't understand the question." I don't think that we can really solve some of these problems, unless we can take a very hard look at the things that we've made and we can accept that the world might need to look dramatically different.
Because we are building a lot of bias into all kinds of technology products.
All of these ways that we create these experiences that we want to be delightful, but we're designing for very narrow use cases and very narrow people, they're creating a lot of problems.
And those problems are embedding themselves deeper and deeper into the products that are out there. I'm going to talk a little bit about one of my favourite examples, which is in things like image recognition.
So, this comes from a story out of FaceApp. FaceApp was super popular last year because they had this really cool new selfie-morphing feature, so you would take your selfie and you could turn it into like a younger version of you or an older version of you, or a hotter version of you.
And if you turned it into a hotter version of you, it would do things like, you know, smooth out your skin, and get rid of any wrinkles, and, slim your cheeks so you look a little more chiselled. Also, if you weren't white, it would dramatically lighten your skin.
And as people started talking about this, the CEO of FaceApp actually apologised and he told us, really, what was going wrong here. So let's look at what he said about this. He said, "We are deeply sorry for this unquestionably serious issue. "It is an unfortunate side-effect "of the underlying neural network "caused by the training set bias, not intended behaviour." I'm going to unpack that a bit.
Here's what he's really saying.
He's saying the underlying neural network was the problem because of training set bias. What he means by that is, we took a large quantity of photos of people that we thought were attractive.
Those photos were pictures of white people, and that is what we trained the A.I. on.
Here. Here is what attractiveness looks like. And that A.I. learned.
Oh, that's what attractiveness looks like.
So, when he says it's an unfortunate side effect and not intended behaviour, I would say that's actually incorrect.
Oh, no, this is operating exactly as designed. You taught the system what attractiveness is. And it is reflecting that back to you.
You said, "Oh, okay, this is what beauty looks like. "I have taken these selfies and I have made them "more beautiful according to the criteria you gave me." The system works.
The problem is you didn't think about the bias and the data set you were using.
And these problems are so constant and they're so frequent.
They are cropping up all over the place.
And one of my most favourite examples, the one that we all could have learned from years ago, comes out of Google Photos.
Back in 2015, Google Photos launched a new feature where they would auto-tag your images, which could be kind of a cool feature, right? If you upload all of your photos, like all of the bazillion photos that everybody takes today. If you upload them all to Google Photos, and you have those synced, it would be nice to be able to type in a keyword and find all the pictures of it right? And so, you could be like, "Okay, this is a bicycle, "and this is a cat".
But then it took this photo of Jacky Alciné and his friend at an outdoor concert.
And labelled them as Gorillas.
Gorillas is a racial slur.
So this story blew up.
It blew up.
It was all over the news.
People were calling him from publications around the world, because it was every single photo they took that day too. Wasn't just one.
It was every single photo that they took that day, had been categorised as gorillas.
Now, if they had been categorised as literally any other kind of error, I don't think this story would have gotten any attention at all.
We would have written it off.
We would have written it off as, "Well, you know, "image recognition is pretty new technology. "The system's going to get it wrong sometimes." Which is true.
But I think what this allowed us to really see, is why these systems are getting it wrong.
Yonaton Zunger worked at Google, worked on this product, he was like one of the chief architects of it, and he said, you know, we're going to fix this. And in a Tweet back to Jacky, this is what he told him. He said, "We're working on longer term fixes "around both linguistics, words to be careful about, "and photos of people." So like before we tag stuff with words that we've deemed to be on a sensitive list, we're going to kind of like ask some questions about that. "And then we're also going to work "on image recognition itself, "e.g. better recognition of dark-skinned faces." What this says to me, just like what we learned from the FaceApp CEO, is that Google thought it was fine to launch a product to the world that was worse at identifying dark-skinned faces than light-skinned faces.
They thought that was okay.
In fact, it's not just Google, of course, because there's now a bunch of research happening to figure out if this is quantifiable, and it is. There's a researcher at M.I.T. Media Lab, Joy Buolamwini, and she started looking at this more quantitatively.
And, she was looking at all of these different popular data sets that are in use, and she said, "You know, when a person in the photo "is a white man, the software is almost always right "in its ability to identify that the person is a man." So she's looking at a really basic identification. Right? Can the system get it right with regards to gender. "The darker the skin, the more errors arise. "Up to nearly 35% for images of darker skinned women." I am very uncomfortable with this being the reality in our industry.
I think that we should all be pretty uncomfortable with this being the reality in our industry. Right? I'm not okay with continuing to design products that work better for white people than everybody else. And I think that's what we've been doing.
And we've often been doing it without acknowledging or admitting that that's what's happening.
But if you're willing to put a product out that has only been tested on white people.
If you're willing to put a product out that does a better job for white people than other people, then that's what you're doing, right? You're saying it's more important that it works for white people than other people.
And these kinds of biases, they're getting baked in all over the place. For example, Coco is an image set.
Coco takes 100,000 images from the web, from all over the web.
And it labels them.
And so what they did, it's a project between Microsoft and I think now Facebook is involved, and an A.I. startup is involved. And, what they did, is they labelled these 100,000 images right? And then they took that data set and they applied it new photos.
And they said, okay, you have learned.
You have learned what the world is like.
Well, the A.I. that they trained on it, started automatically associating kitchen implements with women.
So what that means is that it was learned from the data set, the A.I. was doing the best it could do.
From the data set it was fed, it learned that pictures of kitchen implements were likely to be affiliated with women, so when you showed it new pictures of spatulas or whatever, it would go, "Yup, women." These are the kinds of relationships that are getting built in to the underlying software packages that are then being used to build other things on top of them, right? So if you're trying to do something with image processing, if you have a product feature that includes natural language processing, let's say for example, chatbots, all of those kinds of systems are relying on underlying artificial intelligence that's relying on datasets, where God knows where that data came from.
And these are not only design problems, but these are also design problems.
They are not just the problem with the technologist working on the project, these are the problems of people like us, who might take a package of natural language processing, for example, and assume that that's going to be just fine when it's working within whatever feature we're building. So what are we going to do about all of this? Like, how do we actually start to rectify the problem? Well, the bad news is that it's a very big problem, and that if you're working in a day-to-day role in a company it's not like you can just upend the entire apple cart. It's a lot going on here.
But I think that there's a lot that we can start to think about.
And there's ways that we can start to make an impact within our teams. And I think the very first thing that we need to all be doing is really rethink what it is we think we're doing here. What do we think our job actually is? What matters? Zeynep Tufekci who is a digital sociologist, whose work is amazing, you should definitely be following her if you're not. She says, "Silicon Valley is run by people "who want to be in the tech business "but are in the people business.
"They are way, way in over their heads." And that is actually why I was so frustrated, well, several reasons, why I was so frustrated last year when the Google memo came out because one of the key recommendations that James Damore, who wrote the memo, had. Because he felt like Google was over-investing in diversity and that the reason there weren't more women in technology was because they just didn't want to be there and they were biological differences that were keeping them from being there and there was some very tenuous research he included. But, one of his prime recommendations was that what Google needed to do to fix this problem he identified, was de-emphasize empathy.
Being emotionally unengaged helps us better reason about the facts.
I'll tell you what.
None of the problems that I've identified in this talk today, none of the problems that are really plaguing the tech products that we are designing and making, are going to be solved by de-emphasizing empathy, and by being emotionally unengaged.
We're not talking about reasoning about facts. What this is, is this is a misunderstanding of what the job is in technology.
This is somebody who thinks that the job in technology is all about facts, that data is neutral.
But what we know is that it is not neutral, that none of what we create is neutral.
And so, we can't afford to be emotionally unengaged from our work.
In fact, we need to be more emotionally engaged in our work. And we need to be thinking about some really tough questions because I can tell you that, when you're not just reasoning about the facts, when we have to decide what good means, like what is a good algorithm? What is a fair algorithm? Who is deciding that? Who's involved in that process? Who understands historical context on our teams? Like, if you're going to be designing a feature that is going to impact a certain segment of the population, and you don't know anything about that history of that segment of the population, and nobody in your team does either, do you have any business even working on that? In the United States, we talk a lot about things like how technology is affecting, for example, whether or not you can get a loan to buy a home. Also in the United States, we have a very long history of extremely racist housing policy that allowed people to systematically exclude people of colour from getting traditional home loans, often times based off of where they lived.
So, if you're designing a system where, let's say, your zip code is going to be part of the equation that determines whether you are eligible to get funding for something, and you don't know anything about the history of using people's location, where they live, to deny them credit, can you design that product? Whose job is that? This is not a traditional job that we have in technology. This is not "The designer can just do it" right? This is the kind of role that we haven't really talked about.
Who is responsible for thinking about unintended consequences? Sometimes people ask me, "If we're talking about ethics in tech "should we have an overarching ethics body "for the organisation or is this the responsibility "of individuals?" And I'm kind of like, you know, if you're going to look at this in a serious way you have to think about it at all levels, right? Like, should there be oversight across an organisation? Absolutely.
Do people also need to have an individual practise for this? Yes.
Should teams be prioritising this? Yes.
Should managers be managing this? Yes.
Like at every single level, at every single step of the way, at every single bit of the process, you have to be thinking about unintended consequences. And if you think about unintended consequences, somebody else actually has to listen to those concerns. We need to build it into our practise to uncover our own assumptions, our own biases in our work, right? To sit down and think through the ways that things might fail.
Because I think when it comes down to it, our own ideas about who and what is normal, who or what is typical, they say a lot about us, but they actually say very little about our audience and the world.
So when you look at something like Eric's experience with the Year in Review and the other people for whom it did not go well, that is somebody's apartment burning down.
What you can see is there's actually a whole lot of assumptions that have been built in.
You know the saying, about when you make an assumption, you make an ass out of you and me.
It's my dad's joke. It works. It's a helpful reminder.
When you assume things, it's very easy to not see potential downsides. A lot of assumptions in Year in Review.
The assumption is that the user probably had a good year, that they want to relive it, that they want to share it, and that their most popular content from the year is going to be good content, it's going to be positive content.
So a whole lot of assumptions also that if any of those things aren't true, that it's not going to be that big of a deal. Move fast, break things, and the consequences are going to be small enough that it doesn't matter.
So we need to get a lot better at unpacking the assumptions that we are building into our work.
We all make assumptions, everybody has to make assumptions all day, you can't walk around the world without making assumptions because you have to make decisions without having perfect knowledge.
But if we're not taking stock of the assumptions we're building into our design decisions, that we are going to have huge unintended consequences all across the board.
And maybe most importantly, we need to be thinking about the people who are the most vulnerable first.
When we are paying the most attention to the people who are the most vulnerable, the people who are often at the edges, what we can actually do is we can ensure that products work for people at all stages. The people who have sort of the fewest needs, that's where we spend a lot of our time prioritising right now, the people who are the safest and the more secure. I think we should spend a lot more of our time thinking about those folks who are the most vulnerable, those folks who need our help.
And we have to think about whether our work is going to be used to harm someone.
Really explicitly, how could this be used to harm someone? Zoë Quinn is the woman who was at the centre of Gamer Gate, the woman who was attacked relentlessly for years. And she started asking this question, how could this be used to hurt someone? She says if you're not asking this question, your design and engineering process has failed. When you start questioning the source of everything that we're working with, where are we getting the data in our company? Where are we getting the information that might be going into this algorithm? Where are we getting all of these decisions? Who decided? Who chose this data set? Who said that this was okay? How did they test it? Was there criteria for that? Did we not use other sources of data and why didn't we use those? Was anything excluded from the data set? Who decided that it was good enough? How do we know if it's working? One of the researchers at the M.I.T. Media Lab who was working on that project looking at how well different data sets used in image processing were actually reflective of different races and genders, said that a lot of that stuff goes out the door by doing the down the hall test, which is like you get someone from down the hall to take a look and see if it works for them, and they go "Works for me!" And it goes out the door.
The down the hall test is probably not a good representation of the diversity of your audience and the world. And it allows you to ignore that some of the data that you have might be incredibly biassed.
And I think we really need to be demanding more from our companies, and this is hard because I think that for a lot of us it's like, how can we really affect these kinds of big changes, right? But I think we need to ask our companies to be specific about what they actually want to stand for. Have they written these values down? And, like, what are they going to do? What are they going to do when they have tension, right? Like, what needs to change in the way that they do business and they way that they're designing products in your process and who you hire.
What needs to change, and then when inevitably a question comes up of should we do this, because it seems like it could hurt people, but also it might make us some money, how do you respond to that? What happens? If every single time there is an ethical concern, and engagement or ROI will win that argument immediately, then you don't really have ethics in the first place. It's just a piece of paper.
So what I want to leave you with today is that, you know, oh, that's a random thing on that slide, (laughs) cool, cool...
Is that if we want to design to include people, then we need to be thinking a little bit differently. So I'm going to go back to the conversation about the U.S. Census, where I started.
It's just a few form fields, it's just a little bit of data, and how it is designed, how it is built, has huge outcomes for people.
It matters in real peoples' lives.
It matters in the way that people interact with the world. It matters in whether they feel safe, and it matters in terms of their ability to get resources and to be represented.
Ultimately, I think design is very powerful. It is more powerful than we often give it credit for. And so, it's not a question of whether our work has big power for people, we know that it does, we have tremendous evidence that it does.
The question today is how will we use that power? Do we want to use that power to continue as things are, to keep chasing the same sort of tired dreams about larger and larger unicorn valuations, or whatever it is that your company has been focused on, or do we want to push back against business models, products that can harm people and can leave them out? What are we going to do with the power that we have? And what do we want to see in the world as a result? So thank you for your time this morning, I appreciate it.
(Audience applause) (rhythmic synthetic music)
