Magical UX and the Internet of Things

Josh Clark at Direction 2016
Sign up for Conffab Free to watch this and hundreds of other videos for free
(whooshing) - I wanna introduce a wonderful speaker to kick off with us. You'll almost certainly be familiar with his work, he speaks around the world, he's worked on amazing projects. We have the great good fortune to get him out here in the past to speak and give workshops and in fact, I had very short notice a couple of years back, he did not one but two presentations for us. I think, in the last time, we were at the convention centre which some of you had certainly will have seen. So, Josh Clark has been working in around the web and user experience for many, many years. He's literally someone who needs no introduction. I think, he's the perfect person to kick off today to think about what comes next for the systems and things that we are building, particularly the way we interact with them. So, to kick off, the first ever Direction here in 2016, please welcome, Josh Clark. (audience applauding) - Thank you so much, pardon me. Thank you so much, John. Thanks for those good words, John, and thanks also for the introduction. The introductory comments about the world that we find ourselves in today. You guys, I'm not gonna lie, it's a little hard for me to be up here today. I'm feeling a little confused and distraught that my country just gave enormous power to a dangerous man, a demagogue, a bigot, a sexist, a vindictive narcissist, and it's hard to read that not as anything other than millions and millions of people choosing hate and fear over love and optimism. There likely to be some dark times ahead for all of us around the world. I think, he's gonna have a huge impact on global culture, global security, global economy, I think our jobs and our work are gonna be uncertain. So, in the context of this frankly global tragedy, it's a little hard to find optimism. So, I appreciate John's words, galvanising words about that we do have an opportunity to make some change, to see this tragedy as a design challenge because as designers, we know that the small changes, small interventions can have a big difference. And I'm talking about things really do make people's everyday lives better and the values that we bring to our work and the techniques that we use, and the projects we choose, they matter. They matter just in ways large and small in the world's lives. So, I think John's asking us to consider what shall we make to make world a better place, to make the web a better place, to make people's lives a better place. So, in that context, I'm talking a lot about, thinking about what the opportunities are that we can make now with some emerging interfaces. These remarkable things are happening now in the way that we interact with information and the digital services that we create. And there are things that after 30 years of essentially working with mouse and keyboard exclusively, we suddenly have this explosion of new ways to talk to these digital services. Touch came on fast, but speech and predictive interfaces through some big data and machine learning, natural gesture, camera vision, all these biometrics, all these new ways to talk to digital services and get information from them in ways that I hope and I believe that we can use to make more humane and meaningful interfaces for people, more effortless ones, more accessible ones, a whole lot of it of it comes down to how are we going to use those, new opportunities and what values do we bring to them. So, that's what I'd like to talk about today, all these new opportunities that we have to combine and mix and match new kinds of input and new kinds of outputs to create even physical interfaces to digital systems. It's a brand new thing that's just beginning to start and all of us are involved in figuring out what that looks like. So, I think, I'm specially excited about, thinking about the combinations of these. So, for example, if you combine speech with natural gesture, you get some new kinds of interesting combinations. In fact, I bought a demo, it sounds a little bit vague to talk about physical interfaces to digital systems and natural gesture and speech. So, here's a demo of what that looks like when you combine those two. - Expelliarmus. - Reductor. - Immobulus. - Expecto Patronum. - Yeah, Expecto Patronum, friends. I'm talking about spells. Say a word, make a gesture and something over there happens. Only the magic that we're talking about is these digital systems that we've typically kept to screens but now are escaping out into the worlds that we inhabit everyday. I don't know if you, guys, were paying close attention there to that video but if you really look closely, you could see that they are waving these sticks around. I did some research and it turns out they call these things magic wands. These are devices for transmitting content and media at a distance. But crucially what they're doing is collapsing the time between intent and action and that's really what our jobs boil down to at the most fundamental. How can we take this thought of what I would like to happen and create and interface to make that real, make that true. Of course, this idea of magic is something that collapses those things instantly. I did wanna say as I did some research, I managed to find that you can also buy a magic wand on the internet and you, guys, this is my magic want, so nerdy. I'm pretty new at it so I have to be careful about where I point the thing. It's what might happen but I'm gonna give it a try over here in this candle and see if I can do it, are you, guys, with me? All right so, here we go. Direction illuminatis. Ahh, all right, okay. (audience applauding) Thank you. Let's see if I can get it here. Direction noctunis, see it takes a little bit of practise, as some of them, it's little tricky things about some magic, right? Noctunis, all right, good. So, thing is, all right, it didn't exactly work right off the bat. One of the things we have to think about as we start to try to give physical objects some digital presence that we can control with devices like this and, by the way, this is just a piece of technology, right? This actually is a universal remote, an infrared remote control with 13 gestures that you can control your TV with this thing, right? These are these remote control LED lights. So, this is a parlour trick, right? But I think it illustrates something that, it took me a little while to get these things to turn off. If this was the only way that I could this, if the batteries ran out on my infrared controller, I'm stuck with these things on or off, right? It's having these mechanical switches that we can turn things on or off, let's not break the physical world as we try to enhance it with digital stuff. But the idea here of what we're showing is that, again, it's collapsing intent to action that these things right here, a civilised pointer from a more civilised era in a way, right? We're collapsing this technology into this little wand. A great man once said, if you use a stylus, you've blown it so let's put aside our wand and see what happens if we try to do some of these magic with our bare hands. This is my friend, Aral Balkan, he looks a little tired here because he's just come off of a 24-hour hackathon and I wanna point out, see those wine bottles in the back there? He's actually on a yacht off the French Riviera at a hackathon. So, it's like, I don't know, like a hackathon of the 1% or something, I guess. (audience laughing) He would hate it if I said that, please don't tweet that, you guys. Aral has moved on from doing what I'll show you essentially a parlour trick here, to work at much more important things. He's written something called the Ethical Design Manifesto which I encourage all of you to take a look at. Tweet that out. But anyway, what he did here is he brought to this hackathon a phone, a projector, a laptop, and a Kinect from an Xbox, this natural gesture, and this is the hack that Aral put together. - So, you're sitting at home on your sofa watching television and something interesting comes on and you wanna share it, say, you tweet it. So, I walk up to my TV and I just waved at it so it knows I'm there and then when something interesting comes up, I can just grab it and boom, put it over there. - [Josh] Wait, what? - You grab and boom, I put it over-- - [Josh] Oh no, he didn't. - So, I'm just holding it in my hand and I'm putting it on my phone and that's my hack. - I just wanna be clear, he did this overnight drinking wine on a boat (audience laughing) which suggests it's probably not that tricky a thing, right? I mean, I don't wanna take anything away from Aral, a great designer and developer but if he did this really quickly drinking wine overnight on a boat, there must have been something to it and you think about it, all that he did was teach the Kinect this gesture, take a screenshot and then you can walk around like you got it in your fist here, right? But now, it's already on the phone, touch to release. These two most simple interactions for these two things but when you combine them, it creates something really surprising and effortless and at least in the context of home entertainment, somewhat useful, right? But the point that I'm making here also is that all of these things are just regular everyday devices. This is the stuff, standard, middle-class devices that we all have, it's a phone and a laptop, a TV, perhaps you've got an Xbox game system in your living room but combining them as technology that just lying around suddenly can create this really new things. In other words, we're not waiting for a new technology to be invented, we already have tonnes of raw material to combine and create new kinds of interactions that hopefully can have some lightness and humour but also utility and humanity to them. So, what are the possibilities with that. That's what we're gonna explore today and we do it in three acts. We're gonna talk a little bit about the relationship between magic and technology, what happens when physical meets digital, and how do we start to actually work with this? What happens if we actually, purposely try to work with magic as designers? How does that change our perspective? So, let's get started with the first act, magic and technology. I'm gonna begin with what I think is a probably fairly familiar quote to most of you from Arthur C. Clarke that, "Any sufficiently advance technology is indistinguishable". The design community is co-opted the language of magic and delight without actually delivering the wonder that's associated with those words. We talk about it all the time, we're gonna delight our users with this really efficient web form but somehow, the delight I think has gotten little bars, gotten little low on delight, right? The thing is that delight deprecates. The thing that is amazing and magical today is just the norm tomorrow. So, we have to constantly really lean into this. What are the new possibilities? Good news is, like I said, we have tonnes of raw material to work with, to invent new kinds of delight and wonder and utility and meaningful interactions but I think that one of the things that's interesting about this, is you think about what is magic and one way to think about it is that technology becomes invisible. This is a quote from Alan Kay, one of the creators of the graphical user interface. He said this back in 1982, two years before the Mac came out and what he's talking about here is not creating an invisible interface like a lot of people are interested in now where just technology just happens around you. Obviously, the technology of that day was very visible and right in front of you. The idea what he was trying to get at is that technology should get out of your way in a same way that a pencil gets out of your way, just channels your thoughts. How can technology bend to fit your will and intention and ergonomics in a way that's that effortless. And he also was talking about it in terms of a sense of fantasy. They can create essentially illusion that things are simpler than they are, that's the whole idea of the graphical user interface pointing at pictures. In a sense, all user interface's illusions, this little thin layer of magic that we stretch over this churn of ones and zeroes that would otherwise be completely intelligible. We do it with cars, right? Somehow, we believe that driving a car, the way that it operates is as simple as turning a wheel and pushing a couple of pedals, where our HVAC systems are all about turning a dial. In other words, very successful interfaces are often about creating a new and simplified mental model that we are confident about. So, that's part of the possibility now if we think about, all right, let's actually make some magic, that's the idea of this thing. How can we hide technology into physical tools that let us manipulate the digital world? Now, where do you get one? The good news is, you don't necessarily have to go to a store like this. Although the personalization options are always intriguing if we see all the different models that are there. The truth is, is that, you guys, are already have your own magic wands. The thing is that the shelf where you got yours looks less like this, and more like this. Either your phone or your tablet is your magic wand. The phone is magic 1.0 and what I mean by that is that we etch these digital interfaces onto these slabs of glass and we carry them out into the world, and naturally interact with the world. It's a bridge between the digital and the physical so another way to put it, is that the phone is the first Internet of Things device that's really gone huge. And I know what you, guys, are thinking, like I've heard about this Internet of Things, think I've got it. I thought it was refrigerators that's saying and bracelets that sends you a text when you have a body odour problem. Pretty sure that's what it's about. We do have a lot of gimmicks and gadgets twice for the reg right now, experiments that are worthwhile, worth exploring but by Internet of Things, I really mean something a little bit broader which is that it's an everyday object, it's embedded with sensors and a processor and a network connection. So, the phone was the first truly everyday device to get imbued with all those things and go truly global and mainstream. It's sensors are new and revolutionary bet, the smartphone is a device that made sensor-based computing in everyday phenomenon, moving, interaction off the screen and into the world around us. Remember, the first time you saw somebody use Shazam to identify a song, (mimics explosions) right? Interaction with the world around us. Specifically, interaction at the point of inspiration. So, mobile has been incredibly powerful because for the first time wherever you are, you can get information or media or services or commerce about the thing right in front of you. Often again, with those sensors, using the microphone or the camera, to read text with Google Translate and just instantly see what that text is, but the computer again that interacts with the world around us. MyStudio, big medium, designs, user experience, how people actually interact with design systems so it's about crafting the interface and for the last decade, been focused really particularly around mobile but I've found that lately those experiences have been jumping off-screen and into the physical world. As I'm working with big fashion brands that design clothing as a digital interface, where asthma inhalers as an interface, retail spaces, jewellery. So, in other words, what we're starting to see is that mobile is bringing computing power to immobile objects because phones are so common place, product designers can often assume that there's this powerful network computer nearby. You put a sensor in something and your phone can start to control it or vice versa. So, suddenly we're seeing all these innovation in things that are decidedly immobile and traditionally dumb, locks, and light bulbs, and garage doors. Everything can be a client or a sensor or a controller. I don't know, maybe you can even do it with a diaper. (calming piano music) (iPhone message alert) Yeah, so this is the nappy notifier from Huggies. So, you insert this little sensor into the diaper and it detects wetness and since it knows what your diaper stock is, it can also order for you to keep you knee-deep in diapers which is probably better than be in knee-deep in something else. So look, this is obviously also a novelty and so, what I'm not suggesting here, you guys is, hey diaper UI, forget this whole web thing, diapers, you guys, stop now, leave the room, get after it. What I'm saying is, if something is literally disposable as a diaper, can become an input to the web or to our digital services, we can do it with anything. I think that that is super relevant to all of us in the way that we approach our work because it turns out the physical world is where we actually live, where we actually consume the services that we create. So, in the context that I've been talking about though is seeing the phone at the centre of everything which means, for better or worse, better and worse, I should probably say, the phone not only has the power but also our attention. Even as we embed sensors everywhere, (orchestral music) most popularly, onto our bodies, this experience tends to be mediated through screens. So, this is an Apple ad, so it understandably emphasises the phones but I think it captures an accurate dynamics that the more we try to use technology to build a stronger tie to the physical world or even to our physical selves, the more we shove a screen in between us and the world. So, it is making us more powerful, no doubt about it, come to the agency but it's also taking something away as we're more and more absorbed in our phones. I mean, we've all been this dude, right? You know what he's texting, right? "This whale watch sucks, we haven't seen anything". Been missing the thing right in front of him. And I think that we're just missing things that are happening right in front of us with people we care about, we don't even know it. It's a little tidbit for you. The average smartphone screen time, three hours and 16 minutes a day, 20% of our waking hours are spent looking at these little screens and I would say that those of us who've been focused on creating mobile experiences for the last decade, I think we got it wrong. I think that we focused on engagement and that was a kind thing to do and I think instead what we've done is we've created and designed for addiction and to trap people's attention. And so, perhaps, engagement isn't such a great goal after all because it's disengaged us from things that matter more. So, if this is the awesome thing about mobile, it's also it's Achilles' heel. It's the thing that at the point of inspiration, we turn away from the thing that's inspiring us to go into the screen instead. Screens are always gonna be with us, I love screens. I think they have so much potential to, and have changed the world in a lot of ways but perhaps, screens can merely caption our lives instead of frame them. And so, I'd like to propose that we find a new way to move from this idea of availability at the point of inspiration, to interaction with the point of inspiration. How can we push those same digital interactions off of our pocket computers and into the objects and places that inspire us? So, here's fun thing that maybe get you thinking about that, it's Neiman Marcus, the department store, just installed this mirror at some of their stores and they call it the memory mirror 'cause it holds on to your image so after you do a little spin, you can see your outfit in 360 degrees, dressing room instant replay. Which also means that you can compare your current outfit to one you tried on a minute ago or see your dress in different colours. So then, you can also take those mirror memories and share them with friends to ask their advice. It turns out super commonly used case, right, is this mirror experience as like, how does this look? Should I get it? It's this, you know, and it's like in this case, what the mirror is capturing that and can share that for you. So, it's the thing that's looking at this small piece of context but trying to figure out how can I make you more present in that moment and interacting with the actual key activity instead of distracted from it at crucial points. Now, I don't know if you, guys, know this but this very technology was until recently only available to evil stepmother queens. (audience laughing) And what I'd like to propose is that we raid the evil queen's castle for ideas because we have centuries-long fascination with magic objects. What if magic could bring things to life? Make us smarter or stronger or live longer? We have all of these design patterns, centuries of design patterns that tell us exactly what these things should do and be if we think of it through this lens. Sometimes, it's helped us with the mundane details of life. All these stories of how these things should work. What if objects could speak to us? Share us with us what they've seen and what they know. You get things like the sorting hat from Harry Potter. - Hmm, difficult, very difficult, plenty of carriage, I see, not a bad mind either. - It is mind reading wearable that knows you better than you know yourself, right, that can make decisions, you don't know how to make, it's Google Now in hat form, (audience laughing) or household tools that are just smart enough to do the dumb stuff that we've gotta do. A Roomba for the mediaeval era, right? It's a lovely book about some of these topics if you're interested in it, it's called Enchanted Objects by David Rose of the MIT Media Lab. Again, he's pointing out that we have this centuries of experience of imaging what objects could for us if they were smarter or more magical. So, the mirror is one example of interaction with the point of inspiration. What else can magic teach us? What if we go back to Dorothy (speech drowned by music). So, her magic shoes, her ruby slippers give her the power to escape and that is a super strong motif in magic, this idea of escape. And we often use technology to do it too, right? For those of you who, maybe a little bit introverted, uncomfortable in groups like this, I feel ya, I understand that pain and I think we all know how to escape that problem when you're at a conference. I'm just gonna pretend I'm really busy here in the corner, you escape that difficulty. All right so, how can we think about this very same notion of magic and escape with a little bit of inspiration from Dorothy. Well, the Dorothy Project imagines the same idea using your shoes as a magic escape method. - [Woman] Going on a first date? Select Receive a call, create a fake contact and slip a ruby in your shoe. Now, when things get unbearable, click your heels together three times and answer that pressing phone call from your boss. - [Woman] Oh sorry, my boss is calling, I've gotta take this. - [Man] Oh, I'll see you on Tinder, I love you. (audience laughing) - What's not to love, what's not to love? So, more recently than the Wizard of Oz, the office supply chain, Staples, imagined another magic object for both escape and creation. - So, N equals what, Josh? (dramatic tango music) (horse neighing) - [Surgeon] Today, we'll be performing a triple half and over procedure. - [Nurse Assistant] But you've never done that before. - [Surgeon] That's okay. - [Man] Wouldn't it be nice if there was an easy button for life? Now, there's one for your business. Staples, that was easy. - That is basically the American health care system in a nutshell right there. (audience laughing) But a magic button that can do things for you on demand, summon your magical wish, we actually have some things like this. In fact, there's tonnes of these. These little Bluetooth buttons, this one is called bttn that you can set the trigger, any arbitrary digital API. Many of them use, IFTTT, if this then that which if you're not familiar with it, it's a collection of APIs that let create triggers to trigger another digital API. So, something that happens on Twitter can make your lights turn on, for example, or pushing this button can summon a taxi or send a text, or tell your kid that when they come home, hit that button and sends a text automatically that says, "I'm home safe". So, this idea that you can push a button and make something happen is very much at the heart of what Amazon has been doing with its buttons for Amazon Dash. The idea here, right, is stick one of these to your washing machine, press a button to automatically order detergent when you run out. So, you're essentially adding a magic button that's specific to this context to your washing machine and it turns out, that is a bit unusual, just ask Bill Murray. - Generally you don't see that kind of behaviour in a major appliance. - That's right, Bill. (audience laughing) Or for example, if you run out of coffee, you can instantly order some terrible coffee. (audience laughing) That the point of this, instead of it's turning this into interaction with the point of inspiration, right? That is saying, well, if I want the thing, how do I make the thing give that to me? This is not the future we want right here, right? With all kinds of logos all over our homes, that's not what we're looking for, although, I will call it that it's really interesting way for Amazon to explore this concept and prototype with very low financial implications, because, essentially, they've got the brands to underwrite this experiment. So, I think, one of the things to think about is to be creative about how can we create worthwhile experiments that can be nimble and funded in some ways that we aren't just holding back but trying to make something happen and I think it is working out all right because Amazon released the same hardware as the general purpose, Amazon Web Services IoT button, so that makers can spin up their own, connect the Amazon services to trigger all kinds of events. I would encourage if you're interested in exploring some of these to try things like the bttn or this Amazon IoT device to make it really easy with a basic set of web skills to wire these things up to the services you care about. What's the push button service that you would like to create? So, this is the time to gather Act I here, this connection between magic and technology. Let's round up what I'm talking about here that the goal is to make technology invisible and that doesn't mean literally get it out of the way but to bend to our lives and to the creative acts that we wanna take or the context that we are in so that it doesn't distract us from what we're doing but rather enable it. Thinking about what we can to at the point of inspiration and particularly what happens when that point of inspiration has some embedded smarts or smarts are brought to it by a mobile device. Again just, as we start to think about what should physical interfaces be and look like? We've got tonnes of design patterns already. All right, well, that brings to Act II, and what these examples are showing us is that physical objects are becoming digital too. Physical interactions with digital APIs, in other words. And so far, all of us have been attaching our digital services, only the screens but here's the new opportunity is to attach those services to the whole world. In other words, the world is the interface, which, by the way, it's always been the interface, right? This is where we live. How can be just start to light up that world with the digital services that we've been creating in this separate parallel alternate universe. I think that the ability to do that means that this is much more than Kickstarter projects or hardware startups and it's more than just fitness trackers and wearables and gadgets for the rich. It's not about novelty, it shouldn't be. So, I think this is really relevant to every business and every service. So, I'd like to share four ways that I'm thinking about this new kind of physical interface. The first is just that the world is a data source. There's so-called the ambient intelligence. These are interfaces that listen and try to add value through big data. So, for example, Progressive, the insurance company, offers this snapshot sensor, plugs into the standard data port on every car that's been made since 1996. Your car has an API that is a talking, communicating, complaining car basically, right? So, Progressive uses this to make insurance premiums tie to your driving habits. So, in this case, you can make the world a passive interface that gathers data to optimise business processing cost or to give transparency to customers about why they pay, what they pay. But there are many directions you can go in with the same technology. So, I think, it's one thing it's useful as we look at this and we see examples of this, use that as an inspiration to go in new directions. So, for example, Automatic does that. It uses a very similar sensor in your car to deliver a very different service. So, Automatic is focused on delivering actionable advice and insight. So, you plug in the sensor and it syncs with your car and its goals are to reduce environmental impact and improve safety so it watches how you drive and takes note when you accelerate quickly, so it gives your feedback, and they have this programme that's specifically aimed to teens and new drivers. So, it's a driving coach but it doesn't just watch you, it watches the car so it can decipher what those dashboard icons mean and suggests how to fix it, let you dismiss the alert. So, it's empowering in that way, it's like a magic translate for your car that helps you be in better control of that environment. So, Progressive uses theirs to give the company insight, Automatic uses it to give the driver insight, both are legitimate, both are very different directions to go with this passively gathered data. Here's another example of that. It's a project that I worked on in designing user experience for called Asthmapolis at that time, they changed the name to Propeller Health because Asthmapolis (audience laughing) and it's its way to help people with asthma control their asthma or understand how well it's being controlled. So, you just put this little cap on the top of your asthma inhaler and every time you take a puff, every time you have an attack and take a puff, it registers the time and location of that and relays that to the service through your phone when that connection is available. And you get these reports daily just letting you know how well your asthma is controlled. It turns out that already pretty helpful. As human beings, we tend to be a little overly optimistic about how well we're actually doing but then every week, because of the time and location information, actually, it's very useful in determining how to improve your asthma or to stay in the direction that it's going in, is that you get these actionable advice about how to improve your condition, right? So, that's already really useful just for the individual, for the individual patient, but what's really exciting, is that because these are distributed through clinics, you have a couple of thousand of these in the community, they're able to start gathering epidemiological information about, oh, an hour after people go through this part of town, we see an uptake of attacks. We have a public health problem here that we need to start to address. So, as we start to think about these things, right now, we have this very tight focus, I think, often, how do we serve the individual who is carrying the device? What can we do to improve things at a community level for public health or for industry specific causes? How can we improve our society for just doing these sort of things? But all of these things passive interfaces, right? These services are the new crystal ball. If fortune-telling used to be about celestial phenomena and reading tea leaves and trails and it gets grosser from there. There might simply also, right now, we're turning to big data is being the source of this. The thing is about all of these is that they tend to all very automagical, they just do stuff on their own without any direct intervention or direction and this is very much the fascination of the moment, I would say, predictive interfaces, letting the algorithm do everything for you. Google loves this area, they're all in with it. From interaction perspective though, I think things get a lot more interesting when you just go ahead and make things straight up magical because that's where you're in control, the idea that technology extends our will instead of replacing it 'cause we leave a wake behind us as we go through the physical world, right? We leave a physical signals behind us that we've been here, this additive reaction. So, if our actions can be a source of data, they can also be a source of control, where the world reacts so this is the second way to think about physical interfaces that the world can respond to your actions, you're very direct the actions. This is what really feels like magic where the Harry Potter stuff comes in where it makes a physical gesture and there's a physical response. So, here in my automated home, I don't actually need the light bulb to be smart, it's actually more magic, it simply listens. So, an example of that. For centuries, the military have used sand tables, these little sandboxes as models of combat zones to get the lay of the land, to run simulations, and of course, with the digital world, these things have gone into the digital sphere, moved to digital representations which tend to be more flexible and more accurate than their sand counterparts but they lack the sense of physicality and scale. So the Ares Sand Table addresses that problem by equipping a plain old sandbox, sand table with a projector and a connect camera, so, you can project the landscape onto the sand or elevation lines as you see here and you can use the connect to read the topography, the shape of the sand so when you move the sand, the digital model updates and the projection updates. So, physical and digital are totally in sync. The sand is the controller for the digital model. And you can project your war games onto the thing or satellite images onto this little landscape, see how troops and vehicles there, I love that (mimics gun fire). See how troops and vehicles content with it. So, the idea here that, is that the world is literally responding to your physical actions, and the digital world is responding to it as well. The world is reactive or in other words, rather than passive interfaces, we're really talking about intentional interfaces. They don't just listen, they extend my will. So, this is an area where I'm tending to be working extensively lately. My clients tend to be big media companies and wearable companies, home automation companies, that need to create products and interactions that effortlessly integrate the physical and digital. A lot of these what this has been meeting lately is that we're instrumenting our own bodies often through wearables. So, we're carrying that technology around. It's just an armful of bracelets apparently, it's what we're headed there. But what that basically means is that we have to literally heave this technology from one space to another. So, forgive me, I'd like to introduce maybe an alternate term, really please do forgive me. This suggests, perhaps, thereables is a way to think about this which is where we have rooms or spaces that are smart or magic, so that the objects inside them can be dumb, that we don't have to carry stuff around. In other words, a third way to think about this is that the world itself becomes a canvas that is the smart place, a canvas for digital interaction. So, if you thought designing for mobile and desktop, it's pretty challenging. How about designing an interaction for an entire room or an entire city? Let's stick with the room. So, this is an idea that Jared Ficklin of Frog Design-- - Turn on the lights. - [Josh] In a project called Room-E. - Turn off this light. Turn off that light. Turn on these lights. (playful piano music) - [Josh] What he's showing is that it consider there's something in the way and moves the projection. - What sort of of take-out? What did I order last time? Order that. Show me the backyard. - [Josh] And again, even though there's a projection, it sees that there's an object there and moves the display-- - Hide the backyard. Goodbye. - So, the idea or thereables is that the room is smart enough to know what's up. So, instead of a bracelet in perfectly measuring your sleep, perhaps, your bed should do it. The trick is, of course, there's then a sort of like, all right, so now my bed is being watched, right? And who's watching it, and to what means? If there's an opportunity to make the environment smarter so that the individual objects inside of it can be dumber, then that means, you know, that there's some complicated stuff about instrumenting the world, and we're already seen some pretty screwed up stuff there. Right now, we're very much based on the web in this world of surveillance capitalism and you experience that in those little web ads that follow you around everywhere. So, what happens when we introduce that into our bedroom? Think that the thing, I think that there are genuine, real opportunities here, I think that it's a scary and nervous making thing, this idea of total surveillance. I'm beginning to think that total surveillance is inevitable and so, what's not inevitable is what we do with it. How do we start to take control of that as designers, get ahead of it, create transparency, find opportunities to watch the watchers or at least let us know when we're being watched and to what end. But I think that a lot of times, the first notion around this thing of making the kit, that the whole world, the digital canvas tends for whatever reason to go straight to commerce. You think about this with beacons for example, and everything that people ever talk about with beacons is somehow always tied to a retail store and giving people coupons when they walk into the store. You guys, it's gonna be great, when they walk by in a sidewalk, we can punch 'em in the face with a coupon, it's gonna be awesome. Actually, I've got a demo of what that looks like. (speaking over each other) - [Man] John Anderson, you can use a Guinness right about now. - [Man] You can use a Guinness right about now. - This kind of thing can work in a commercial context. Disney's magic bands, incredibly successful in the context of being in park and not knowing what you can do and just being able to quickly order things or get in line for a ride when you're on the other side of the park. It's a different story when your government asks you to do that, right? Or a big corporation. So, why do we always jump straight to commerce with these kinds of opportunities? Well, I'd like to share with you something that uses that same technology, a beacon technology, to do something rather different. That's a good notion for that, by the way, as we think about this. Wayfindr is an experiment in the London Underground to help the blind and visually disabled find their way in the tube. So, they've instrumented a station, several stations now, with Bluetooth beacons to track the users' location and offers directions. So, this young woman, has the app and she's got the headphones to go with it and she's gonna get ready now for a trial run in London's Pimlico Station. - [Man] Welcome to Pimlico Station. Follow the ramp down to the ticket hall. Turn left and walk down the stairs. There are nine steps. Bear right the walk forwards to the ticket barrier. You are approaching the barrier. You are approaching the end of the escalator. Walk forwards to the end of the hall. You are now at the platform. - This is the world that I wanna be part of. This is the world that I wanna help create. It's making meaningful differences to people in just their everyday lives. Why do we always think about how to give people a discount off of some cheap thing that they could buy at the department store as you pass by when we can do something like this? So, this is still very much in the prototype stages but the feedback that they've gotten in their surveys from the people who've tried this, it's really astonishing. Words like, "I felt empowered. "For the first time in my life, "I felt independence in the tube. "I felt like any other commuter". Real significant change in the daily experience of people's lives. So, one thing about this that I think is important is that you're actually moving through a physical space, a 3D space, so this brings up the last thing that I wanna talk about what a physical interface might be which is this idea that the world has depth and mass. You might have heard, the world isn't flat but we often think about the world as flat in our interfaces. These are two-dimensional screens and yet this isn't two-dimensional, all right? This is an actual, physical object. This is a physical object. They have a relationship together. As we design some of these interactions, how do we recognise and embrace the physicality of devices that carry these interfaces? What are some of the experiments that we might do there? Thaw is a project from MIT's Tangible Media Group that plays with that physical relationship. So, you can press your phone against the laptop screen to make them suddenly take each other and turn it into a fancy mouse. Like drag icons around the screen. Frankly, a mouse is probably better than this, right? So, probably this isn't that much of an improvement. It gets a little bit more interesting though when you go the other direction and you can start bringing that data down into the phone. There's your instant data transfer, you push them together and make that happen and it's done. But I think the best part is that Thaw starts to recognise your phone as having physical dimensions so you can insert this illusion of physicality into your digital experience. So, here what's they're showing in the scan contact so it's also saying, "You know what, "this thing is a block that has height and width to it", a box, as it says here. You can start to use that as part of the interaction between these things. What if you wanted to transfer content from one device to another and then transform it on that other device and then put it back into your laptop. So, this is a fun demo in a game context but you can think of tonnes of possibilities of being able to say, "You know what, "I wanna be able to do this to the photo "that's on here with the app here "and then put it back on to my phone", we're doing that all the time, we just constantly emailing crap to ourselves, right? Couldn't we recognise the physicality and proximity of these two devices? Right now, I've got this very expensive phone and a very expensive computer from the same company and when I put them next to each other, nothing, right? Nothing. Can't we do better than that? Well, this is an experiment. This is my buddy, Larry Legend, by the way. You guys, his real name is Larry Legend, love this guy. He and I worked on projects over the weekend. This is an experiment, some of the interactions that we might create. So, we wanted to explore this idea of making our phones and computers a little friendly together so we made this project called Happy Together, and he's always been in the studio listening to music on his headphones but he gets back to his desk and wants to keep listening to music at the same time there and this is what he does. Yeah. ("Happy Together" by The Turtles) (speech drowned by music) The music right over his, playing at the very same point and look how happy Larry is. It's my whole goal is make that guy that happy. So again, the idea is saying, these are physical devices, how can I create a physical interaction to move among them? And also crucially, to reimagine the data itself is having physicality. It's like water, or salt and pepper in a shaker that you're shaking from one device to another. You've got this working with URLs and maps and music as you saw here, photos, text, whatever's front and centre on your phone, you just shake it into your computer and you're done. So, it takes on this whole new cast when we start to associate information with physical objects. It's something, by the way, that we've been doing with touch, right, is reimagining data as flat, physical objects that we can move around on a screen. The opportunity now is to move that out into 3D space. So, as we think about this second act of combining the physical and the digital, we've got these ideas that we can gather data for insight, passive interfaces. We can channel intention with these reactive or intentional interfaces. We can use the whole world as just a giant canvas of interaction. And we can also start to think about interactions having mass if they aren't virtual anymore, that they have some physicality as we push objects and devices together. Though, I think the crucial question now though is how do we start to think about this? How do we start to imagine these new kinds of things? And how can magic itself be a source of influence. Well, that's what we're gonna talk about in this third and final act which, of course, naturally begins with Google Glass. (audience laughing) Google Glass obviously defunct and never really got out of the prototype stage. I think, it's a worthwhile experiment but obviously didn't succeed in its current, in its initial phase. I think there are probably tonnes of reasons for that but just from the outside. I think one of the challenges of it was it was just clearly always an engineering project, it's like, "Oh, we've got this camera "and this little tiny screen and a processor, "what if we put it on your face?" And that said, I think it would've been a much more interesting question if Google had started with the question of what if these glasses were magic? And just leads immediately to different places and what are associations with vision or the role even of our face and expression and interactions but it means so many stories of magic spectacles and spy glasses and things like that we could go on of really going back to that essential human need and want in association with the core thing, that we start to amplify what vision already is rather than try to disrupt it into something else. I think a better question is what if this thing was magic? If we start there, we go to different places. So, it's just a little thought experiment to start with something as simple as a coffee cup. If really don't wire this up for digital interaction and we first asked, what if this thing was magic? The implicit in that question are questions of context. It's not just what does that cup do lift liquid to my face and hopefully not burn me as I do it. I think it's got all of these context, what's the witness to, what actions is it adjacent to? It's the first thing that I see in the morning as I become myself, as I gained consciousness. It's the last thing I see on a day that I'm working on a late deadline. Or it's what's sitting between you and me as we share what's happening in our lives at a cafe or between a group of us as a meeting, as we're trying to figure out what to do and with all those context in mind, not just physical context but emotional context. You can start to ask all kinds of interesting things like what can I hear? Can I talk? Do I have siblings? Can I talk to them? When I do talk, is it soothing and quiet? Or is it annoying? Do I remember things? And you have all these different roles and context and then start to feed into how can I make this coffee cup more of a coffee cup? More of the emotional and contextual role that it already plays. It adds to a very different question than if you start with the components and you're saying, what happens if I slap sensors onto this thing. In other words, starting with this idea of just asking this thought experiment of what if this was magic, pulls you out into designing for the things, essential things. What is the thing that this does? More into not just with its function but its context because I don't want a smart home. I don't want home automation, those are not actual goals. Home that is calm and in harmony and in an oasis is what I want. A place where I can connect with friends and family, a sanctuary. So, if we can light things up to do that and help that, that's great, a means to an end that is larger than the simple automation. So, perhaps by understanding the digital environment that these places or objects are in, they can help to make them more of that. But the idea's that we should bend technology to our lives, not the reverse, make things more like they are, make us more human. That's that crocks of magic. And Harry Potter, Hermione's beaded bag, it's magic because it's bigger on the inside, it holds an impossible quantity, it is more of a bag than it was before, enhancing its essential role. I'd a good fortune to visit the Burj Khalifa in Dubai, tallest building in the world for the moment. Like a lot of tall buildings, it has these viewfinders, binoculars for enjoying the view, but the Burj Khalifa, you can hit a little button and see what it looks like at night, it changes that, or you can look at what it looks like during the day or you can hit another button and it shows you what it looked like 50 years ago before the city was even there. This magic lens for seeing around you, embracing the essential role of the thing but making it more of that. So, the idea here is really that we're kind of, again, as I mentioned earlier, banking on illusion and misdirection and I don't mean that about trying to fool people but instead trying to create a mental model that fits the way that they see the world. I don't wanna oversimplify this, this can go terribly, all right, turns out someone did this exercise and brought a magic cup to market and this is what they came up with. (liquid pouring) So, this is a product called Vessyl. The idea is you pour something into it, it takes a moment and then it's like, "I know what that is, that is juice". So, for example, this guy naturally forgot, just got his Vessyl, and it's like, "I've got some beer, I'm gonna pour it in "and see what happens", and he's like, "Yes, that's beer". You think it's kind of remarkable, and it's helpful, it can track how hydrated you are, if you're interested in cutting back on coffee, it'll tell you how much caffeine you've had for the day. And also, if it doesn't recognise the thing that you've put in, it can still those some analysis to understand what its nutrient levels are. So first, I wanna say, amazing technology. I mean, that is crazy that something that, it's pricey for a cup, let's face it but the idea that a consumer level product could do this on-the-fly analysis, stuff that actually comes out from quality assurance from beverage companies, that is amazing technology. I know, Stephen Colbert, the comedian was impressed. - Wait, (audience laughing) a digital cup that can tell me what's in the cup and how many calories and allow me to drink it? That level of information was previously available only on the can, you just poured it out of. I mean, there are so many. Think about it, there are so many times when Vessyl's beverage identifying technology will come in handy like when you order a Coke but it takes kind of like a Diet Coke but you're not sure and, perhaps, other times. - Perhaps other times. So, I mean, we see this a lot, right? But we're just like, but we can gather data, let's just see but we don't add enough insight. Because, you guys, connected does not mean smart, and smart does not mean good. For this stuff to actually be smart and good, we need to actually provide a real human need. I think that's a big problem right now for all these data analysis, sites and products. Do all of those Fitbit steps and extra data actually give you actionable information to make your life better? Or we're just getting this depressing stream of noise? (audience laughing) Always ask how can we improve the data signal? What kind of remarkable insight can we offer? While collecting and reporting data is on its own a pretty neat trick. That shouldn't be the goal and in it of itself. When you dive into the data, try to come back with something even better. (audience laughing) It's not about the data, it's about the genuine human insight we can provide or put it another way, talking is not the goal. The goal should be better conversation. Today, a billion new devices are rising on the network. Let's design 'em to speak only when there's something useful to say, to reduce the chatter in favour of joy or insight or real utility. I don't wanna make that seem easier than it is. Sometimes, it's hard to distinguish the useful gadget from the bad idea, and well, we'll play with some goofy ideas until we get it right. It's okay to play in our practise and then our research but I think, we just with to be a little bit more cautious about what we actually release as a product 'cause this is an essential truth, not just a technology but magic 'cause we know that magic always comes back to bite you somehow. Technology always lets us down. So, the fear of losing agency to unreliable technology is definitely with us as we build it more and more into the fabric of our lives. That's a real thing. Well, we saw this just a couple of weeks ago when the whole internet came down because of a botnet attack using DVR machines and little Internet of Things devices because we have this whole class of the insecure devices that security, and to some extent, privacy and it's not a value that was cooked into this. And again, you know what happens when we start to have corporations in our bedrooms. There's something creepy, even sinister about the devices of big companies listening in everything you do at home. The smart home network, Wink, came out with an ad that plays on that. - Honey, we need to talk about Robot Butler. Look, I love the way to controls the lights and unlocks the door when I forget my keys. It's just that, I feel like-- - [Both] He's always watching us. - Yes, that's why we should just use Wink. Look, it can monitor and manage our house, but it won't develop human emotions. (buzzing) - Hey buddy. (audience laughing) - I mean, we all like smart technology but when it gets too smart and when it invades the most intimate spaces of our lives, our homes, our bedrooms, our bodies, it can feel like too much. At what point do you stop being in control or does the technology make decisions that you don't agree with. If we get like not agreeing with you, what if it just attacks you, actual headline here. This Roomba tried to eat her. Human being should always have access to the off switch. No, let me use a mechanical light switch if I need it. My hand on the door knob should override the Smart Lock. In other words, this isn't a real challenge is that we need to be a little bit more sophisticated about building systems that are smart enough to know that they're not smart enough, that we can't always rely on an algorithm 'cause we don't always get this stuff right and we feel that with the technology that we use everyday. So, in a car, you might want to design a UI that's hands free and eyes fee, and this how that works in practise. - Call Gerardo. - [Woman] Name unknown. - Call Gerardo. - [Woman] Name unknown. - Call Gerardo. - [Woman] Calling Gerardo. - [Josh] Yeah, (audience laughing) right? - Gerardo. - This is a project called Curious Rituals by Nicolas Nova of the Near Future Lab and it's expected that's about what everyday life might really look like if we embed that fancy tech everywhere, like letting you hang up a call with a gesture. - Oh. - [Woman] Call ended. - Call Gerardo. - Calling Gerardo. - Yeah. And you see this a lot right, contorting ourselves or often our language to the expectations of the technology where again, we bend to the technology instead of the reverse. That is not what Kay was talking about when he said that technology should be invisible. So, I mean, this is the natural area of mundane technology, this kind of work works well enough for us to use it but it meets us halfway. So, as we try to make technology that acts like us, the risk is that we just act more robotic. (audience laughing) I'm a big fan of technology that empowers us to do more and to be more human. Building stuff out of a non-bridled enthusiasm for technology, we've all been there but it can isolate us and damage social connection, make us feel trapped. So, you guys, there's a lot at stake right now, and because we're just figuring out how this will go, and it's uncertain and there are a million ways that could go wrong because again, this is all in the context of what again, what I consider to be total surveillance, inevitable total surveillance. How shall we design to that? Because what's not inevitable, as I said earlier, is what we do with it so I would like to suggest that we focus relentlessly on technology that amplifies our humanity, that bends technology to our lives instead of the reverse, makes us more of who we are, designed for human connection, encourage interaction with our surroundings, design for our essential selfness, as well as the essential thingness of the things around us 'cause, you guys, the magic is not about the thing or about the data or about the technology, it's not Harry Potter's wand that's magic, it's Harry, the same goes for the Internet of Things, it's about this partnership of technology and people where technology supports us, not the opposite. Sceptics fear the robots taking over and optimists suggests the algorithm can take care of everything and it's neither, of course. It's human beings in partnership with the things, with the data, that's what we have to imagine. So, just to close out what we're talking about, it's starting with this question of what if this was magic, yields very different results and I use it in my own practise and with my own clients now, yields very different results than if you start with the technology. If you add insight and not just data, that we have to do that to deliver a real human need, honour the intention of what someone is doing and don't assume it, make systems that are smart enough to know they're not smart enough, and again, let's make this thing humane, let's not make it jittery, let's create technologies that help us be more mindful and aware 'cause, you guys, this is not a challenge of technology, I don't wanna make this, I don't wanna oversimplify this stuff, but we're awash in technologies. That example that Aral showed us of just using regular everyday technologies to do pretty remarkable exchange of data in the living room. You guys, it's a challenge of imagination. How might we deploy this stuff in new and meaningful ways that create ease and efficiency, yes, but also joy and real human emotion, real human value. It's an amazing time to make things, you guys, please do that, make something amazing, take the difficult global times that we do and say, "You know what, screw that, "we're gonna make something positive". Thank you. (audience applauding)