Magical UX and the Internet of Things

“What if this thing was magic?” Designing for the internet of things means blessing everyday objects, places, even people with extraordinary abilities—requiring designers, too, to break with the ordinary. Designing for this new medium is less a challenge of technology than imagination. Sharing a rich set of examples, designer and author Josh Clark explores the new experiences that are possible when ANYTHING can be an interface. The digital manipulation of physical objects (and vice versa) effectively turns all of us into wizards. Sling content between devices, bring objects to life from a distance, weave “spells” by combining speech and gesture. But magic doesn’t have to be otherworldly; the UX of connected devices should build on the natural physical interactions we have everyday with the world around us. The new UX must bend technology to the way we live our lives, not the reverse.

Josh “won’t lie…” – the results of the US election make it very hard to get up and talk positively. It’s a context of global tragedy after millions have chosen hate over hope.

But there is some hope in treating this as a challenge, even a design challenge. The projects we choose matter, the things we work on can make a difference to peoples’ lives.

We have an opportunity to build amazing new things with emerging technology around physical devices, linking the digital and physical. But let’s show a real example!

shows clips of Harry Potter characters using magic wands

“But then I realised you can buy these on the internet…”

pulls out a universal remote made in the shape of a wand, and turns lights on and off with it

We have objects that are collapsing intent into action. A wand collapses the technology of an infrared remote into a smaller, reshaped device.

(Grab Magic video – where the “grab” gesture at a TV screen + Kinect lets you put screenshots onto a phone)

Combining things we already have in the home can create new interactions. They can have lightness and humility, some fun, some humanity.

Three acts

  1. magic & technology
  2. physical meets digital
  3. magic, imagined

Act I: Magic & Technology

Any sufficiently advanced technology is indistinguishable from magic. – Arthur C Clarke

We use the terms of magic so loosely now, everything is going to “delight”… but delight deprecates.

One goal: the computer disappears into the environment. – Alan Kay

Technology should get out of your way, the same way a pencil gets out of your way.

Fantasy fulfills a need for a simpler, more controllable world. – Alan Kay

Very successful interfaces are often about creating a new and simplified model that we feel really confident using.

We all have our own magic wands: mobile phones are the first magic wand for everyone. Phones bridge the digital and physical worlds. They are the first IoT device that really went huge. Everyday objects with sensors, processor and internet connection (sensors + smarts + connectivity).

They provide interaction right at the moment of inspiration. You can use phones to look things up and interact with whatever is right in front of you. Mobiles bring computing power to immobile objects – things that were traditionally ‘dumb’.

Example of Huggies Nappy Notifier – just a novelty really, but if something as disposable as a nappy can be made ‘smarter’ then there really is a lot of possibility out there.

Most existing products and ideas are still mediated via screens. Everything pulls us back to the mobile phone screen; and research suggests the average phone user spends as much as 20% of their waking hours staring at a little screen.

So the availability at point of inspiration is the phone’s Achilles Heel as well as its great strength. So how can we push the interactions right to the point of inspiration rather than onto a phone while you’re there?

(Example of TryOn ‘mirror’ in a store, which helps you try and compare outfits in a more natural way.) “This technology was previous only available to evil step mothers… we should raid the castle for ideas!”

Book: Enchanted Objects by David Rose

Idea from magic and fantasy Real world
magic brooms Roomba
Dorothy’s shoes ‘Ruby’ the escape device
magic button that does things for you on demand BTTN, Amazon Dash

The Amazon Dash is not really the future we want – we don’t want to fill our lives with brands. But it does explore an idea in an amazing way. They released AWS ReInvent, a generic and hackable version of Dash so makers could do what they want.

The idea of making computers invisible is to make them a natural part of our lives.

ACT II: Physical meets digital

The world is the interface – as it has always been. IoT shouldn’t be about novelties.

The world is a data source

  • Progressive Insurance ‘Snapshot’ device measures what the car senses to give the company insights into the customer’s habits.
  • ‘Automatic’ uses similar tech in a more customer-focused way – it coaches the driver to be an efficient driver; and it translates warnings into instructions to make it easier to deal with the car.
  • Propellor Health adds a device to asthma inhalers to track usage and help educate the user on how they manage their asthma.

Automagical – Interfaces that anticipate things we need to do or know; or track actions that give us insight later.

The military uses sandboxes to run scenarios; and digital versions have never had that physicality. So there are devices now that read the shape of a sandbox and project information over that.

Wearables -> maybe we should think about them as “Thereables”, where the place we are in does the sensing, rather than us wearing an armful of bracelets. So how would it look to design interactions for an entire room? Example video of a room that’s aware of the user and uses voice and gesture to handle lights and project information.

We’re currently in an era of surveillance capitalism, where ads follow us around… so why would people want to put that into our bedrooms? How do we take control of this as designers? If total surveillance is inevitable, how can we make that useful?

Why do we always jump straight to commerce? The world is a big canvas: don’t be a jerk!

Wayfindr is an app integrated with sensors that helps vision impaired people use the London Tube.

“This is the world I want to be part of, this is what I want to create!”

The world has depth and mass, but we tend to think of it as flat, two-dimensional.

THAW – brings computer and mobile together. Presented in a game context, but the jump between devices brings up all kinds of new interactions. “We’re constantly emailing crap to ourselves…” why don’t these expensive devices talk to each other more efficiently?

(video: Happy Together, where you tap your phone on your computer and it takes over playing music.)

Act III: magic, imagined

Of course we have to start with Google Glass… shows picture of Mad Eye Mooney

Instead of starting with tech specs and “let’s put it all on your face!”, what if Google Glass had started with the question “how can this device be magical?”

What would a magical coffee cup be like? Where is it in my day, how does it fit into my social interactions? How can a coffee cup play more of an emotional and useful role than it already does?

Josh doesn’t want home automation… that’s not a real goal. He wants a home that is calm, a sanctuary, a safe place.

Make the thing more of itself, “make the thing MORE the thing”. Hermione’s magic bag holds far more than a normal bag – it is more the bag than a normal bag.

Example: Vessyl – ‘smart cup’ that tracks hydration and so on. It’s amazing tech, but as Colbert points out all the information it can tell you is printed on the can you just poured the drink out of.

So… don’t just add data, add insight. Add magic. Talking is not the goal – the goal should be better conversation.

We should also remember that magic always goes wrong. We just saw a botnet attack that used DVRs and security cameras. They did not have security and privacy sufficiently baked into them to protect the users.

Wink’s ads had fun with the idea of people being scared of their robot butler and preferring an app which “won’t develop human emotions”.

We need to build systems that are smart enough to know they are not smart enough. We should not bend to the technology – shouting weird pronunciations at voice diallers on phones… the machines should bend to us.

The magic is not about the thing – Harry’s wand isn’t magical, Harry is. IoT is not about the Thing, it’s about humans.

Technology should amplify our humanity.

It’s not a challenge of technology, it’s a challenge of imagination.