Designing User Experiences with Eye Tracking

(cheerful music) (applauding) - Guess you can hear me.

I'd like to start by saying eye tracking is not bullshit. But that is the topic of today.

So today we're gonna talk about how to use your eyes to interact with digital systems.

Not too long ago, this is what an eye tracker used to look like.

The first eye trackers were so awkward to use, that to use this thing you actually had to inject some cocaine in your eye, I'm not kidding.

Good old days of research.

That one isn't even that invasive.

'Cause it's just camera-based.

But this one even required you to wear a contact lens with a wire coming out, really cringey stuff. Looking more like a mediaeval torture device, or something out of A Clockwork Orange, than something they're actually want to bring into your home, or use in your everyday life. We've come a long way since then.

Nowadays, eye trackers are, we can barely see it there in the background, but modern eye trackers are easy to use, they're inconspicuous, they're cheap.

They're much more robust, much more precise. Cost only a couple of hundred bucks.

Some of them even look like your regular pair of Ray-Bans. So this stuff is getting to a consumer-grade level. In popular culture, eye tracking is often portrayed as part of these dystopian futures, like in Minority Report, as a way for the government to track what you're doing. Or as in Black Mirror, to force you to watch ads, and so on. But I personally believe that eye tracking is a very exciting piece of technology with a huge potential.

To actually change how we interact with computer systems by using attention, visual attention, as a design material.

So, I'm gonna discuss some of the strategies that we've been using in our research to develop systems imagining a future where eye tracking could ship by default with interactive systems like in this MSI laptop that comes with an embedded eye tracker.

So those in the audience that work with UX might have used an eye tracker before, for usability evaluation.

And indeed, the jury's still out on whether this is actually useful.

But this not what I'm gonna talk about today. You might also be familiar with applications for eye tracking as an assistive interface, to replace manual input for disabled users. This is a very important part of research, and there's been some very good work done in this topic. But this is also not what I'm gonna talk about today. Rather than talking about how the eyes can replace the hands, I wanna discuss ways in which the eyes can be used together with the hand.

To support manual interaction.

So my name's Eduardo Velloso, I am a lecturer in Human-Computer Interaction at The University of Melbourne.

And today's talk is called Designing User Experiences with Eye Tracking.

So this is work that I've been doing over the past, roughly five years.

Started in the UK at Lancaster University, where I was based at before.

And then I came here to Melbourne Uni, where I'm continuing some of this work.

So the main message I want to convey today is that the eyes work in a substantially different way to our hands.

So designing gaze-based interactions actually requires a different way of thinking about interaction design when you are designing for manual input.

Gaze offers many opportunities for interaction design. Your eyes make some of the fastest movements your body is able to make.

And they're inherently tied to action, intention, perception.

They're always there with you, they're part of every action that involves visual attention, and they also play an important role as a nonverbal behaviour, which is still underexplored in interaction design.

On the other hand, it also suffers from many challenges that hinder its adoption as a mainstream input device.

First of all, eye tracking can be very inaccurate, can be very imprecise.

You can't get that pixel-perfect accuracy that you wish. Also, it requires these tedious calibration procedures to make it work, so that you can know where the user's looking at.

And finally, it plays multiple roles in a context of gaze interaction.

'Cause you're using it as an input device for the system, but as an input device for yourself, right? As a sensor, as you take in information from the world. So let's talk about these challenges.

So let's start with the inaccuracies.

So with a mouse you can select any point on the screen with pixel-perfect precision.

But not with your eyes.

With your eyes, the precision's more like a region around the gaze point.

And this is partly because the inaccuracies of the hardware, the software, the user moving around, the calibration. But also because of the inherent nature of human vision. So this picture kind of illustrates roughly how our vision works.

So we have this small region around the centre of the gaze direction called the fovea.

And in the periphery we have this region that is responsible for night vision and motion detection, that actually sees things very blurry and in black and white.

And your brain fills in the gaps.

Even though you don't perceive the world in this way, our brain fills in the gaps.

So, for you to see how much our brain actually fills in the gaps, you can do this test.

So extend your hands like this.

Close your left eye and look at your left index finger. And wiggle your other finger.

And see how it disappears.

So we all have these blind spots in our retinas that our brain are constantly filling in.

The size of our foveal vision, meaning the area where we can see in high resolution, is roughly the size of a thumb at arm's length. Which means that, even if we had perfect eye tracking, we'd still have a region that the user's seeing that is still not a point.

And this region only increases with the distance. (sighing) All right, this looks rubbish.

If the eyes are inherently inaccurate, what do we do about it? Well, instead of trying to make your eyes control a cursor, how about use the eyes to help the hand and maintain the precision of manual input. And combine that with the speed and versatility of the eye movement.

So an example of this is a project from a colleague, Ken Pfeuffer, who built a system called Gaze-touch. And the basic idea here is to relay touch gestures to wherever the user is looking.

So it's as if your touch gestures would teleport to the area on the screen that you're looking. So this is how the system works.

In grey, you can see where the user's looking at. And you see the touchpoints being relayed to that area. And this means that you can make a distinction from a touchpoint made when you are looking at that point versus a touchpoint made when you are not looking at that point.

So here's how we can use this.

So in this application, if you are looking at where you're touching, the touch draws.

But if you are not, the touch makes a selection. See, making a selection, and drawing.

If you add pen input in the mix, you can even expand your repertoire of interactions. So now we have this direct pen input, when you are looking at the pen.

You have this indirect pen input when you're not looking at the pen.

You have direct touch when you're looking at the finger, you have indirect touch when you're not.

Here's how we can use this.

So, the same example as before.

You have this really precise input with a pen, and really quick selection with the eyes.

You can also modulate the effect that the pen has based on whether you are looking at it or not. Like in this case, where if you are looking at where you're drawing, you're dropping anchors when you're tracing the hand.

But if you put the pen away, it changes the curvature of the segment.

So using visual attention to modulate the behaviour of your manual input.

Keep in mind that the cool thing about this is that your attention is always where it matters. Let's talk about the second challenge, calibration. Who's sat through one of these? Either as an experimenter or as a participant? A few of you.

So you know how boring this can be.

So you sit your participant there, you make them look at these points in sequence, and then the system will tell you, Yeah, that was a good calibration, or Nay, that was a terrible calibration.

Do it all over again.

This can be very boring.

If you're trying to, now, let's consider this scenario.

You have this interactive public display.

You want to enable some kind of hands-free interaction on the display, so you thought, all right, let's use our eyes.

Of course you can't expect that people who'll approach a display will calibrate the eye tracker for them to use, right? So how do we solve this? Until someone comes up with a way of tracking precisely where you're looking on the screen without a calibration, if we need the absolute coordinates on the screen where the user is looking, you need to calibrate. There's no other way.

So instead of trying to make calibration more fun, or making calibration more faster, we decided to start looking at things that don't require calibration.

And our insight was that if all you need is relative movement, you don't need any calibration.

Most of the time our eyes are switching between these steady fixations and these quick jumps called saccades.

Except when we're following a moving object. Then our eyes make these really smooth movement that follow the trajectory of your target.

Try to make a circle with your eyes.

It's impossible.

The best you can do is a bunch of segments around that circle.

But if you have a target moving around that circle, all the sudden you can.

So, if our second challenge is calibration, our solution is, let's use movement.

And all the sudden we don't need calibration anymore. So how we can use this.

So, going back to the public display scenario. The idea is to show a bunch of moving targets on the screen. And then each target's moving differently.

And then you check how the eyes of the user are moving and compare the movement of the eyes and the movement of the target.

And then you can figure out which one the user's seeing. So here's the prototype they implemented.

So this is something for a city display.

So you have a bunch of targets moving.

In this case, you have a little button there in the back, saying Back that is moving horizontally.

This is a display for a music shop.

So you have a bunch of album covers spinning around. And if you follow one of them, the music starts playing. The cool thing about this, is that because you no longer precise coordinates, this technique not only works with large displays, but even with tiny displays and smartwatches. So here's how we implemented this technique for smartwatches.

So you got a missed call.

The target starts spinning and you can decide what to do with that call, just by following the corresponding moving target. You don't need to recalibrate your eye tracker every time that you raise your hand.

We built another prototype where app notifications would pop up like this.

And you can use the speed of the targets and the radius of the circle to encode additional information to that.

So designing for movement's completely new matter for us for interface design. Finally, I wanted to talk about the multiple roles that our eyes play in gaze interaction.

So what I'm proposing here is this type of direction, right? So you're using the eyes to control the interface. But our eyes are already busy taking in visual information. So if you're forcing your eyes to control something on the interface, you gotta balance these two roles that the eyes are playing. You're probably familiar with the story of the King Midas, who would touch things and they would turn to gold. And everything he touch turned to gold, and hero went out of control.

A similar thing happens if everything in your interface becomes gaze-activated.

As you visually observe the interface, you start activating a bunch of stuff unintentionally. So we want to minimise that.

And there are many ways out there where you can minimise this.

You can use a longer threshold until you trigger actions. You can use a supporting modality, like the hands, or some other input device.

But this is not what I'm talkin' about.

What I'm gonna propose here is to embrace this double role, and use this to create playful user experiences. So a good example of this is this game built by my friend Melodie Vidal, in which the player must drag these little creatures into that pen.

And while the player's doing that, these dinosaurs are trying to steal away the little creatures.

But if you look at the dinosaur, the dinosaur gets shy and runs away.

So you gotta balance the visual observation of clicking and dragging the little creatures with the explicit gaze-based interaction to scare away the dinosaurs.

So, balancing the eyes as a sensor for your body and as an actuator on the interface.

But our eyes do not only play a role of control and observation.

Gaze also plays an important role in our social interactions.

It tells a lot about how we interact with human beings. In terms of attraction, in terms of hierarchy, dominance. Just compare the differences between the shy glances of a couple on a first date, to the deep staring of a couple in love, to the distant looks after a fight.

And a friend of mine built this game that used gaze as a social game mechanic.

So in this game you wanted to become the trainer of the Royal Corgi.

So you had to convince the king's court that you were the right person for the job. And each character in the court would react to your gaze behaviour in a different way.

So you had one character that had his wife next to him. And if you started looking at the wife too much, he would get pissed off.

You had an artist who had his painting behind him, and another painting on the other side.

And if you looked at the other painting for too long, he would get pissed off at you.

And I'm gonna show you a clip of a character that demanded your full attention, so you always had to look at his face.

If you started looking away, he would get pissed off. And we got a very famous actor to do the voice acting for this.

Okay, see if you can identify who it is.

- [Soldier] Good morning.

The closest soldier to the king is his most loyal and last defender.

He should be fearless, ferocious, and fierce. A true soldier that should be taught by a cunning royal mas--.

(grunting) Am I no interesting enough for you? You lack discipline, and respect.

- So halfway through the user looked away.

Anybody detected who it is? It was me.

(audience laughing) All right.

Finally, I wanted to discuss how simply the awareness of someone else's gaze can change social relationships. So I have now a PhD student who is looking at how multiplayer games change when you start seeing where your opponent is looking.

So we are using the digital version of the game Ticket to Ride.

Anybody play Ticket to Ride? And the idea there is they have to build these routes across a map.

And when you start seeing someone else's gaze being displayed on the screen, on your screen, you can start figuring out what are their strategies. But the cool thing about this is what happens when the opponent realises that you can see their gaze. Because then they start using gaze to try to deceive you, and deliberately throw you off.

So you start adding this extra layer on top of conventional gaming that we have, and when we play the board game version of this, right? So you can see where your opponents are looking. But when we go to our digital version of the game, we lose this nonverbal communication that is very rich. So at the moment I am looking at formalising these concepts. So I'm writing a book with my colleague Hans Gellersen from Lancaster, in which we're formalising some of the principles for designing gaze-based interactions.

So soon enough, you'll be able to hear more about them. So the takeaways of today's talk is designing for gaze requires new ways of thinking about input.

The eyes are a complex organ.

So you really need to rethink how you build interfaces. I want to plant the seed in your mind of using gaze not only as a replacement for the hands, but also the possibilities for using it in a supporting role.

You can use movement to avoid calibration, and enabling these walk-up-and-use types of interactions. And because visual attention is challenging, and plays multiple roles, you can embrace these rather than trying to minimise them, and create these playful interactions.

So now, do get in touch.

I have a nice report on a bunch of game mechanics and how you can use this for games.

So if you're a game designer, or work with games, do get touch to see some of the stuff.

You know, I do a lot of stuff with eye tracking and training.

And if you're interested in PhD and Masters, and working with these things, do get in touch. Thank you very much.

(audience clapping) (cheerful music)