Fun with Sensors and Browser APIs for the web!

Developments in Sensor & Browser Apis has opened up many opportunities for your website to react and respond based on the users device, environment or input. As of right now there are many great Sensor & Browser APIs that you can take advantage of including Light Sensors, Gyroscopes and Audio and because these APIs are all available in more devices than you would think they are more accessible than ever.

In this presentation we’ll look at some of my favourite features and examples, like how we can improve the visual representation of our conversational UIs using web audio and speech apis, or how we can change the user interface based on the ambient light detected in the users environment. We’ll explore the the possibilities, tools and resources so you can get started creating engaging and creative effects, visualisations and experiences in your projects tomorrow.

We start with an important introduction, to Mandy’s dog Jello! So when you see the word ‘jello’ later, it’s a pooch not a pudding.

Preface to the technical: this a session called FUN with sensors and APIs, not practical production applications! There are practical applications but they’re not as fun to demo. Remember that everything starts with experimental implementations; and also keep an eye on the logos for browser support (not going to spend all the time talking about that).

(code samples in codepen, not attempting to transcribe them)

Web Speech API

Two interfaces:

  • Speech recognition (input)
  • Speech synthesis (output)

Focusing on speech recognition today.

The first demo is a simple capture of speech to rendered text – noting this does need an internet connection to phone home to a speech service. Interesting thing is you can see the words changing as it gathers more input and gained confidence in its context. You can also choose to wait and render in one go, but it’s not as illustrative.

Notes:

  • There are methods for starting and ending input, there are events for getting a result, and for when speech input has stopped.
  • The result is an object; and the actual results are a little convoluted to get to as the API is designed to bring back multiple alternative results.
  • You must trigger this from user input, which is a security consideration.

Demo: fire-breathing dragon, with bonus variable font because Mandy has to include one by law at this point. Interesting sidenote that the result comes with a confidence rating.

Demo: using a variable font to render text differently according to the volume and inflection of the input speech. It’s a bit of a failed experiment as there’s no timestamp in play to match things up to that level of detail. The sad thing is current speech to text tends to lose the richness of the input, it just turns things into strings. However you can detect certain words like ‘love’ and ‘Jello’ and then style those differently. In future hopefully we will get a much richer API.

Demo: a dog that listens to commands and responds to them. Demo ran into the classic problem that Australian accents are not supported well! But the virtual dog was able to respond to commands after a couple of tries… quite realistic when you think about it 😉 There are some really practical reasons to want to accept basic commands – Netflix should accept verbal yes/no prompts, people often don’t have their hands free when an alarm or something goes off in the house.

The way the Web Speech API works at the moment:
Browser API → Speech Proxy → Speech To Text provider (hosted by the browser vendor)

Google does keep the recordings to improve the service; currently Firefox uses Google’s service but pays to have them not keep the recording. Or you can choose to use Mozilla’s Deep Speech service, which doesn’t work as well yet (in Mozilla’s own judgement). Mozilla is working to improve Deep Speech and also looking into options to download language packs, so you can use the API offline. It’s a feature they are keen to provide.

Audio and speech have lots of interesting possibilities for providing useful and accessible interfaces.

Orientation sensors

The accelerometer detects a change in the orientation of the device. This does differ depending on the device’s specific hardware.

Demos: text and animation effects responding to the position of the phone…

  • text slides to the bottom of the screen
  • sliding colour flip for text – would have been great in some agency/brand work
  • changing font weight and zoom as the phone tilts – would be great for a conference website

In more practical applications you’d use this to trigger subtle UI effects to reinforce interactions; or to add an interactive element to storytelling experiences. A lot of our current content is very static, and these APIs could bring some dynamic changes to it.

Light sensors

Ambient light sensor does what it sounds like – reads the amount of light the device is currently reading. There are two versions of the API, the first being deprecated, so be sure you get the right one if you google it.

There have been legitimate security concerns about these, because the specific implementation made it possible to steal data. The main fix is to add a frequency cap and round off the values being returned – this gave security but has not practically diminished the usefulness. Since the security concerns have been addressed, Chrome do plan to release this as a mainstream feature quite soon.

AmbientLightSensor creates an object; has methods for onreading and illuminance (current light level). That’s really all you need.

Demo: a page that shows a sun or moon depending on a light threshold. Quite fun for making different website designs.

Demo: changing font weight based on light conditions – this would make a real difference to readiability in those scenarios.

Demo: variable font with vines of flowers that bloom or close depending on the ambient light.

Don’t be limited by the things we can already do. The web is still young and there is so much for us to create. Make interesting, useful, cool and fun things!

Combo demo… Smello the wizard who will only do magic in the dark; and the UFO + Caticorn game.

“That went really well, so I’m gonna stop there!” 😀

There are lots of way to combine these interfaces and inputs, to create cool things. But in the meantime, hopefully you enjoyed abducting a caticorn with a UFO!

@mandy_kerr | Mandy’s slides and resources | Mandy’s codepens