Machine learning for front-end developers

Machine learning can have some pretty complicated concepts to grasp if you’re not a data scientist. However, recent developments in tooling make it more and more accessible for developers and people with little or no experience.

One of these advancements is the ability to now train and run machine learning algorithms and models in the browser, opening this world to front-end developers to learn and experiment.

In this presentation, we will talk about the different applications, possibilities, tools and resources, as well as show a few examples and demos, so you can get started building your own experiments using machine learning in JavaScript.

Charlie Gerard

Charlie points out she is not an expert in ML, but to remember that if she can learn enough to demo just tinkering in her own time… you can learn it too!

What is ML? Using statistical techniques to train code. For example a spam filter in the old style would be an epic IF statement; but a ML solution takes a model trained with example data and tests new data against it.

Types of algorithms:

  • Supervised learning
  • Unsuperised learning
  • Reinforcement learning

Supervised: create a predictive model based on a set of features and labels (inputs and outputs).

  • Labels – the name of how you would classify each entry (‘dog’, ‘cat’)
  • Features – characteristics of the entries in your datast (‘dogs are more friendly’, ‘cats have whiskers’)
  • Predictive model – mathematical representation of of the outcome of your training (how would you determine if the new entry is a dog or a cat?)

Basically it’s maths trying to understand the data you are giving it.

Example: predicting the price of a house. You can feed data like size, features, location into the model.

Classification vs regression – classification is dog vs cat, regression is a continuous response like the price of a house.

Unsupervised learning just uses characteristics.

(I was interrupted for a moment here so the notes are incomplete, sorry!)

Neural network – Charlie provided a link to a video, it’s a topic to itself.

Choosing the right algorithm:

  • What are you trying to do?
  • What is your dataset made of?

ML is more of an art than a science. You will really never stop at your first prediction, you will need to test and refine… it could be your data, it could be your params, it could be the wrong algorithm.

Applications of ML

  • Home automation – example of a system that watches a person closing their book; and turns out the light when they do
  • Art – we recently had an artwork created by a neural network sold for $400k, using code the sellers didn’t write… and it challenged the definition of art!
  • OCR – there are datasets available online. This is often how people get started, because you can be so certain of the results.
  • Accessibility
    • if speech interfaces are the future, what about people who can’t speak? People have created solutions to have devices like Alexa understand sign language.
    • Sarah Drasner has created a cool demo of a system to autogenerate ALT text.

So why do all this in JavaScript? Data scientists will object that there are better languages!

  • easier learning curve makes it more accessible to more people
  • big ecosystem, lots of modules you can use
  • why not?!

What you can do in the browser:

  • Import an existing pre-trained model
  • Retrain an imported model (transfer learning)
  • Define, train and run models entirely in browser

Tools – tensorflow.js, keras.js, ML5, MagentaJS (music)…

Demos!

The Willy Detector
Charlie had to create something that allowed people to draw on a canvas, which meant she needed to stop people drawing willies. She had to train it with hand-drawn willies, and also hand-drawn things that were not willies (from Google Quickdraw doodles). The model only has 40 examples of willies so it’s not very accurate. Then the model needed to be trained at different sizes; and you need 80% for training and 20% of your dataset for testing. Process – CNN useing Keras… training, then paramater tuning, prediction. The code in JS is surprisingly short.

Teachable Keyboard
Charlie was able to quickly train the camera to recognise left, right and enter based on the position of her head in the field of view; which meant she could type on a keyboard just by moving her head.
This one works by importing a pre-trained module (mobilenet), uses KNN classifer algorithm, define the number of classes (how many gestures), feed the examples into the module, then you can run predict which returns left/right/enter/neutral.

Face Pong
(playing pong by jumping up and down in front of the webcam) Taken directly from ML5 example using mobilenet.

Limits

  • you need a lot of data, unless you have a big pre-trained model
  • takes a lot of time to train your own model
  • think about the mobile experience – this code will chug on lower-powered processors
  • liability – some models are black boxes, you don’t really know what’s happening and how they work. So if you get sued over it you won’t even know how or why it did what it did.
  • bias and ethics – algorithms aren’t biased, we are. If you feed sexist data into an algorithm, you’ll get sexist results. A simple example is a translation tool which assigned gender-stereotyped jobs to men and women, and negative emotions to women rather than men.

How to get started

  • Start with ML5.js as they explain things so well
  • Start small
  • Follow a basic example – MNIST handwritten character recognition
  • try a few different classification algorithms
  • try to replicate a project – doesn’t matter if it has been done before, just do it to learn

Tips

  • Avoid training your model in the browser unless you have a really critical reason to do so
  • Retrain your model regularly (or your predictions will become wrong)
  • Be careful of overfitting (this is when you model gets too used to the training set – if the accuracy hits 1 it is probably wrong)
  • Think about mobile
  • Check quality of training data (a lot of bad data will create a bad result – beware the quality of open source content, if you don’t read and clean your data you will get crazy results)

Ideas

  • games
  • accessibility tools
  • design tools (font pairing, colours…)
  • OCR
  • sentiment recognition (eg. filtering nasty comments)
  • fraud detection
  • …so much more!

@devdevcharlie